How to Create and Manage Threads in Python

When you need to run multiple tasks concurrently in Python, the threading module provides the most direct way to python create thread instances. The core class is threading.Thread, which wraps a callable and executes it in a separate thread of control. This article focuses on the mechanics of creating, starting, and coordinating threads, with attention to the runtime behavior that affects real applications.

The Minimal Thread Creation Pattern

The simplest way to create a thread is to pass a target function to the Thread constructor and call start(). The thread begins executing as soon as the interpreter schedules it, which is not guaranteed to be immediate.

import threading
import time

def worker():
    print("Worker starting")
    time.sleep(1)
    print("Worker done")

thread = threading.Thread(target=worker)
thread.start()
thread.join()

The join() call blocks the main thread until the worker finishes. Without it, the program might exit before the worker completes because the interpreter waits for non-daemon threads to finish, but join() gives you explicit control over the synchronization point.

Passing Arguments to the Target Function

Thread functions often need input data. The Thread constructor accepts args and kwargs that are forwarded to the target.

import threading

def process_item(item, retries=3):
    print(f"Processing {item} with {retries} retries")

thread = threading.Thread(target=process_item, args=("order-42",), kwargs={"retries": 5})
thread.start()
thread.join()

Arguments are passed positionally and by keyword exactly as they would be to a normal function call. This keeps the thread creation code readable and avoids relying on global state.

Daemon Threads and Program Exit Behavior

A daemon thread is one that does not block the interpreter from exiting when only daemon threads remain. Set daemon=True in the constructor or assign the attribute before start().

import threading
import time

def background_poll():
    while True:
        print("Polling...")
        time.sleep(2)

t = threading.Thread(target=background_poll, daemon=True)
t.start()
print("Main continues")
# Program exits immediately; daemon thread is killed.

Daemon threads are useful for background tasks that should not prevent shutdown, such as telemetry or cache warming. However, they are abruptly stopped when the interpreter exits, so they are not appropriate for work that must complete cleanly.

Thread Safety and the Global Interpreter Lock

Python's GIL (Global Interpreter Lock) means only one thread executes Python bytecode at a time. This makes CPU-bound Python code effectively serialized, but threads still help with I/O-bound tasks where the lock is released during blocking operations. Creating many threads for CPU-heavy work may not yield the expected speedup; the multiprocessing module is often a better fit for that scenario.

When threads share mutable data, race conditions can occur. The standard remedy is a threading.Lock to protect critical sections.

import threading

counter = 0
lock = threading.Lock()

def increment():
    global counter
    for _ in range(100000):
        with lock:
            counter += 1

threads = [threading.Thread(target=increment) for _ in range(10)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print(counter)

Without the lock, the increments can interfere because the read-modify-write sequence is not atomic. The with statement acquires and releases the lock even if an exception occurs.

Handling Exceptions Inside Threads

Exceptions raised inside a thread do not propagate to the caller of start() or join(). They are printed to sys.stderr and the thread terminates. To capture exceptions, wrap the target function and store the exception in a shared structure.

import threading
import sys

def safe_target(result_container):
    try:
        raise ValueError("boom")
    except Exception as e:
        result_container["error"] = e

result = {}
t = threading.Thread(target=safe_target, args=(result,))
t.start()
t.join()
if "error" in result:
    print(f"Thread failed: {result['error']}")

This pattern lets the main thread inspect the outcome after joining. For more structured concurrency, consider concurrent.futures.ThreadPoolExecutor, which returns Future objects that can hold exceptions.

Joining Threads and Timeouts

join() accepts an optional timeout. If the thread does not finish within the timeout, join() returns and the main thread continues. The thread is still running in the background.

import threading
import time

def slow_worker():
    time.sleep(5)

t = threading.Thread(target=slow_worker)
t.start()
t.join(timeout=2)
print(f"Thread alive after timeout: {t.is_alive()}")

This is useful for graceful shutdown sequences where you cannot block indefinitely. You can check is_alive() to decide whether to force a stop or leave the thread to finish.

Practical Thread Lifecycle Management

A thread can be started only once. Calling start() a second time raises RuntimeError. After a thread finishes, it cannot be restarted. For recurring tasks, create a new thread each time or use a worker pool.

Threads also hold references to their target function and arguments until they finish. If you create many short-lived threads in a loop, ensure they are joined or allowed to complete to avoid accumulating unreferenced objects. In CPython, the threading module keeps track of non-daemon threads; a program with many threads may consume significant memory for stack space, so avoid creating thousands of threads without a bounded pool.

For most production scenarios, ThreadPoolExecutor from concurrent.futures provides a higher-level interface with a fixed number of workers, which simplifies resource management and result collection.

from concurrent.futures import ThreadPoolExecutor

def square(x):
    return x * x

with ThreadPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(square, range(10)))
print(results)

The executor handles thread creation, reuse, and shutdown automatically. Use it when you need to run many independent tasks rather than managing individual threads manually.

When Threads Are the Right Tool

Threads in Python are most effective for I/O-bound workloads: network requests, file operations, database calls, or any operation that blocks on external resources. The GIL is released during most blocking I/O, so multiple threads can make progress concurrently. For CPU-bound computation, the GIL prevents true parallel execution, and multiprocessing or asyncio may be more appropriate.

Creating a thread is cheap relative to a process, but not free. Each thread has its own stack (typically several megabytes of virtual memory). If you need to handle thousands of concurrent tasks, an event loop or a thread pool with a bounded number of workers is more scalable than creating an unbounded number of threads.

Thread safety also extends to the standard library. Many operations, such as print() or list append, are thread-safe in CPython, but relying on that without understanding the underlying guarantees is risky. Always protect shared mutable state with locks or use queue.Queue for producer-consumer patterns.

The threading module gives you fine-grained control over thread creation and coordination. Start with the Thread class for simple cases, move to a pool for repetitive tasks, and consider multiprocessing when the GIL becomes the bottleneck. The key is to match the concurrency model to the nature of the work, not to the number of lines of code you want to write.