Using the Python Threading Module for Concurrency

The Python threading module provides a straightforward way to run multiple operations concurrently within a single process. It is useful for I/O-bound tasks where the program spends time waiting on network responses, file reads, or database queries. This article explains how to use the module effectively, what synchronization primitives you need, and where the global interpreter lock (GIL) affects your results.

Creating and Starting Threads with the threading Module

The core of the module is the Thread class. You can create a thread by passing a callable to its constructor and then calling start(). The thread runs until the callable returns, and join() blocks the caller until the thread finishes.

import threading
import time

def worker(name: str, delay: float) -> None:
    print(f"{name} starting")
    time.sleep(delay)
    print(f"{name} finished")

thread = threading.Thread(target=worker, args=("A", 1.5))
thread.start()
thread.join()

The target argument accepts any callable, and args supplies positional arguments. Keyword arguments go in kwargs. The thread runs in the same process and shares memory with the main thread, which is why synchronization is often necessary.

For simple cases, you can also subclass Thread and override run(). This is useful when you need to store thread-specific state or expose additional methods.

class WorkerThread(threading.Thread):
    def __init__(self, name: str):
        super().__init__(name=name)
        self.result = None

    def run(self) -> None:
        self.result = 42

thread = WorkerThread("worker")
thread.start()
thread.join()
print(thread.result)

Subclassing is more verbose but gives you a clear place to keep per-thread data. For most cases, passing a target function is simpler and keeps the logic separate from the thread lifecycle.

Passing Data and Handling Thread Results

Threads share memory, so you can pass mutable objects like lists or dictionaries and have the thread modify them. However, you must protect those modifications with locks if multiple threads can write concurrently.

A common pattern is to use a queue to collect results from worker threads. The queue.Queue class is thread-safe and designed for this purpose.

import queue
import threading

def worker(q: queue.Queue, item: int) -> None:
    q.put(item * 2)

result_queue = queue.Queue()
threads = []
for i in range(5):
    t = threading.Thread(target=worker, args=(result_queue, i))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

while not result_queue.empty():
    print(result_queue.get())

Here each thread puts its result into the queue, and the main thread collects them after all threads finish. The queue handles locking internally, so you do not need a separate lock for the queue itself.

If you need a return value from a thread, the standard Thread class does not provide one. You can store the result on the thread object (as in the subclass example) or use a shared container with a lock. For a more modern approach, concurrent.futures.ThreadPoolExecutor returns Future objects that give you the result directly, as shown later.

Synchronizing Threads with Locks

When multiple threads read and write the same shared data, you need a lock to prevent race conditions. The threading.Lock class provides mutual exclusion: only one thread can hold the lock at a time.

import threading

counter = 0
lock = threading.Lock()

def increment() -> None:
    global counter
    for _ in range(1000):
        with lock:
            counter += 1

threads = [threading.Thread(target=increment) for _ in range(4)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(counter)  # 4000

The with lock: block acquires the lock before entering and releases it on exit, even if an exception occurs. Without the lock, the increment operation is not atomic, and the final counter could be less than 4000 because of interleaved reads and writes.

Use a lock only when you need to protect a critical section. Holding a lock while performing slow I/O can block other threads unnecessarily. Keep the critical section as short as possible.

Using Events and Conditions for Coordination

Sometimes threads need to wait for a signal from another thread. threading.Event provides a simple boolean flag that can be set, cleared, and waited on.

import threading
import time

event = threading.Event()

def waiter() -> None:
    print("Waiting for event")
    event.wait()
    print("Event received")

def setter() -> None:
    time.sleep(2)
    event.set()

threading.Thread(target=waiter).start()
threading.Thread(target=setter).start()

The wait() method blocks until another thread calls set(). This is useful for signaling that a resource is ready or that a shutdown has been requested.

For more complex coordination, threading.Condition combines a lock with a wait/notify mechanism. It allows threads to wait until a specific condition is true and to be notified when the condition may have changed.

import threading

condition = threading.Condition()
items = []

def producer() -> None:
    with condition:
        items.append("item")
        condition.notify()

def consumer() -> None:
    with condition:
        while not items:
            condition.wait()
        item = items.pop()
        print(item)

The consumer waits inside a with condition: block. The wait() releases the lock and blocks until another thread calls notify() or notify_all(). When notified, the consumer reacquires the lock and rechecks the condition in the while loop to avoid spurious wakeups.

Conditions are more flexible than events but also more error-prone. Use them when you need to wait on a predicate rather than a simple flag.

The Global Interpreter Lock and What It Means for Threads

The GIL is a mutex in CPython that allows only one thread to execute Python bytecode at a time. This means that pure Python code cannot run in parallel across multiple CPU cores. For CPU-bound tasks, threads will not speed up the computation; they may even slow it down due to context switching overhead.

However, the GIL is released during blocking I/O operations, such as time.sleep(), network reads, or file writes. That is why threads are effective for I/O-bound workloads: while one thread waits for I/O, another can execute.

If your workload is CPU-bound and you need to use multiple cores, consider the multiprocessing module, which uses separate processes with their own GIL. The tradeoff is higher memory usage and more complex data sharing.

For mixed workloads, you can combine threads and processes, but that adds complexity. Always measure your actual bottleneck before choosing a concurrency model.

Thread Pools with concurrent.futures.ThreadPoolExecutor

Managing threads manually is tedious and error-prone. The concurrent.futures.ThreadPoolExecutor provides a high-level interface for submitting callables to a pool of threads and retrieving their results as Future objects.

from concurrent.futures import ThreadPoolExecutor, as_completed

def square(x: int) -> int:
    return x * x

with ThreadPoolExecutor(max_workers=4) as executor:
    futures = [executor.submit(square, i) for i in range(10)]
    for future in as_completed(futures):
        print(future.result())

The executor manages a fixed number of threads and reuses them for multiple tasks. The with block ensures that all threads are joined and resources are cleaned up when the block exits.

You can also use executor.map() to apply a function to an iterable, but submit() gives you more control over individual tasks.

Thread pools are ideal when you have many short-lived tasks that would otherwise incur thread creation overhead. They also make it easier to limit the number of concurrent threads, preventing resource exhaustion.

Common Threading Pitfalls and How to Avoid Them

One frequent mistake is sharing mutable data without a lock. Even simple operations like list.append() are not guaranteed to be atomic across threads in all Python implementations. Always protect shared state with a lock or use thread-safe data structures like queue.Queue.

Another issue is deadlock. This happens when two threads each hold a lock and wait for the other to release its lock. To avoid deadlock, always acquire locks in a consistent order and use timeouts where possible.

lock1 = threading.Lock()
lock2 = threading.Lock()

# Potential deadlock if threads acquire locks in different order
# Use a consistent order or try-acquire with timeout
if lock1.acquire(timeout=1):
    if lock2.acquire(timeout=1):
        # critical section
        lock2.release()
    lock1.release()

The timeout parameter prevents indefinite blocking, but it does not solve the ordering problem. The best fix is to design your locking hierarchy so that every thread acquires locks in the same global order.

Finally, be careful with daemon threads. A daemon thread is killed abruptly when the main thread exits, which can leave resources in an inconsistent state. Use daemon threads only for background tasks that can be safely interrupted, and always join non-daemon threads before the program ends.

Threading is a powerful tool when applied to the right problem. For I/O-bound tasks, the threading module and ThreadPoolExecutor can significantly improve throughput. For CPU-bound tasks, look beyond threads to multiprocessing or asynchronous programming. Understanding the GIL and synchronization primitives will help you write correct and efficient concurrent code.