Python Condition Variable: Synchronizing Threads with wait and notify

A Python condition variable lets one or more threads wait until another thread signals a state change. It is part of the threading module and is commonly used for producer-consumer patterns where a worker thread must wait for data to become available. The core idea is simple: a thread acquires an associated lock, checks a predicate, and if the predicate is false, it calls wait() to release the lock and sleep. When another thread changes the state, it calls notify() or notify_all() to wake the waiting thread, which then reacquires the lock and rechecks the predicate.

What a Condition Variable Does in Python

A threading.Condition object is always tied to a lock. You can create it with an existing lock or let it create its own RLock by default. The condition provides three essential methods:

• wait(timeout=None) — releases the lock, blocks until notified or timeout expires, then reacquires the lock.
• notify(n=1) — wakes up at most n waiting threads.
• notify_all() — wakes up all waiting threads.

All three must be called while the thread holds the associated lock. The lock is what protects the shared state you are waiting on. Without it, you risk race conditions where a notification arrives between checking a predicate and going to sleep.

Minimal Example: Waiting for a State Change

Here is a minimal scenario: one thread waits for a flag to become true, and another thread sets it.

import threading
import time

condition = threading.Condition()
flag = False

def waiter():
    global flag
    with condition:
        while not flag:
            print("Waiter: waiting")
            condition.wait()
        print("Waiter: flag is now True")

def setter():
    global flag
    time.sleep(1)
    with condition:
        flag = True
        print("Setter: setting flag and notifying")
        condition.notify()

t1 = threading.Thread(target=waiter)
t2 = threading.Thread(target=setter)
t1.start()
t2.start()
t1.join()
t2.join()

The while not flag loop is not optional. It guards against spurious wakeups and against the possibility that another thread consumed the notification before this thread reacquired the lock. Always recheck the predicate after wait() returns.

Producer-Consumer with a Condition Variable

The classic use case is a shared buffer where a producer adds items and a consumer removes them. The consumer must wait when the buffer is empty, and the producer must notify when it adds an item.

import threading
import time
import random

class Buffer:
    def __init__(self, capacity):
        self.capacity = capacity
        self.items = []
        self.condition = threading.Condition()

    def put(self, item):
        with self.condition:
            while len(self.items) >= self.capacity:
                self.condition.wait()
            self.items.append(item)
            self.condition.notify()

    def get(self):
        with self.condition:
            while not self.items:
                self.condition.wait()
            item = self.items.pop(0)
            self.condition.notify()
            return item

def producer(buffer, count):
    for i in range(count):
        buffer.put(i)
        print(f"Produced {i}")
        time.sleep(random.uniform(0.1, 0.5))

def consumer(buffer, count):
    for _ in range(count):
        item = buffer.get()
        print(f"Consumed {item}")
        time.sleep(random.uniform(0.2, 0.6))

buffer = Buffer(5)
p = threading.Thread(target=producer, args=(buffer, 10))
c = threading.Thread(target=consumer, args=(buffer, 10))
p.start()
c.start()
p.join()
c.join()

Notice that both put and get use notify() after modifying the buffer. This wakes up a single waiting thread. If multiple consumers are waiting, notify() wakes only one, which is usually sufficient. If you have multiple producers and consumers, you may need notify_all() to avoid starvation.

Handling Spurious Wakeups and Timeouts

Python's Condition.wait() can return without an explicit notification. This is called a spurious wakeup and is allowed by the underlying OS. The while loop around the predicate is the standard defense.

You can also pass a timeout to wait(). This is useful when you want to give up after a certain period, for example in a polling loop or when handling shutdown signals.

def wait_with_timeout(condition, predicate, timeout):
    with condition:
        while not predicate():
            if not condition.wait(timeout):
                return False
        return True

If the timeout expires, wait() returns False; if it was notified, it returns True. The predicate must still be checked, because a notification does not guarantee the predicate is now true.

Condition Variable vs Event and Queue

Python provides other synchronization primitives that may be simpler for specific use cases. A threading.Event is a one-time flag that threads can wait on. It is simpler but does not support complex predicates or multiple state changes. A queue.Queue internally uses condition variables and locks, and is often the best choice for producer-consumer communication because it handles buffering, blocking, and thread safety for you.

Use a condition variable when you need to wait for a specific condition that is more complex than a simple flag, or when you need fine-grained control over notification behavior. For straightforward FIFO data exchange, queue.Queue is less error-prone.

Deadlock and Performance Considerations

Condition variables are prone to deadlocks if you do not hold the lock when calling notify() or if you call wait() without a predicate loop. Always acquire the lock using with condition: before calling any of its methods.

Another common mistake is calling notify() while holding the lock and then performing a slow operation before releasing it. The notified thread will wake up and immediately try to reacquire the lock, so it will block until you release it. This is not a deadlock, but it can cause unnecessary contention. If you have many waiting threads, notify_all() is more expensive than notify() because it wakes every thread, but it may be necessary to prevent starvation when multiple threads wait on different predicates.

From a performance perspective, condition variables are lightweight compared to busy-waiting, but they still involve OS-level thread blocking and wakeups. If you have a high-throughput producer-consumer workload, consider using queue.Queue with a bounded size, which is implemented in C and avoids Python-level lock overhead for each item.

Advanced Usage: Multiple Conditions and Predicates

You can use a single condition variable with multiple predicates, but you must be careful with notify_all() to ensure the correct thread wakes up. Alternatively, you can use separate condition variables for different state changes, each with its own lock. This reduces unnecessary wakeups.

class SharedState:
    def __init__(self):
        self.lock = threading.Lock()
        self.ready = threading.Condition(self.lock)
        self.done = threading.Condition(self.lock)
        self.data = None

    def produce(self, value):
        with self.ready:
            self.data = value
            self.ready.notify()

    def consume(self):
        with self.ready:
            while self.data is None:
                self.ready.wait()
            value = self.data
            self.data = None
            return value

Using separate conditions can improve responsiveness when threads wait on unrelated conditions. However, it also increases complexity because you must ensure consistent lock ordering to avoid deadlocks. In practice, a single condition with a well-defined predicate is often sufficient and easier to reason about.