Python Wait Futures: How to Wait for Concurrent Tasks

python wait futures requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

Waiting on futures is a core part of concurrent programming in Python. The concurrent.futures.wait function blocks the calling thread until the given futures meet a condition you specify. This is the primary tool for synchronizing on multiple tasks when you need to react to their completion in a controlled way. Here's how to use it correctly, including timeout handling and the return_when parameter.

What Does concurrent.futures.wait Do?

concurrent.futures.wait takes an iterable of futures and waits until one of three conditions is met: all futures are done, the first future completes, or the first exception is raised. It returns a named tuple DoneAndNotDoneFutures containing two sets: the futures that are done and those that are not. The function does not raise exceptions from the futures themselves; it only reports their state.

The signature is:

concurrent.futures.wait(fs, timeout=None, return_when=ALL_COMPLETED)

• fs is the iterable of futures to wait on.
• timeout controls how long to wait before giving up. If None, it blocks indefinitely.
• return_when determines when the wait returns. The default is ALL_COMPLETED.

Using wait with Thread and Process Pools

You typically get futures from ThreadPoolExecutor or ProcessPoolExecutor. Here's a minimal example with a thread pool:

from concurrent.futures import ThreadPoolExecutor, wait

def square(n):
    return n * n

with ThreadPoolExecutor(max_workers=3) as executor:
    futures = [executor.submit(square, i) for i in range(10)]
    done, not_done = wait(futures)
    print(f"Done: {len(done)}, Not done: {len(not_done)}")

After wait returns, done contains all futures that have finished. You can then retrieve results with future.result(). If a future raised an exception, result() will re-raise it, so you should handle that accordingly.

Understanding the return_when Parameter

The return_when parameter accepts three constants from concurrent.futures:

• ALL_COMPLETED: waits until every future in fs is done. This is the default.
• FIRST_COMPLETED: returns as soon as at least one future is done. The remaining futures are placed in the not_done set.
• FIRST_EXCEPTION: returns when the first future raises an exception. If none raise an exception, it behaves like ALL_COMPLETED.

These options let you implement different coordination patterns. For example, you might want to start processing results as soon as the first task finishes, rather than waiting for all.

Here's how FIRST_COMPLETED works:

from concurrent.futures import ThreadPoolExecutor, wait, FIRST_COMPLETED

def slow_task(x):
    import time
    time.sleep(x)
    return x

with ThreadPoolExecutor(max_workers=2) as executor:
    futures = [executor.submit(slow_task, 2), executor.submit(slow_task, 5)]
    done, not_done = wait(futures, return_when=FIRST_COMPLETED)
    print(f"First done: {len(done)}, still running: {len(not_done)}")

Handling Timeouts with wait

The timeout parameter lets you avoid blocking forever. If the timeout expires before the return_when condition is met, wait returns immediately with the current state. The not_done set will contain futures that have not finished yet.

from concurrent.futures import ThreadPoolExecutor, wait

def long_task():
    import time
    time.sleep(10)

with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(long_task)
    done, not_done = wait([future], timeout=2)
    if not_done:
        print("Task still running, cancelling?")
        future.cancel()

Note that timeout does not cancel the futures; it only stops waiting. You must decide what to do with the unfinished futures, such as cancelling them or continuing to poll later.

Inspecting the Returned Futures

The DoneAndNotDoneFutures named tuple has two fields: done and not_done. Both are sets of Future objects. You can iterate over them to retrieve results or handle exceptions. A common pattern is to process only the completed futures after a timeout:

done, not_done = wait(futures, timeout=1)
for future in done:
    try:
        result = future.result()
    except Exception as exc:
        print(f"Task failed: {exc}")

Remember that future.result() blocks only if the future is not done. Since done contains only finished futures, calling result() on them is safe and immediate.

Waiting on Futures in Asyncio

For asynchronous code, asyncio.wait serves a similar purpose but works with coroutines and tasks. It is a coroutine itself and must be awaited. The API is slightly different: it accepts a set of tasks or futures, and return_when uses asyncio.tasks constants like ALL_COMPLETED, FIRST_COMPLETED, and FIRST_EXCEPTION.

import asyncio

async def fetch_data(delay):
    await asyncio.sleep(delay)
    return delay

async def main():
    tasks = [asyncio.create_task(fetch_data(i)) for i in range(3)]
    done, pending = await asyncio.wait(tasks, timeout=2, return_when=asyncio.FIRST_COMPLETED)
    print(f"Done: {len(done)}, Pending: {len(pending)}")

asyncio.run(main())

Unlike the synchronous version, asyncio.wait does not automatically cancel pending tasks when it returns. You must explicitly cancel them if you no longer need them. Also, in Python 3.11 and later, asyncio.wait is deprecated in favor of asyncio.TaskGroup for many use cases, but it remains valid for fine-grained control.

Choosing Between wait and as_completed

concurrent.futures.as_completed yields futures as they complete, one at a time. This is useful when you want to process results as soon as they become available, without waiting for all tasks. In contrast, wait gives you a snapshot of the state at a single point. Use as_completed when you want to iterate over results in completion order, and use wait when you need to pause until a certain condition or timeout occurs.

For example, if you have a batch of network requests and want to handle each response immediately, as_completed is more natural. If you need to wait for all requests to finish before proceeding, wait with ALL_COMPLETED is simpler.

Performance and Resource Considerations

Blocking on wait holds a thread. In a thread pool, this can reduce the number of available workers if the calling thread is also a worker. Avoid calling wait from inside a worker thread if possible. For long-running waits, consider using a timeout and periodically checking state, or use as_completed to avoid blocking altogether.

Memory usage is also a factor: keeping many futures in memory while waiting can be expensive. If you have a very large number of tasks, consider batching them with wait in chunks rather than all at once.

Common Pitfalls and How to Avoid Them

One frequent mistake is ignoring exceptions in futures. wait does not propagate exceptions; you must call result() on each done future to surface them. Another pitfall is assuming that wait with a timeout cancels tasks. It does not; you need to explicitly cancel or handle pending futures. Finally, when using FIRST_EXCEPTION, be aware that if no exception occurs, it waits for all futures, which may be unexpected if you intended to return early on success.

To handle these issues, always inspect the not_done set after a timeout and decide whether to cancel or continue. Use try/except around result() calls to catch exceptions from individual futures. And remember that wait is a blocking call; for asynchronous code, use asyncio.wait or consider asyncio.gather if you don't need fine-grained control.