Python asyncio wait: Managing Concurrent Tasks

When you need to run several coroutines concurrently and react to their completion as it happens, python asyncio wait gives you more control than asyncio.gather. It returns two sets of tasks—those that are done and those that are still pending—and lets you specify when the call should return. This makes it useful for partial results, timeouts, and first-error handling.

What asyncio.wait Does and When to Use It

The asyncio.wait coroutine takes a collection of awaitables (usually tasks or coroutines) and waits until a condition defined by the return_when parameter is met. Unlike asyncio.gather, which waits for all tasks and then returns their results in order, asyncio.wait returns two sets: done and pending. You can inspect these sets to see which tasks finished and which are still running.

import asyncio

async def worker(name, delay):
    await asyncio.sleep(delay)
    return f"{name} finished"

async def main():
    tasks = [
        asyncio.create_task(worker("A", 2)),
        asyncio.create_task(worker("B", 1)),
        asyncio.create_task(worker("C", 3)),
    ]
    done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
    print("Done:", [t.result() for t in done])
    print("Pending count:", len(pending))

asyncio.run(main())

In this example, asyncio.wait returns as soon as the first task (B) completes. The pending set still contains the other two tasks, which you can continue to await or cancel. This is the core difference from gather: you get immediate visibility into partial progress.

Understanding the return_when Parameter

The return_when parameter accepts three constants from the asyncio module:

FIRST_EXCEPTION is particularly useful for fail-fast scenarios. If you have a set of independent requests and any one fails, you may want to stop waiting and handle the error immediately. With ALL_COMPLETED, asyncio.wait behaves similarly to gather in terms of waiting, but it still gives you the done and pending sets separately, which can be helpful for cleanup.

Working with the Done and Pending Sets

After asyncio.wait returns, you can iterate over done to retrieve results or exceptions. The pending set contains tasks that have not finished yet. You can choose to await them later, cancel them, or ignore them. If you ignore them, they will continue running in the background, which may cause warnings about pending tasks when the event loop closes.

async def main():
    tasks = [asyncio.create_task(worker(f"Task {i}", i)) for i in range(1, 4)]
    done, pending = await asyncio.wait(tasks, timeout=2.0)
    for task in done:
        try:
            print(task.result())
        except Exception as exc:
            print(f"Task raised: {exc}")
    # Cancel remaining tasks to avoid background execution
    for task in pending:
        task.cancel()
    await asyncio.gather(*pending, return_exceptions=True)

This pattern is common when you want to enforce a deadline. If a task does not finish within the timeout, it remains in pending, and you can cancel it explicitly. Without cancellation, the event loop may complain about unretrieved exceptions or unfinished tasks.

Handling Exceptions and Cancellation

Tasks in the done set may have completed normally, raised an exception, or been cancelled. To check which happened, use task.exception() or task.cancelled(). Calling task.result() on a task that raised an exception will re-raise it, so you should wrap it in a try/except or use task.exception() first.

async def failing_task():
    raise ValueError("boom")

async def main():
    task = asyncio.create_task(failing_task())
    done, _ = await asyncio.wait({task})
    if task.cancelled():
        print("Task was cancelled")
    elif task.exception() is not None:
        print(f"Task failed: {task.exception()}")
    else:
        print(task.result())

Cancellation is a separate state. If you cancel a task before it completes, it will appear in done with cancelled() returning True. When you use asyncio.wait with FIRST_COMPLETED, a cancelled task can trigger the return, so you need to check for cancellation explicitly rather than assuming every task in done has a result.

asyncio.wait vs asyncio.gather

asyncio.gather is the simpler choice when you want all results in the original order and are prepared to wait for everything. It automatically propagates the first exception raised by any task, and it returns a list of results. asyncio.wait is better when you need partial results, timeouts, or the ability to act on tasks as they finish.

Use gather when you need all results and can tolerate waiting for the slowest task. Use asyncio.wait when you want to implement a timeout, process results incrementally, or handle failures without aborting the entire batch.

Timeouts and Partial Completion

The timeout parameter in asyncio.wait is a float number of seconds. When the timeout expires, the function returns whatever tasks have completed so far. The remaining tasks stay in the pending set. This is not the same as asyncio.wait_for, which cancels the underlying task after a timeout. With asyncio.wait, the pending tasks continue to run unless you explicitly cancel them.

async def main():
    tasks = [asyncio.create_task(worker(f"Task {i}", i)) for i in range(1, 5)]
    done, pending = await asyncio.wait(tasks, timeout=2.5)
    print(f"Completed {len(done)} tasks, {len(pending)} still pending")
    for task in pending:
        task.cancel()
    await asyncio.gather(*pending, return_exceptions=True)

This pattern gives you a soft deadline: you collect what finished within the time limit, then decide whether to cancel the rest or let them run. If you want a hard deadline that cancels automatically, wrap the entire asyncio.wait call in asyncio.wait_for.

Common Pitfalls and Runtime Behavior

One frequent mistake is passing coroutine objects directly to asyncio.wait instead of wrapping them in tasks. Coroutines passed to asyncio.wait are automatically wrapped in tasks, but this happens only when the coroutine is not already a task. If you pass a list of coroutines, they will be converted to tasks internally, but you lose the reference to the task objects unless you create them yourself. To avoid confusion, always create tasks explicitly with asyncio.create_task before passing them to asyncio.wait.

Another issue is forgetting to consume exceptions from tasks in the done set. If a task raises an exception and you never call task.result() or task.exception(), the event loop will log an "exception was never retrieved" warning. This is not just noise; it can hide real failures. Always inspect the outcome of each task in done.

Finally, remember that asyncio.wait is a coroutine, so it must be awaited. It cannot be used directly with asyncio.run without being inside another coroutine. The event loop behavior also depends on the Python version; asyncio.wait has been stable since Python 3.4, but the recommended way to create tasks is asyncio.create_task, which was added in Python 3.7. For older versions, use asyncio.ensure_future instead.

When you need to coordinate multiple concurrent operations and want fine-grained control over completion, timeouts, and partial results, python asyncio wait is the tool to reach for. Its done and pending sets give you a clear view of what has finished and what is still running, and the return_when parameter lets you decide exactly what condition should end the wait.