Understanding Python Coroutines with async/await

A Python coroutine is a function defined with async def that can be suspended and resumed while awaiting asynchronous operations. Unlike a regular function that runs to completion, a coroutine yields control back to the event loop at each await point, allowing other tasks to make progress. This is the foundation of cooperative concurrency in Python's asyncio library.

Coroutine Syntax and the await Keyword

Declaring a coroutine is straightforward:

async def fetch_data(url):
    response = await http_get(url)
    return response.text

The async def marks the function as a coroutine. Inside, await suspends execution until the awaited awaitable completes. The awaited object can be another coroutine, a Future, or a Task. When you call fetch_data(url), the function body does not execute immediately. Instead, it returns a coroutine object that must be driven by an event loop.

coro = fetch_data("https://example.com")  # no execution yet

To actually run the coroutine, you need to schedule it on an event loop. The simplest way is asyncio.run():

import asyncio

async def main():
    result = await fetch_data("https://example.com")
    print(result)

asyncio.run(main())

asyncio.run() creates a new event loop, runs the given coroutine until completion, and closes the loop. It is the recommended entry point for most asyncio programs.

Coroutines vs. Generators

Python has two distinct mechanisms that are sometimes confused: generator-based coroutines and native coroutines. Generators, defined with yield, produce values lazily and can be manually advanced with next(). They were historically used to implement coroutines via yield from, but native coroutines with async def are now the standard.

Key differences:

A generator can produce values and also receive values via send(), but it cannot suspend on an asynchronous operation without an external scheduler. Native coroutines are designed to work with an event loop that knows when I/O is ready.

Running Coroutines Concurrently with Tasks

A coroutine that awaits another coroutine runs sequentially. To achieve concurrency, you need to schedule multiple coroutines as Tasks. A Task wraps a coroutine and schedules it on the event loop independently.

import asyncio

async def worker(name, delay):
    await asyncio.sleep(delay)
    print(f"{name} done")

async def main():
    task1 = asyncio.create_task(worker("first", 2))
    task2 = asyncio.create_task(worker("second", 1))
    await task1
    await task2

asyncio.run(main())

asyncio.create_task() schedules the coroutine to run soon. The event loop interleaves execution between await points. In this example, task2 completes before task1 even though it was created second, because its sleep is shorter.

For gathering multiple tasks and collecting their results, asyncio.gather() is convenient:

async def fetch_all(urls):
    return await asyncio.gather(*(fetch_data(url) for url in urls))

gather() runs the coroutines concurrently and returns a list of results in the original order. If any task raises an exception, gather() propagates it immediately unless return_exceptions=True is set.

Error Handling and Cancellation

Exceptions inside a coroutine propagate to the point where the coroutine is awaited. If a task raises an unhandled exception, it is stored in the task and re-raised when you await that task. You can inspect it without awaiting by using task.exception().

Cancellation is a separate mechanism. Calling task.cancel() schedules a CancelledError to be thrown into the coroutine at its current await point. The coroutine can catch this exception to perform cleanup:

async def long_running():
    try:
        await asyncio.sleep(3600)
    except asyncio.CancelledError:
        print("Cancelled, cleaning up")
        raise

It is important to re-raise CancelledError after cleanup unless you have a specific reason to suppress cancellation. Swallowing it can leave the task in an inconsistent state.

Performance and Concurrency Considerations

Coroutines provide concurrency without threads, but they are not a silver bullet. The event loop runs on a single thread, so CPU-bound code blocks all other tasks. A long-running loop inside a coroutine will stall the entire loop. For CPU-heavy work, use asyncio.to_thread() or a process pool.

I/O-bound operations benefit most from coroutines because the loop can switch to another task while waiting for network or disk. The overhead of creating a coroutine and awaiting it is small, but each await has a cost. Overusing fine-grained awaits can reduce throughput. Batch work where possible.

Memory usage is lower than thread-based concurrency because coroutines do not require separate stack memory. However, each coroutine object has overhead, so creating thousands of them is fine, but millions may strain memory.

Common Pitfalls and How to Avoid Them

Forgetting to await is the most frequent mistake. Calling async def function returns a coroutine object; if you don't await it, you get a warning and the coroutine never runs.

# Wrong: coroutine is created but never awaited
fetch_data(url)

# Correct
result = await fetch_data(url)

Blocking the event loop with synchronous calls like time.sleep() or requests.get() freezes all tasks. Use await asyncio.sleep() and aiohttp or httpx for async I/O.

Mixing sync and async code without care. If you call a blocking function inside a coroutine, you defeat the purpose. Use loop.run_in_executor() or asyncio.to_thread() to offload.

Creating tasks and not keeping references can lead to tasks being garbage collected before completion. Store tasks in a list or use asyncio.gather() to hold them.

Assuming thread safety – coroutines are not thread-safe. Sharing mutable state between coroutines without synchronization can cause races. Use asyncio.Lock or asyncio.Queue for coordination.

When to Use Coroutines vs. Threads or Processes

Coroutines are ideal for I/O-bound applications with many concurrent connections, such as web servers, API clients, or network scanners. They scale well because the event loop can handle thousands of tasks with minimal overhead.

Threads are suitable when you have blocking I/O that cannot be made asynchronous, or when you need true parallelism on multiple cores. Processes are necessary for CPU-bound work. The decision often comes down to the nature of the workload and the ecosystem of libraries available. If your dependency stack is synchronous, threads may be simpler than rewriting everything with async alternatives.

For most modern Python network services, async is the default choice because frameworks like FastAPI and aiohttp are built around coroutines. Understanding how coroutines execute is essential for debugging concurrency issues and writing efficient asynchronous code.