Python Async Generators: Syntax and Real-World Use

When you need to produce a sequence of values that arrive asynchronously—from a network, a database cursor, or a long-running computation—a regular generator cannot await anything. A Python async generator fills that gap: it is a function defined with async def that contains yield, so it can suspend not only on yield but also on await expressions. This makes it the natural tool for streaming data in asyncio-based applications.

The Core Syntax of an Async Generator

An async generator is defined exactly like a normal generator, except the function is declared with async def and the body contains at least one yield. The presence of yield makes it an async generator, not a coroutine.

async def countdown(n: int):
    while n > 0:
        await asyncio.sleep(1)
        yield n
        n -= 1

Calling countdown(3) does not execute the body. It returns an asynchronous generator object, which implements the asynchronous iterator protocol. The function starts running only when you begin iterating with async for or by calling __anext__() directly.

async for number in countdown(3):
    print(number)

Each time the generator reaches a yield, it suspends and returns a value. The next iteration resumes execution from that point, allowing await expressions to run between yields. This is the key difference from a regular generator, which cannot await inside its body.

How Async Iteration Works

An async generator is an async iterator, meaning it implements __anext__ and __aiter__. The async for loop calls __anext__ repeatedly and awaits each result. The loop terminates when __anext__ raises StopAsyncIteration.

ag = countdown(2)
print(await ag.__anext__())  # 2
print(await ag.__anext__())  # 1

You rarely call __anext__ directly, but understanding the protocol helps when you need to manually advance a stream or integrate with code that expects an async iterator.

The async for loop also handles cleanup automatically: if the loop exits early—for example, because of a break or an exception—the generator is closed via aclose(). This is important for releasing resources like open sockets or file handles.

Practical Use Cases for Async Generators

Async generators shine when data arrives incrementally and you want to process it without buffering the entire result in memory. Common scenarios include:

• Streaming lines from a large file read asynchronously.
• Paginating through an API where each page requires an HTTP request.
• Reading from a message queue or WebSocket.
• Producing chunks of data from a slow computation.

Here is a pattern for paginating a REST API:

async def fetch_pages(client, url):
    page = 1
    while True:
        data = await client.get(f"{url}?page={page}")
        if not data:
            return
        yield data
        page += 1

The consumer can iterate over pages with async for and stop early if needed, without downloading every page. The generator holds no extra state beyond its local variables, and the caller controls when the next page is fetched.

Handling Exceptions and Cleanup

Because an async generator can await, it can also catch exceptions that occur during iteration. You can wrap the body in try/finally to ensure cleanup runs when the generator is closed, whether by normal exhaustion, an early break, or an exception.

async def open_stream():
    try:
        async for chunk in network_source():
            yield chunk
    finally:
        await close_connection()

The finally block runs when aclose() is called. This is the async equivalent of finally in a regular generator, and it is the right place to release resources that require an await, such as closing a session or flushing a buffer.

If an exception is raised inside the generator body, it propagates to the consumer. The consumer can catch it in the async for loop. The generator is then automatically closed, and the finally block executes.

Performance and Resource Considerations

Async generators are not faster than regular generators; they are a different tool for a different problem. Their value is that they avoid blocking the event loop while waiting for I/O. When you await inside an async generator, the event loop can run other tasks. This is the core performance benefit.

Memory usage is also a key advantage. Because values are produced lazily, you never need to hold the full sequence in memory. For example, streaming a 10 GB file line by line uses only enough memory for one line, provided the file is read asynchronously in chunks.

There is a subtle backpressure consideration. The consumer controls the pace: the generator does not run ahead unless the consumer requests the next value. This is natural flow control. However, if you wrap an async generator in a task and buffer values manually, you can consume memory quickly. Let the consumer drive the iteration unless you have a specific reason to prefetch.

When to Use an Async Generator vs. Other Patterns

A regular generator is sufficient when you do not need to await inside the producer. If your data is already in memory or can be produced synchronously, a regular generator is simpler and faster.

An async iterator class is an alternative when you need to maintain state across iterations in a more structured way. Async generators are usually more concise, but a class gives you explicit control over __anext__ and __aiter__, which can be useful for complex state machines.

A list comprehension or list() call is appropriate only when the data set is small and you need random access. For streaming, an async generator is almost always the better choice because it avoids buffering the entire collection.

Here is a quick decision guide:

One common mistake is to write an async generator and then wrap it in list() to collect all values. That defeats the lazy streaming benefit and can exhaust memory. If you need all values, consider whether you actually need them at once.

Async generators integrate cleanly with asyncio primitives. You can pass them to functions that accept async iterables, use them with asyncio.gather by materializing them, or chain them with other async generators. They are a core building block for writing non-blocking, incremental data processing in Python.