Python async for: Consuming Async Iterables

The python async for statement is the asynchronous counterpart of the regular for loop. It iterates over an asynchronous iterable, which produces values through an async iterator. This is essential when working with asyncio-based code that reads data from streams, async generators, or custom async iterables.

What Is an Async Iterable?

An async iterable is any object that implements the __aiter__ method, which returns an async iterator. The async iterator itself must implement __anext__, an async method that returns the next value or raises StopAsyncIteration when exhausted.

class AsyncCounter:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __aiter__(self):
        return self

    async def __anext__(self):
        if self.current >= self.limit:
            raise StopAsyncIteration
        self.current += 1
        return self.current

You can then consume this with async for inside an async function.

Basic Syntax of async for

The syntax mirrors a normal for loop, but it must appear inside an async def function. Each iteration awaits the next value from the async iterator.

async def main():
    async for number in AsyncCounter(5):
        print(number)

When you run this with asyncio.run(main()), it prints 1 through 5. The loop automatically calls __anext__ and awaits it, then handles StopAsyncIteration to exit.

How async for Works Internally

Under the hood, async for performs three steps repeatedly:

1. Calls __anext__() on the async iterator.
2. Awaits the returned coroutine to get the next value.
3. If StopAsyncIteration is raised, the loop terminates.

This is equivalent to the following manual implementation:

async def manual_iteration(iterable):
    iterator = iterable.__aiter__()
    while True:
        try:
            value = await iterator.__anext__()
        except StopAsyncIteration:
            break
        print(value)

The loop also handles cleanup by calling aclose() if the iterator provides it, which is important for async generators.

Practical Example: Consuming an Async Generator

Async generators are the easiest way to create async iterables. They are defined with async def and use yield to produce values.

async def fetch_pages(base_url, count):
    for page in range(1, count + 1):
        # Simulate an async HTTP request
        data = await fetch_url(f"{base_url}?page={page}")
        yield data

You can consume this with async for:

async def main():
    async for page_data in fetch_pages("https://api.example.com/items", 3):
        process(page_data)

Each yield suspends the generator, and the loop resumes it after the consumer processes the value. This allows the generator to perform async work between yields, such as I/O operations.

Using async for in Asynchronous Comprehensions

Python also supports asynchronous comprehensions, which use async for inside a list, set, or dictionary comprehension. This is useful when you want to collect all results from an async iterable into a container.

async def main():
    results = [item async for item in async_generator()]
    print(results)

The comprehension awaits each value sequentially, just like a loop. You can also add conditions:

even_squares = [x * x async for x in async_numbers() if x % 2 == 0]

This works because the comprehension is inside an async function and the async for clause is recognized by the parser.

Error Handling in async for Loops

Errors can arise from the async iterator itself. You can catch them with a regular try/except block around the loop.

async def main():
    try:
        async for item in flaky_async_iterable():
            process(item)
    except SomeError as e:
        log_error(e)

If the iterator raises an exception during __anext__, it propagates to the loop and can be caught. Be careful with StopAsyncIteration; you should not catch it manually because the loop handles it internally.

Performance and Concurrency Considerations

async for does not automatically parallelize iterations. Each value is awaited sequentially, which is fine when the producer and consumer are both I/O-bound and you need to process items in order. If you need to process multiple items concurrently, you can combine async for with asyncio.gather or create tasks.

For example, you might collect the first N items and then process them concurrently:

async def main():
    tasks = []
    async for item in async_source():
        tasks.append(asyncio.create_task(process(item)))
        if len(tasks) >= 10:
            await asyncio.gather(*tasks)
            tasks.clear()

This pattern is useful when each processing step is independent and you want to overlap I/O.

Common Pitfalls and Limitations

• async for cannot be used on a regular iterable. If you try, Python raises TypeError because the object doesn't implement __aiter__.
• You must be inside an async function. Using async for at module level or in a synchronous function is a syntax error.
• If the async iterator is an async generator, it is automatically closed when the loop exits, even if an exception occurs. This prevents resource leaks.
• Cancellation: if the task containing the loop is cancelled, the async generator's aclose() is called, which can clean up resources. Be aware of this when handling cancellation.

Understanding these boundaries helps you use async for effectively without unexpected behavior.