Python Async Comprehension: Syntax and Usage

Python async comprehension lets you use async for inside a list, dict, set, or generator expression to consume async iterables. It is a concise way to transform or filter data that is produced asynchronously, such as lines from a network stream, events from a message queue, or rows from an async database cursor.

What Is an Async Comprehension?

A comprehension is a compact syntax for building a container or generator from an iterable. An async comprehension extends this to async iterables. Instead of writing for item in iterable, you write async for item in async_iterable. The async for clause tells Python that each iteration may involve an await point, so the comprehension itself must run inside an async function.

Here is the simplest form:

async def get_values():
    return [item async for item in async_source()]

The async_source() function is an async iterable, meaning it implements __aiter__() and __anext__(). The comprehension collects each awaited item into a list.

Syntax for List, Dict, Set, and Generator Comprehensions

The syntax mirrors the synchronous versions, with async for replacing for. You can also add an optional if filter after the async iterator.

List comprehension:

results = [x async for x in async_gen()]

Dict comprehension:

mapping = {key: value async for key, value in async_pairs()}

Set comprehension:

unique = {x async for x in async_values()}

Generator expression (returns an async generator):

gen = (x async for x in async_values())

Each of these requires the enclosing function to be an async function (async def). You cannot use an async comprehension inside a regular function because async for is only allowed in an async context.

How Async Comprehensions Work Under the Hood

When you write [x async for x in async_iterable], Python creates an implicit async loop. It calls async_iterable.__aiter__() to get an async iterator, then repeatedly calls __anext__() on it, awaiting each result. The comprehension collects the results into a list, dict, or set, just like a synchronous comprehension.

This means the iteration is sequential. Each __anext__() call is awaited before the next one starts. If the async iterable itself performs I/O, each item is processed one at a time. That is often not what you want if the items are independent and you could fetch them concurrently.

Common Use Cases

Async comprehensions are useful when you need to transform or filter an async stream. For example, reading lines from an async HTTP response, processing events from a message queue, or iterating over rows from an async database driver.

A typical pattern is to fetch data from multiple URLs sequentially:

async def fetch_all(urls):
    async def gen():
        for url in urls:
            yield await fetch(url)
    return [response async for response in gen()]

Here, gen() is an async generator that yields each response after awaiting fetch(url). The comprehension collects them into a list. However, this still processes URLs one at a time.

Combining with asyncio.gather for Concurrency

Async comprehensions do not introduce concurrency. They are a syntax convenience for consuming async iterables sequentially. If you need to run multiple independent async operations concurrently, asyncio.gather is the right tool.

responses = await asyncio.gather(*(fetch(url) for url in urls))

This starts all fetches at once and waits for all to complete. The result is a list of responses. You can then process that list with a regular comprehension:

parsed = [parse(r) for r in responses]

If you need to apply an async transformation to each result, you can combine gather with an async comprehension over the gathered results, but that would be sequential again. A better pattern is to use gather on the transformed tasks:

results = await asyncio.gather(*(process(fetch(url)) for url in urls))

This runs each process concurrently.

Performance and Memory Considerations

Because async comprehensions iterate sequentially, they do not improve throughput for I/O-bound tasks. They can actually be slower than a synchronous loop if the async iterable has overhead. Use them when you need to consume an async stream that is naturally sequential, such as a stream of events that must be processed in order.

Memory usage is similar to synchronous comprehensions: a list comprehension materializes the entire result in memory. If you are processing a large async stream, consider an async generator expression to avoid storing all items at once. For example:

async def process_stream():
    async for item in async_stream():
        yield item * 2

This yields items lazily, reducing memory pressure.

Common Pitfalls and Errors

One common mistake is trying to use an async comprehension outside an async function. You will get a SyntaxError because async for is only allowed inside async def.

Another is mixing up async and sync iterables. If you write [x async for x in regular_list], Python raises a TypeError because a regular list is not an async iterable. You must have an object that implements __aiter__.

Also, forgetting to await an async function inside the comprehension will not work. The comprehension itself is not a place to call await directly; you need to have an async iterable that yields the awaited values. If you have a list of coroutines, you cannot do [await coro async for coro in coros] because the async for expects an async iterable, not a list of coroutines. Instead, use asyncio.gather.

Async Generator Expressions and Lazy Evaluation

An async generator expression is a compact way to create an async generator. It uses the same async for syntax but with parentheses. The generator is lazy: it does not produce values until you iterate over it. This is useful for streaming data without holding everything in memory.

async def main():
    gen = (x * 2 async for x in async_source())
    async for value in gen:
        print(value)

You can also use an if filter:

even = (x async for x in numbers() if x % 2 == 0)

Async generator expressions are often used to build pipelines that process data incrementally.

When to Use Async Comprehensions vs. Other Patterns

Async comprehensions are best when you have an async iterable that yields values sequentially and you need to collect them into a container. If you need concurrency, use asyncio.gather. If you need lazy processing, use an async generator expression. If you are just awaiting a list of coroutines, use gather or asyncio.as_completed.

The decision comes down to the the nature of your data source. If it is an async iterator, a comprehension is a natural fit. If it is a collection of coroutines, gather is the standard tool.