C# Async Iterator: yield return and await foreach

C# async iterators, built on IAsyncEnumerable<T>, let you produce a sequence of values asynchronously using yield return and consume them with await foreach. This pattern is useful when each element requires an asynchronous operation, such as reading from a stream, paging through an API, or processing records from a database cursor. Unlike returning a fully materialized Task<List<T>>, an async iterator streams results as they become available, reducing memory pressure and enabling early cancellation.

What Is an Async Iterator in C#?

An async iterator is a method that returns IAsyncEnumerable<T> and uses yield return to provide elements. The method can contain await expressions before each yield, meaning the next value is produced only after the preceding asynchronous work completes. The compiler transforms the method into a state machine that tracks progress across awaits, similar to how async/await works for Task-returning methods.

The key difference from a regular iterator is that the enumeration itself is asynchronous. The consumer does not block while waiting for the next element; instead, it awaits the next value. This is essential when the data source is remote or when the production of each item involves I/O or CPU-bound work that should not block the calling thread.

Declaring an Async Iterator with yield return

To declare an async iterator, define a method that returns IAsyncEnumerable<T> and mark it with async. Inside, use yield return to emit each value. You can also use await before each yield to perform asynchronous work.

public async IAsyncEnumerable<int> GetNumbersAsync()
{
    for (int i = 1; i <= 5; i++)
    {
        await Task.Delay(100); // Simulate asynchronous work
        yield return i;
    }
}

This method produces numbers 1 through 5, waiting 100 ms between each. The yield return suspends the method and returns the current value to the consumer. When the consumer requests the next value, the method resumes from the suspension point, executes the next await, and continues.

The async keyword is required because the method contains await expressions. Without async, you cannot use await inside an iterator. The return type must be IAsyncEnumerable<T> (or IAsyncEnumerator<T> for a lower-level enumerator).

Consuming an Async Iterator with await foreach

To consume an async iterator, use await foreach. This loop awaits each element as it becomes available, allowing the consumer to process values without blocking.

await foreach (int number in GetNumbersAsync())
{
    Console.WriteLine(number);
}

The await foreach statement is the asynchronous counterpart of foreach. It calls MoveNextAsync() on the enumerator, which returns a ValueTask<bool>. The loop body executes after each successful move, and the loop ends when MoveNextAsync() returns false.

You can also use await foreach with a CancellationToken to cancel the enumeration early. The token is passed to the enumerator's MoveNextAsync method, allowing the iterator to observe cancellation.

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2));
await foreach (int number in GetNumbersAsync().WithCancellation(cts.Token))
{
    Console.WriteLine(number);
}

The WithCancellation extension method attaches the token to the enumerable. Inside the iterator, you can check token.ThrowIfCancellationRequested() or pass the token to the underlying async operations.

Cancellation and Error Handling

Async iterators support cancellation through a CancellationToken. The token should be passed to the iterator method and used in the underlying async calls. When the token is cancelled, the iterator should throw OperationCanceledException, which the consumer can catch.

public async IAsyncEnumerable<int> GetNumbersAsync(CancellationToken token = default)
{
    for (int i = 1; i <= 5; i++)
    {
        token.ThrowIfCancellationRequested();
        await Task.Delay(100, token);
        yield return i;
    }
}

If the consumer cancels the enumeration, the iterator stops producing values. The await foreach loop will propagate the exception unless you handle it. It is common to wrap the loop in a try/catch to react to cancellation gracefully.

Error handling in async iterators works similarly to synchronous iterators. If an exception is thrown inside the iterator, it is propagated to the consumer at the point where the next element is requested. You can use try/finally to ensure cleanup, but you cannot use yield return inside a try block that has a catch clause. The compiler forbids that combination because it would require resuming after an exception. Instead, place cleanup in a finally block without yield inside it.

Performance and Resource Considerations

Async iterators stream data, which can significantly reduce memory usage compared to materializing a full collection. If you return Task<IEnumerable<T>>, you must build the entire sequence before returning it, holding all items in memory. An async iterator produces each item on demand, so memory usage is proportional to the number of items currently being processed, not the total sequence length.

However, each yield return incurs overhead from the state machine and the asynchronous move. If the per-item work is trivial and the data source is already in memory, a synchronous iterator or a simple List<T> may be more efficient. Async iterators are most valuable when each element requires I/O or when the consumer may not need the entire sequence.

Another consideration is disposal. An IAsyncEnumerable<T> implements IAsyncDisposable through its enumerator. The await foreach loop disposes the enumerator asynchronously when the loop exits, whether normally or via an exception. If your iterator holds unmanaged resources, implement DisposeAsync to release them. The compiler-generated state machine handles this automatically if you use await using or rely on the loop's implicit disposal.

When to Use an Async Iterator vs Task<IEnumerable<T>>

Choosing between IAsyncEnumerable<T> and Task<IEnumerable<T>> depends on how the data is produced and consumed. Use an async iterator when:

• Each element requires asynchronous work, and you want to avoid blocking while producing the next value.
• The consumer may stop early, and you want to avoid doing unnecessary work.
• The sequence is potentially large, and streaming reduces memory usage.

Use Task<IEnumerable<T>> when the entire sequence is available after a single asynchronous operation, such as a database query that returns all rows at once. In that case, the async work happens before the sequence is returned, and the enumeration itself is synchronous.

The decision also affects error behavior. With Task<IEnumerable<T>>, errors that occur during data production are thrown when you await the task. With an async iterator, errors are thrown during enumeration, at the point where the failing element is requested. This can be more natural for streaming scenarios but requires the consumer to handle exceptions inside the loop.

Common Pitfalls and Limitations

Async iterators have a few constraints worth knowing. You cannot use yield return inside a try block that has a catch clause. The compiler rejects this because it cannot represent the resume point after an exception. You can use try/finally without yield in the finally, but any yield must be outside the try or in a separate method.

Async iterators cannot have ref or out parameters, and they cannot return IAsyncEnumerator<T> directly from a method that also uses yield. The method must return IAsyncEnumerable<T> or IAsyncEnumerator<T>; the latter is rarely used directly.

You cannot use await foreach in a LINQ query directly. LINQ operators like Select and Where work on IEnumerable<T> or IQueryable<T>, not on IAsyncEnumerable<T>. To apply transformations, you need to use System.Linq.Async from the System.Interactive.Async package, or manually iterate and build a new async iterator.

Finally, be careful with the lifetime of the enumerator. If you call GetAsyncEnumerator manually, you must dispose it. The await foreach loop does this automatically, but if you are implementing a custom enumerator or using the enumerator directly, ensure you call DisposeAsync in a finally block to avoid resource leaks.