RDRYUSLOG DEV
Back to Blog
Python

python yield from vs for loop: Delegating Generators

python yield from vs for loop: Understand the difference between `yield from` and a manual `for` loop when delegating to a subgenerator, including return values, excep...

generatorsyield fromfor loopcoroutinesiteration
Diagram comparing yield from and for loop delegation in Python generators

When a generator needs to delegate iteration to another generator, you have two common ways to write it: a for loop that yields each item, or the yield from expression. The choice affects not only readability but also how return values and exceptions propagate. This article compares python yield from vs for loop in practical terms, showing what each approach does and when to prefer one over the other.

The Problem: Delegating to a Subgenerator

Imagine you have a generator that produces a sequence, and you want another generator to yield all of those items, possibly along with some of its own. The naive approach is to loop over the inner generator and yield each item:

def outer():
    for item in inner():
        yield item
    yield "extra"

This works, but it has a subtle limitation: if the inner generator returns a value (using the return statement), that value is lost. In Python 3.3 and later, yield from was introduced to solve this and other issues. The equivalent using yield from is:

def outer():
    yield from inner()
    yield "extra"

Both versions yield the same items from inner(), but yield from also captures the return value and propagates exceptions more naturally. Understanding these differences is the core of the python yield from vs for loop decision.

Basic Syntax and Behavior

The yield from expression is a single statement that delegates iteration to a subgenerator. It yields every value from the subgenerator and then returns the subgenerator's return value as the value of the yield from expression. A for loop, on the other hand, simply iterates and yields each item, but it cannot access the subgenerator's return value.

Consider this simple subgenerator:

def subgen():
    yield 1
    yield 2
    return "done"

Using a for loop:

def outer_for():
    for item in subgen():
        yield item

The outer_for generator yields 1 and 2, but the "done" from subgen is silently discarded. If you try to capture it, you'd have to rewrite subgen to communicate the result differently, for example by yielding a sentinel value.

Using yield from:

def outer_yield_from():
    result = yield from subgen()
    print(f"subgen returned {result}")
    yield "after"

Now result holds "done". The yield from expression evaluates to the return value of the subgenerator. This is a significant difference when the subgenerator needs to produce a final result.

Return Values: What yield from Gives You

The most practical reason to choose yield from over a for loop is the ability to receive the return value of the subgenerator. In Python, a generator can return a value, but that value is only accessible through yield from (or by manually catching StopIteration and reading its value attribute). A for loop does not expose that return value.

Here's a concrete example where the return value matters:

def read_chunks():
    total = 0
    for chunk in data_source():
        yield chunk
        total += len(chunk)
    return total

def process():
    total = yield from read_chunks()
    print(f"Processed {total} bytes")

If you used a for loop, you'd have to restructure read_chunks to yield the total as a final item, which is awkward and can be mistaken for actual data. yield from keeps the protocol clean: data items are yielded, and the result is returned separately.

Exception Propagation: How Errors Flow

When a subgenerator raises an exception, the behavior differs between the two approaches. With a for loop, the exception is raised in the outer generator at the point of the yield inside the loop, and it propagates normally. With yield from, the exception is thrown into the subgenerator at the point where it yielded its last value, allowing the subgenerator to catch it and potentially handle it.

Consider a subgenerator that can handle a ValueError:

def subgen_with_exception():
    try:
        yield 1
        yield 2
    except ValueError:
        yield "handled"

If you delegate with a for loop, you cannot send an exception into the subgenerator; you can only catch it in the outer generator. With yield from, you can use the .throw() method on the outer generator, and the exception will be injected into the subgenerator at its current yield point. This is essential for coroutine-like patterns where the subgenerator needs to respond to external signals.

Example:

def outer():
    yield from subgen_with_exception()

gen = outer()
next(gen)  # yields 1
next(gen)  # yields 2
# Now throw an exception into the generator
gen.throw(ValueError)  # yields "handled"

With a for loop, you would have to manually forward .throw() and .send() calls, which is error-prone. yield from handles this transparently, making it the preferred choice for building complex generator pipelines.

Performance and Overhead Considerations

From a performance standpoint, yield from can be slightly faster than an explicit for loop because it avoids an extra Python-level iteration and yield per item. However, the difference is usually negligible for most applications. The real performance benefit is in the clarity and correctness of the code, not raw speed.

If you are micro-optimizing a tight loop, you might see a small improvement with yield from because it reduces the number of bytecode instructions. But do not let micro-benchmarks drive the decision. The more significant cost is often the overhead of generator creation and resumption, which is identical in both approaches.

For large data streams, the memory behavior is the same: both are lazy and produce items one at a time. The choice between yield from and a for loop should be based on the need for return values and exception handling, not on performance.

When to Use yield from vs a for Loop

Use yield from when:

  • You need to capture the return value of the subgenerator.
  • You are building a coroutine or need to forward .send() and .throw() calls to a subgenerator.
  • You want to delegate iteration without writing boilerplate that manually forwards these methods.
  • You are composing generators and want the code to be concise and readable.

Use a for loop when:

  • You don't need the subgenerator's return value.
  • You need to transform each item before yielding it (e.g., yield item * 2).
  • You are targeting Python versions before 3.3 (though this is rarely a concern today).
  • You want to add additional logic around the iteration, such as counting items or handling exceptions in the outer generator only.

There is no rule that says you must use one over the other; the decision depends on the specific requirements of your code. In modern Python, yield from is the idiomatic way to delegate to a subgenerator, and it is often the better choice for readability.

Edge Cases and Common Pitfalls

One common pitfall is assuming that yield from works like a simple loop when the subgenerator is infinite. Both approaches will run indefinitely if the subgenerator never ends, so that is not a differentiator. Another pitfall is forgetting that yield from can only be used inside a generator function, not in a regular function. Attempting to use it elsewhere raises a SyntaxError.

Another subtlety is that yield from accepts any iterable, not just generators. For example, yield from [1, 2, 3] works and yields the list elements. A for loop also works with any iterable, but yield from is more concise when you simply want to yield all items from an iterable without additional logic.

When a subgenerator returns a value, that value is only available after the subgenerator has exhausted. If you try to access it before the yield from completes, you'll get a StopIteration exception. This is expected behavior, but it can be confusing if you are not aware of the protocol.

Finally, be careful when mixing yield from with explicit return in the outer generator. The return value of the outer generator is separate from the value of the yield from expression. If you write return yield from subgen(), the outer generator returns the subgenerator's result, which is legal but may be surprising. In most cases, you'll want to capture the result in a variable and then use it.

A Practical Example: Chaining Generators

To bring everything together, consider a scenario where you have a hierarchy of generators that need to pass data and results upward. yield from makes this clean:

def read_lines(file_path):
    with open(file_path) as f:
        for line in f:
            yield line.strip()
    return "eof"

def process_lines(file_path):
    result = yield from read_lines(file_path)
    print(f"Reached {result}")
    yield "done"

for item in process_lines("data.txt"):
    print(item)

The process_lines generator yields each line, and after the file is exhausted, it prints the return value from read_lines. This pattern is difficult to replicate with a for loop without extra state variables. The yield from expression makes the delegation explicit and the code easier to follow.

When you are deciding between python yield from vs for loop, think about whether you need the subgenerator's return value, whether you need to forward send and throw, and whether the added clarity of yield from is worth the slightly more advanced syntax. For most generator composition tasks, yield from is the right tool.

python yield from vs for loop: Generator Delegation | RYUSLOG DEV