Understanding the Python yield Keyword

The python yield keyword changes how a function behaves at runtime. When a function contains yield, it no longer executes like a normal function that runs to completion and returns a single value. Instead, it becomes a generator function: calling it returns a generator object that produces values one at a time, pausing after each yield and resuming when the next value is requested.

What Happens When a Function Contains yield

Consider a simple function that returns a list of squares:

def squares_list(n):
    result = []
    for i in range(n):
        result.append(i * i)
    return result

If you replace return with yield, the function becomes a generator:

def squares_gen(n):
    for i in range(n):
        yield i * i

Calling squares_gen(5) does not execute the loop. It returns a generator object. The loop only runs when you iterate over that object, and it pauses at each yield until the next value is requested. This is the core behavioral difference: a normal function builds the entire result before returning, while a generator produces values on demand.

How Generators Execute Lazily

Generator execution is lazy. When you call next() on a generator, execution runs from the previous yield point to the next one, then pauses again. This is visible in the following example:

def countdown():
    print("start")
    yield 3
    print("after 3")
    yield 2
    print("after 2")
    yield 1

gen = countdown()
print(next(gen))  # prints "start", then 3
print(next(gen))  # prints "after 3", then 2
print(next(gen))  # prints "after 2", then 1

The first next(gen) runs the function until the first yield, returning 3. The second call resumes right after that yield, prints after 3, and returns 2. This pause-and-resume behavior is what makes generators useful for streams of data that would be expensive or impossible to hold in memory all at once.

yield vs return: Key Differences

yield and return serve different purposes inside a function. A return terminates the function and optionally provides a result. A yield suspends the function, preserving its local state, and allows it to be resumed later. A generator function can have multiple yield statements, but only one return (which normally signals the end of iteration).

The table below summarizes the main contrasts:

When you need a sequence of values, yield avoids constructing an intermediate list. That is especially important when the sequence is large or infinite.

Practical Example: Streaming Large Files

A common use case for generators is processing files that are too large to read entirely into memory. Reading a file line by line with yield keeps only one line in memory at a time:

def read_large_file(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip()

for line in read_large_file("huge_log.txt"):
    process(line)

Without a generator, you might write return f.readlines(), which loads every line into a list. For a multi-gigabyte file, that can exhaust available memory. The generator version processes the file incrementally, which is both memory-efficient and often faster because it avoids the overhead of growing a large list.

Using yield from to Delegate Generation

The yield from expression lets a generator delegate part of its work to another iterator. This simplifies generator code that would otherwise require nested loops.

def flatten(nested):
    for sublist in nested:
        yield from sublist

pairs = [[1, 2], [3, 4]]
print(list(flatten(pairs)))  # [1, 2, 3, 4]

yield from sublist iterates over sublist and yields each item to the caller. This is equivalent to writing an inner loop, but it reads more clearly and handles the iterator protocol correctly, including propagating exceptions and return values from delegated generators.

Memory and Performance Considerations

Generators reduce memory usage by not materializing the entire sequence. The tradeoff is that accessing elements requires sequential iteration; you cannot index into a generator or know its length without consuming it. This is a deliberate design choice: generators trade random access for lower memory footprint.

Performance-wise, generators add a small per-item overhead due to the pause-and-resume mechanism. For most I/O-bound or streaming workloads, this overhead is negligible compared to the cost of reading from disk or a network. For CPU-bound loops that fit in memory, a list comprehension may be faster because it avoids generator machinery. The right choice depends on whether you need all values at once or can process them incrementally.

When Not to Use yield

Generators are not always the best fit. If you need to access elements multiple times, a generator must be recreated or converted to a list. If you need random access by index, a list or tuple is more appropriate. If the sequence is small and already known, a list comprehension is simpler and more readable.

Consider this scenario: you need to compute the median of a set of numbers. The median requires the full dataset, so a generator that yields values one by one forces you to store them anyway. In that case, building a list directly is clearer. Similarly, if you need to sort the data, you must materialize it. Generators shine when the consumer processes each value exactly once and then discards it.

Another limitation is that generators are single-use. Once exhausted, they cannot be reset. If you need to iterate over the same data multiple times, you must either create a new generator or store the results. Understanding these boundaries helps you decide when yield is the right tool and when a regular collection is more practical.