python generator: Practical Usage and Code Examples

When you need to process a large sequence of values, building a full list in memory can become a bottleneck. A Python generator solves this by producing values one at a time using the yield keyword. Instead of allocating a container with all results, a generator function returns an iterator that computes each value on demand. That single difference changes how you handle streams, pipelines, and infinite sequences.

The Problem Generators Solve

Consider a function that returns the first n square numbers:

def squares(n):
    result = []
    for i in range(n):
        result.append(i * i)
    return result

For a small n this is fine. For n = 10_000_000, the list holds ten million integers, consuming hundreds of megabytes. If the caller only needs to iterate once, that memory is wasted. A generator avoids the list entirely:

def squares_gen(n):
    for i in range(n):
        yield i * i

Calling squares_gen(10_000_000) returns a generator object immediately. No values are computed until you iterate. This is the core advantage: lazy evaluation.

The yield Keyword and Generator Functions

A function that contains yield is a generator function. When called, it does not execute the body. Instead, it returns a generator object that implements the iterator protocol. Each call to next() runs the function until the next yield and pauses there.

gen = squares_gen(3)
print(next(gen))  # 0
print(next(gen))  # 1
print(next(gen))  # 4

After the last value, the generator raises StopIteration. The for loop handles this automatically:

for value in squares_gen(3):
    print(value)

The state of the function—local variables, the current position in the loop—is preserved between next() calls. This is what makes generators suitable for stateful iteration without writing a separate iterator class.

How Lazy Evaluation Works

Lazy evaluation means values are produced only when requested. In a generator, the computation happens during iteration, not before. This has two practical effects.

First, you can represent infinite sequences without exhausting memory. For example, a generator that yields Fibonacci numbers forever:

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

The caller decides when to stop:

for value in fibonacci():
    if value > 1000:
        break
    print(value)

Second, you can chain operations without intermediate lists. Each stage consumes from the previous generator, so the total memory footprint stays constant regardless of the number of stages.

Generator Expressions for Simple Pipelines

Generator expressions provide a concise syntax for simple transformations. They look like list comprehensions but use parentheses:

squares = (x * x for x in range(10))

This creates a generator, not a list. You can pass it directly to functions that accept iterables, such as sum() or max():

total = sum(x * x for x in range(10))

The expression is evaluated lazily, so the underlying sequence is not materialized. This is useful when you need to transform data from a file or a network stream without loading it all into memory.

Memory and Runtime Behavior

The primary benefit of generators is memory efficiency. A list stores every element; a generator stores only the current state. The difference becomes significant when dealing with large datasets or unbounded streams.

There is also a runtime tradeoff. Because each value is produced on demand, there is a small overhead per next() call compared to indexing a list. For tight loops over small collections, a list may be faster. For large or infinite sequences, the memory savings usually outweigh the per-item cost. The right choice depends on whether you need random access and whether the entire sequence fits in memory.

Common Pitfalls and How to Avoid Them

Generators are single-use. Once exhausted, they cannot be reused. If you need to iterate twice, either create a new generator or convert it to a list when the data is small.

Another pitfall is mixing generator expressions with functions that expect a list. For example, list(generator) materializes the values. Be explicit about when you need a concrete collection.

Finally, be careful with yield inside a function that also returns a value. A return statement in a generator raises StopIteration and can carry a value in Python 3, but that value is not part of the iteration. It is only accessible through the StopIteration exception, which is rarely what you want.

When to Use a Generator vs a List

Use a generator when:

You only need to iterate once.
The sequence is large or infinite.
You want to chain transformations without intermediate storage.
You are reading data from a stream or file.

Use a list when:

You need random access by index.
You need to iterate multiple times.
The collection is small enough that memory is not a concern.
You need to modify the elements after creation.

A practical rule: if you can avoid building a list, do so. But do not force a generator into a situation where a list is clearer and the memory difference is negligible.

Advanced: Sending Values into a Generator

Generators are not limited to producing values. The send() method allows you to pass a value back into the generator, which becomes the result of the yield expression. This enables two-way communication.

def accumulator():
    total = 0
    while True:
        value = yield total
        if value is None:
            continue
        total += value

acc = accumulator()
next(acc)  # start the generator
print(acc.send(10))  # 10
print(acc.send(5))   # 15

This pattern is useful for building coroutines and state machines. It is an advanced feature, but it shows that a Python generator is more than a simple iterator—it is a resumable function with state.