Python Generator Comprehension: Syntax and Use

A python generator comprehension, also called a generator expression, produces a lazy iterable in one line of syntax. The only syntactic difference from a list comprehension is the use of parentheses instead of brackets:

squares = (x * x for x in range(10))

This creates a generator object, not a list. No multiplication is performed until you actually iterate over the generator.

Generator Comprehension Syntax and Lazy Evaluation

The comprehension syntax follows the same pattern as a list comprehension: an expression, a for clause, and optional if filters.

evens = (n for n in range(100) if n % 2 == 0)

The critical behavior is that nothing is computed at creation time. The generator holds only the current iteration state: the current value of n and a reference to the underlying iterator. Each call to next() advances the state and produces the next value.

gen = (x * x for x in range(3))
print(next(gen))  # 0
print(next(gen))  # 1
print(next(gen))  # 4

After the third next(), the generator is exhausted. A fourth call raises StopIteration.

How Generator Comprehensions Differ From List Comprehensions

A list comprehension eagerly builds the entire sequence in memory:

squares_list = [x * x for x in range(100_000)]

A generator comprehension defers all computation:

squares_gen = (x * x for x in range(100_000))

The list holds 100,000 integer objects immediately. The generator holds a single integer and a reference to the range iterator. The memory difference grows with the size of the sequence.

There is also a behavioral difference beyond memory. A list is re-iterable and supports indexing and len(). A generator is single-use: once you iterate over it, it is empty. You cannot index into a generator or ask for its length.

Memory Behavior of Generator Comprehensions

The memory advantage becomes obvious when processing data that does not fit comfortably in memory. Reading a large log file line by line is a typical case:

lines = (line.strip() for line in open("app.log"))

The generator reads one line at a time as it is consumed. A list comprehension over the same file would read the entire file into memory before any processing begins.

The same principle applies to range with very large bounds, streaming data from a socket, or any iterator that produces values incrementally. The generator comprehension does not change how the underlying iterator produces data; it only ensures that no intermediate collection is built.

Practical Patterns: Passing Generators to Functions

Built-in functions such as sum, min, max, any, and all accept any iterable, so a generator comprehension can be passed directly:

total = sum(x * x for x in range(1000))

When the generator is the only argument to a function, the outer parentheses can be omitted. This is a common idiom and is equivalent to writing sum((x * x for x in range(1000))).

The same pattern works with itertools functions and with any function that consumes an iterable once. For example, itertools.islice can limit the number of values pulled from a generator:

from itertools import islice

gen = (x * x for x in range(1000))
first_five = list(islice(gen, 5))

This consumes only the first five values from the generator, leaving the rest untouched.

Performance Considerations and Runtime Cost

The per-item computation cost of a generator comprehension is the same as a list comprehension. The difference is in allocation and iteration overhead.

A list comprehension allocates a full list and stores every result. A generator comprehension avoids that allocation entirely. For large sequences, this reduces memory pressure and can prevent the interpreter from spending time on garbage collection of large temporary lists.

The tradeoff is that iterating a generator has slightly higher per-item overhead than iterating a list, because each next() call involves a frame resume. For small sequences, a list comprehension is often marginally faster. For large sequences, the memory savings dominate and the generator is usually the better choice.

The decision should be based on the size of the data and whether you need random access. If you need to index into the results, iterate multiple times, or know the length, a list is required. If you only need to consume the values once, a generator comprehension avoids the intermediate allocation.

Common Mistakes and When a List Is the Better Choice

A frequent mistake is treating a generator like a sequence. Indexing and len() do not work:

gen = (x for x in range(10))
print(gen[0])  # TypeError: 'generator' object is not subscriptable

Another mistake is reusing a generator. After it is exhausted, it produces nothing:

gen = (x * x for x in range(3))
print(list(gen))  # [0, 1, 4]
print(list(gen))  # []

If you need to iterate over the same values more than once, materialize the results into a list first.

A list comprehension is the right choice when the sequence is small, when you need repeated iteration, or when you need to access elements by index. A generator comprehension is the right choice when the sequence is large or unbounded, when you only need to consume the values once, and when you want to avoid building an intermediate collection.

Chaining Generator Comprehensions

Generator comprehensions compose into lazy pipelines without intermediate lists:

numbers = (int(line) for line in open("data.txt"))
positive = (n for n in numbers if n > 0)
squares = (n * n for n in positive)

Each step is a generator that pulls from the previous one. Nothing is computed until the final generator is consumed, for example by a for loop or a function like sum.

This pattern is useful for streaming data processing, where the full dataset is too large to hold in memory. Each generator adds a transformation without copying data. The cost is that debugging such pipelines is harder, because the state is spread across multiple generator frames. If a transformation fails, the traceback points to the consuming code, not to the generator that produced the bad value.