Python Iterator vs Generator: Key Differences

When developers compare python iterator vs generator, the distinction often comes down to how each implements lazy iteration. Both produce sequences of values on demand, but they differ in syntax, state handling, and the amount of code required. This article explains the practical differences and the conditions that should drive your choice.

The Iterator Protocol: What Makes an Object Iterable

An iterator in Python is any object that implements the iterator protocol, which consists of two methods: __iter__() and __next__(). The __iter__() method returns the iterator object itself, and __next__() returns the next value in the sequence. When no more values are available, __next__() raises StopIteration.

class Counter:
    def __init__(self, limit):
        self.limit = limit
        self.value = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.value >= self.limit:
            raise StopIteration
        current = self.value
        self.value += 1
        return current

for number in Counter(3):
    print(number)

This custom iterator keeps its state in instance attributes. Each call to next() advances the internal counter until the limit is reached. The for loop implicitly calls iter() on the object and then repeatedly calls next() until StopIteration is raised.

Generator Functions: Iteration Built on yield

A generator function is a function that uses the yield keyword instead of return. When called, it returns a generator object without executing the function body immediately. Execution starts on the first call to next() and pauses at each yield, preserving the local state.

def counter(limit):
    value = 0
    while value < limit:
        yield value
        value += 1

for number in counter(3):
    print(number)

The generator above produces the same sequence as the custom iterator, but the code is shorter and the state is stored in local variables rather than instance attributes. The generator object itself is an iterator, so it implements __iter__() and __next__() automatically.

Key Differences Between an Iterator and a Generator

The most direct way to understand the relationship is to compare them side by side.

A generator is a convenient way to create an iterator, but not every iterator is a generator. If you need complex state transitions or multiple methods beyond iteration, a custom iterator class gives you more control. If you only need to produce a sequence of values, a generator is usually simpler.

Lazy Evaluation and Memory Behavior

Both iterators and generators are lazy: they produce one value at a time and do not store the entire sequence in memory. This is the primary reason to use either approach when working with large datasets.

def read_lines(file_path):
    with open(file_path) as file:
        for line in file:
            yield line.strip()

This generator reads a file line by line, so the memory footprint stays constant regardless of file size. A custom iterator could achieve the same behavior, but the generator version is more readable and less error-prone because the with block manages the file lifecycle.

A generator expression offers the same laziness in a compact form:

squares = (x * x for x in range(1000000))

This does not create a list of a million squares. It creates a generator that computes each square only when requested. The equivalent list comprehension would allocate memory for all values at once.

When to Write a Custom Iterator Instead of a Generator

Generators are not always the best fit. A custom iterator is preferable when you need to implement additional methods or maintain complex internal state that is easier to express as instance attributes. For example, an iterator that supports resetting its position or exposing metadata about the iteration may be clearer as a class.

class ResettableCounter:
    def __init__(self, limit):
        self.limit = limit
        self.value = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.value >= self.limit:
            raise StopIteration
        current = self.value
        self.value += 1
        return current

    def reset(self):
        self.value = 0

Here the reset() method is part of the iterator's public interface. A generator cannot expose extra methods without wrapping it in a class, which defeats the purpose. If you need to pass the iterator to code that expects the full iterator protocol, a custom class is also more explicit.

Common Pitfalls With Iterators and Generators

One frequent mistake is assuming that an iterator can be reused. Both iterators and generators are single-use. Once exhausted, calling next() raises StopIteration permanently. If you need to iterate multiple times, you must create a new iterator or generator.

gen = (x for x in range(3))
print(list(gen))  # [0, 1, 2]
print(list(gen))  # []

Another pitfall is mixing up iterables and iterators. A list is iterable but not an iterator; it does not have a __next__() method. Calling next() on a list raises TypeError. You must call iter() first to obtain an iterator.

Generators also have a subtle behavior: they are single-pass and cannot be indexed. If you need random access to a sequence, convert it to a list first, but be aware that this defeats the memory advantage.

Performance and Maintainability Considerations

Generators generally have lower overhead than a custom iterator class because they avoid the boilerplate of defining a class and managing instance attributes. The interpreter handles state suspension and resumption internally, which is both faster and less error-prone in typical use.

However, a custom iterator can be more performant in specific cases where you need fine-grained control over the iteration logic, such as skipping values based on external conditions or maintaining multiple counters. The performance difference is usually negligible unless the iteration is extremely tight and called millions of times.

From a maintainability perspective, generators are often easier to read because they express the sequence generation as a linear flow of yield statements. Custom iterators are useful when the iteration logic is complex enough that splitting it into methods improves clarity. Choose the approach that makes the code's intent most obvious, and reserve custom iterators for cases where generators cannot express the required behavior.