Python Generator vs Iterator: Key Differences

When you write a loop in Python, you are often iterating over an object without thinking about what makes that object iterable. The distinction between a generator and an iterator is a common source of confusion, yet it directly affects memory usage, code structure, and how you control iteration. This article clarifies the relationship and the practical tradeoffs between Python generator vs iterator.

The Iterator Protocol in Python

An iterator is any object that implements the iterator protocol, which consists of two methods: __iter__ and __next__. The __iter__ method returns the iterator object itself, and __next__ returns the next value from the collection, raising StopIteration when no more items are available. This protocol is what makes objects usable in a for loop.

Here is a minimal custom iterator that produces a sequence of squares:

class SquareIterator:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.limit:
            raise StopIteration
        result = self.current ** 2
        self.current += 1
        return result

for square in SquareIterator(5):
    print(square)

This works, but it requires boilerplate: maintaining state, updating it, and raising the termination exception. The protocol itself is simple, but writing a new iterator class for every sequence is tedious.

How Generators Implement the Protocol

A generator is a function that contains at least one yield statement. When called, it returns a generator object without executing the function body. Each call to next() on that object runs the function until the next yield, pauses, and resumes later. The generator automatically implements __iter__ and __next__, so it is an iterator.

The same square sequence can be written as a generator function:

def square_generator(limit):
    current = 0
    while current < limit:
        yield current ** 2
        current += 1

for square in square_generator(5):
    print(square)

The generator function is shorter and keeps the state in local variables. The yield expression handles the pause and resume mechanics, and StopIteration is raised automatically when the function returns.

Generator vs Iterator: Core Differences

Every generator is an iterator, but not every iterator is a generator. The key difference is that a generator is defined by a function with yield, while a custom iterator is a class that explicitly implements the protocol. This leads to several practical distinctions.

Generators also support send(), throw(), and close() methods, which are part of the generator protocol. Custom iterators do not have these by default. This makes generators more powerful for coroutine-style programming and cooperative multitasking.

When to Use a Generator Instead of a Custom Iterator

In most cases, a generator is the better choice. It is less code, easier to read, and less prone to off-by-one errors in state management. Use a custom iterator when you need to expose additional methods or maintain complex state that is clearer as a class.

For example, if you need an iterator that can be reset, a custom class allows you to add a reset() method. A generator cannot be reset; you would need to call the generator function again to create a fresh iterator.

class ResettableIterator:
    def __init__(self, data):
        self.data = data
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.index >= len(self.data):
            raise StopIteration
        value = self.data[self.index]
        self.index += 1
        return value

    def reset(self):
        self.index = 0

This kind of reusability is not possible with a generator without recreating it. If you only need a one-pass sequence, a generator is simpler.

Memory and Performance Considerations

Generators are lazy: they produce values on demand and do not store the entire sequence in memory. This is the primary reason to prefer a generator when working with large datasets or infinite sequences. A list comprehension creates a full list, while a generator expression creates a lazy iterator.

# List comprehension: builds a list of 10 million squares
squares_list = [x ** 2 for x in range(10_000_000)]

# Generator expression: produces values one at a time
squares_gen = (x ** 2 for x in range(10_000_000))

The list consumes memory proportional to the number of items, while the generator uses a constant amount of memory regardless of the sequence length. This does not mean generators are always faster. There is a small overhead per next() call, and for small sequences a list may be faster because it avoids the generator machinery. The tradeoff is memory versus per-item overhead.

When processing a file line by line, a generator is the standard approach because reading the entire file into memory is often infeasible. The same applies to streaming data or recursive algorithms that would otherwise exhaust memory.

Common Pitfalls with Generators and Iterators

One common mistake is assuming a generator can be reused. Once a generator is exhausted, it stays exhausted. If you need to iterate over the same sequence twice, you must create a new generator object. Custom iterators can be designed to be reusable by resetting their state, but that is not automatic.

Another pitfall is confusing an iterable with an iterator. An iterable has an __iter__ method that returns an iterator. A list is iterable, but it is not an iterator. When you call iter(list), you get a list iterator. A generator is both iterable and an iterator because it returns itself from __iter__.

my_list = [1, 2, 3]
print(iter(my_list) is my_list)  # False

gen = (x for x in my_list)
print(iter(gen) is gen)          # True

Understanding this distinction prevents errors when passing objects to functions that expect an iterator, such as next() or zip().

Choosing the Right Approach for Your Code

The decision between a generator and a custom iterator comes down to the complexity of the state and the need for extra methods. For most iteration tasks, a generator is the idiomatic Python solution. It is concise, lazy, and automatically conforms to the iterator protocol. Custom iterators are justified when you need a reusable, stateful object with additional behavior beyond simple iteration.

Consider a scenario where you are building a paginated API client. The page cursor is stateful, and you might want to expose a method to jump to a specific page. A custom iterator class gives you the flexibility to add such methods while still being usable in a for loop. On the other hand, if you are simply transforming a stream of numbers or lines, a generator function is the cleaner choice.

The practical takeaway is to start with a generator and only reach for a custom iterator when the generator's limitations become a real constraint. This keeps your code readable and maintainable while leveraging Python's lazy evaluation model effectively.