Understanding the Python Iterator Protocol

The Python iterator protocol is the contract that Python's for loop, iter(), next(), and many built-in functions rely on to consume sequences of values. An object is an iterator if it implements two methods: __iter__() and __next__(). The __iter__ method returns the iterator itself, and __next__ returns the next value or raises StopIteration when exhausted.

Every time you write a for loop, Python calls iter() on the iterable to obtain an iterator, then repeatedly calls next() on that iterator until StopIteration is raised. Understanding this protocol lets you create custom iterators that behave like built-in ones.

The Two Methods: iter and next

The protocol is minimal. An iterator must have:

• __iter__(self): returns the iterator object itself. This is what makes an iterator also an iterable.
• __next__(self): returns the next item, or raises StopIteration if there are no more items.

Here's a simple iterator that yields numbers from 0 up to a limit:

class CountUp:
    def __init__(self, limit):
        self.limit = limit
        self.current = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.limit:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

You can use it directly:

for num in CountUp(3):
    print(num)

This prints 0, 1, 2. The for loop handles the StopIteration exception internally.

Using iter() and next() Explicitly

The iter() function calls __iter__ on an object, and next() calls __next__. You can drive an iterator manually:

counter = CountUp(2)
it = iter(counter)
print(next(it))  # 0
print(next(it))  # 1
print(next(it))  # raises StopIteration

This is useful when you need fine-grained control over iteration, such as in a parser or when combining multiple iterators.

Iterator vs Iterable: A Common Confusion

An iterable is any object that can be passed to iter() to produce an iterator. Lists, tuples, strings, and dictionaries are iterables. An iterator is an object that produces values on demand and is exhausted after one pass.

Many objects are both iterable and iterators (like our CountUp), but they don't have to be. For example, a list is iterable but not an iterator: calling iter() on a list returns a new iterator object. This is why you can iterate over the same list multiple times, but not over an iterator.

Generators: The Practical Shortcut

Writing a full iterator class is often overkill. Generators are functions that use yield and automatically implement the iterator protocol. The same CountUp can be written as:

def count_up(limit):
    current = 0
    while current < limit:
        yield current
        current += 1

Generator functions return a generator object, which is an iterator. They are lazy: values are produced only when requested. Generator expressions, like (x*x for x in range(10)), work similarly.

Generators are the idiomatic way to create iterators in Python because they reduce boilerplate and make the code more readable.

Common Pitfalls and Edge Cases

One frequent mistake is forgetting to raise StopIteration when the iterator is exhausted. If you return None instead, the loop will continue indefinitely or behave unexpectedly.

Another issue is reusability. An iterator is single-use. Once exhausted, it won't produce values again. If you need to iterate multiple times, you must create a fresh iterator each time. For a class, that means returning a new instance from __iter__ if the object is meant to be iterable but not an iterator.

Infinite iterators are valid but require care. They never raise StopIteration, so a for loop over them will run forever unless you break out explicitly. Use itertools.islice or a manual counter to limit consumption.

Performance and Memory Considerations

The iterator protocol enables lazy evaluation. Values are generated on demand, which can drastically reduce memory usage when working with large or infinite sequences. For example, reading a file line by line with a generator avoids loading the entire file into memory.

However, there is a small per-iteration overhead compared to a simple list comprehension, because each next() call involves a function call and state update. For most applications this is negligible, but in tight loops with millions of iterations, a generator may be slower than a precomputed list. Profile before optimizing.

When to Use a Custom Iterator vs a Generator

Custom iterator classes are useful when you need to maintain complex state that doesn't fit naturally in a generator, or when you want to expose additional methods alongside iteration. Generators are simpler and usually sufficient.

If you need to support multiple independent iterations over the same data, a class that returns a fresh iterator from __iter__ is the right choice. If you only need a one-pass sequence, a generator is cleaner.

In practice, start with a generator. Move to a custom iterator only when you have a concrete reason, such as needing to reset the iteration state or provide extra functionality.