RDRYUSLOG DEV
Back to Blog
Python

Python Iterator Protocol and Custom Iterators

python iterator: Learn how Python iterators work, how to implement custom iterators, and when to use them for memory-efficient lazy evaluation.

PythonIteratorsIterablesGeneratorsLazy EvaluationMemory Efficiency
Python iterator protocol illustration showing a sequence of items being consumed one by one from an iterable container.

Python iterators are objects that implement the iterator protocol, which consists of two methods: __iter__() and __next__(). The __iter__ method returns the iterator object itself, and __next__ returns the next item in the sequence, raising StopIteration when exhausted. This protocol is the foundation of Python's for loop and many built-in functions like sum, list, and map. Understanding the python iterator protocol is essential for writing memory-efficient code that processes large or infinite sequences without loading everything into memory at once.

The Iterator Protocol

The iterator protocol is defined by two methods:

  • __iter__(self) – returns the iterator object itself. This method is what makes an object an iterable when called on a container, but for an iterator, it simply returns self.
  • __next__(self) – returns the next item from the sequence. If there are no more items, it raises StopIteration.

When you use a for loop, Python calls iter() on the object to obtain an iterator, then repeatedly calls next() on that iterator until StopIteration is raised. This is true for lists, tuples, dictionaries, sets, and strings, all of which return a new iterator each time iter() is called.

numbers = [1, 2, 3]
iterator = iter(numbers)
print(next(iterator))  # 1
print(next(iterator))  # 2
print(next(iterator))  # 3
# next(iterator) would raise StopIteration

The protocol is deliberately minimal. Any object that implements these two methods correctly can be used as an iterator, which means it can be consumed by a for loop, passed to functions that expect an iterable, and used in comprehensions.

Iterables vs. Iterators

A common source of confusion is the difference between an iterable and an iterator. An iterable is any object that can return an iterator via iter(). Lists, tuples, strings, and dictionaries are iterables. An iterator, on the other hand, is an object that keeps state and produces the next value on demand. Every iterator is also iterable because it implements __iter__ returning itself, but not every iterable is an iterator.

For example, a list is iterable but not an iterator. Calling iter() on a list returns a new list iterator. The list itself does not implement __next__. This distinction matters when you need to reuse a sequence. An iterator is one-shot: once you consume it, it is exhausted. An iterable can be converted to a fresh iterator each time.

my_list = [1, 2, 3]
list_iter = iter(my_list)
print(list_iter is my_list)  # False

This one-shot behavior is intentional. It allows iterators to represent streams of data that cannot be rewound, such as lines from a file or network packets. If you need to iterate over the same data multiple times, you must keep the underlying iterable and create new iterators from it.

Building a Custom Iterator

To create a custom iterator, define a class that implements __iter__ and __next__. The __iter__ method should return self, and __next__ should return the next value or raise StopIteration. Here is an iterator that yields the Fibonacci sequence up to a maximum value:

class FibonacciIterator:
    def __init__(self, max_value):
        self.max_value = max_value
        self.a, self.b = 0, 1

    def __iter__(self):
        return self

    def __next__(self):
        if self.a > self.max_value:
            raise StopIteration
        result = self.a
        self.a, self.b = self.b, self.a + self.b
        return result

You can use this iterator directly:

for num in FibonacciIterator(100):
    print(num, end=' ')
# 0 1 1 2 3 5 8 13 21 34 55 89

The class maintains state between calls to __next__. The StopIteration exception signals the end of the sequence. This pattern gives you full control over the iteration logic, which is useful when the sequence is not naturally representable as a list or when you want to avoid materializing it in memory.

Using Generators as Iterators

Generators are a concise way to create iterators using the yield keyword. A generator function returns a generator object, which is an iterator. The state of the function is preserved between calls, and yield produces the next value. The Fibonacci iterator from above can be written as a generator:

def fibonacci_generator(max_value):
    a, b = 0, 1
    while a <= max_value:
        yield a
        a, b = b, a + b

Usage is identical:

for num in fibonacci_generator(100):
    print(num, end=' ')

Generators are usually preferred over custom iterator classes because they are less boilerplate and easier to read. They also support generator expressions, which are similar to list comprehensions but produce items lazily:

squares = (x * x for x in range(10))

This generator expression does not create a list; it produces each square on demand. When you need a simple sequence, a generator is often the right tool. A custom iterator class is justified when you need to encapsulate more complex state or provide additional methods beyond __iter__ and __next__.

When to Use a Custom Iterator

Custom iterators and generators shine when dealing with large or infinite data streams. Reading a file line by line, processing network packets, or generating permutations are typical cases where you do not want to store the entire sequence in memory. The iterator protocol allows you to process one item at a time, reducing memory usage and enabling pipelines that would otherwise be impossible.

Consider a scenario where you need to read a multi-gigabyte log file and extract lines containing a specific error code. A generator that yields matching lines avoids loading the whole file into memory:

def filter_log(file_path, error_code):
    with open(file_path) as f:
        for line in f:
            if error_code in line:
                yield line.strip()

This function returns an iterator that reads the file lazily. The file is opened and closed properly, and each line is processed as the iterator is consumed. If you used a list comprehension instead, you would allocate a list of all matching lines, which could exhaust available memory.

Custom iterators are also useful when you need to implement a sequence that is not naturally indexable, such as a tree traversal or a combination generator. By implementing the protocol, you make your object compatible with Python's iteration tools, including itertools functions, map, filter, and zip.

Performance and Memory Considerations

Iterators provide a significant memory advantage over containers because they do not store all elements at once. The memory footprint of an iterator is typically constant, regardless of how many items it yields. This is crucial for large datasets. However, there is a tradeoff: iterators may have slightly higher per-item CPU overhead due to the function call or method invocation for each next(). In practice, this overhead is negligible for I/O-bound or CPU-bound loops, but it can matter in tight numerical loops.

When performance is critical, you should measure whether a list or a generator is faster for your specific workload. Lists have faster iteration because they are backed by a contiguous array and Python's internal iteration is optimized. Generators and custom iterators involve an extra layer of state management. For small datasets, the difference is usually irrelevant; for huge datasets, the memory savings far outweigh the CPU cost.

Another consideration is that iterators are single-pass. Once you consume an iterator, it is exhausted. If you need to iterate multiple times, you must recreate the iterator or keep the underlying iterable. This can affect algorithm design. For example, if you need to perform two passes over a stream of data, you cannot use the same iterator twice. You either buffer the results or re-read the source.

Common Pitfalls with Iterators

One common mistake is assuming an iterator can be reused. After StopIteration is raised, the iterator is permanently exhausted. Calling next() again will raise StopIteration again. If you need to iterate again, you must obtain a new iterator from the original iterable.

Another pitfall is confusing iterators with iterables when passing arguments to functions. Some functions, like sum, consume the iterator immediately. If you pass an iterator to a function and then try to iterate over it later, you will get an empty sequence. For example:

it = iter([1, 2, 3])
total = sum(it)  # consumes it
print(list(it))  # []

This behavior is often surprising. To avoid it, be aware of which functions consume iterators and whether you need the data afterward.

Finally, when implementing __next__, ensure you raise StopIteration correctly. If you accidentally return None instead of raising, the loop will continue indefinitely or produce unexpected values. Also, remember that __iter__ must return an iterator object; if you return a different object, it must itself be an iterator, otherwise the for loop will fail.

python iterator: Practical Usage and Code Examples | RYUSLOG DEV