Python Iterator Exhaustion: Why Iterators Are Single-Use

When you iterate over a Python iterator a second time, you may be surprised to get no results. This is not a bug; it is the iterator protocol working as designed. Python iterator exhaustion means that once an iterator has yielded all its values, it cannot be restarted. Understanding this behavior is essential for writing correct code, especially when passing iterators to functions or looping over the same data source multiple times.

What Happens When an Iterator Is Exhausted?

Every iterator in Python follows the iterator protocol: it implements __iter__() and __next__(). Calling next() on an iterator repeatedly returns successive values until the iterator is exhausted. At that point, next() raises StopIteration. A for loop catches this exception internally and exits cleanly, but any direct call to next() will propagate it.

numbers = iter([1, 2, 3])
print(next(numbers))  # 1
print(next(numbers))  # 2
print(next(numbers))  # 3
# The next call raises StopIteration

Once StopIteration has been raised, the iterator is permanently exhausted. Calling next() again will raise the same exception again. This is true for all iterators, including lists converted with iter(), generator expressions, and generator functions.

Why Iterators Are Single-Pass by Design

The single-pass nature is a deliberate tradeoff. Iterators are lazy: they produce values on demand and do not store the entire sequence in memory. This is what allows processing of infinite streams or files larger than RAM. For example, a generator can yield lines from a huge log file without loading the whole file into memory.

def read_lines(path):
    with open(path) as f:
        for line in f:
            yield line

Once a generator has been consumed, it cannot be rewound. The state is gone. This design keeps memory usage low but requires the developer to plan for multiple passes explicitly.

Detecting Exhaustion Without Catching StopIteration

Sometimes you need to know whether an iterator still has values without triggering an exception. The next() function accepts a default argument that is returned instead of raising StopIteration.

it = iter([1, 2])
print(next(it, None))  # 1
print(next(it, None))  # 2
print(next(it, None))  # None

This pattern is useful when you want to peek at the first element or check if an iterator is empty. However, it consumes the value you peek at. If you need to keep that value, you must store it separately.

Another approach uses the two-argument form of iter() with a sentinel. This is handy for reading until a specific value appears.

with open('data.txt') as f:
    for block in iter(lambda: f.read(64), ''):
        process(block)

Here the iterator calls the callable until it returns the sentinel '', which marks the end. This is a clean way to build an iterator that stops at a custom condition.

Reusing an Iterator: Recreating vs. Copying

Because iterators cannot be reset, the only reliable way to iterate twice is to create a fresh iterator from the original data source. If the source is a list, you can call iter() again. If it is a generator function, you can call the function again. But if you have a generator object that has already been consumed, you cannot recover its values.

def generate():
    yield 1
    yield 2

gen = generate()
print(list(gen))  # [1, 2]
print(list(gen))  # [] because gen is exhausted

# To iterate twice, create a new generator:
gen2 = generate()
print(list(gen2))  # [1, 2]

For iterators that wrap a list, you can always get a new iterator from the list. But if you only have an iterator object (e.g., from map() or filter()), you cannot re-create it without the original arguments.

Practical Patterns for Multiple Passes

When you need to iterate over the same sequence more than once, you have several options. The simplest is to materialize the iterator into a list, but that defeats the memory advantage. For large or infinite sequences, itertools.tee() is the standard tool.

from itertools import tee

def process(iterable):
    it1, it2 = tee(iterable, 2)
    # Now it1 and it2 are independent iterators
    for x in it1:
        pass
    for y in it2:
        pass

tee() clones the iterator into multiple independent iterators. However, it does so by buffering values that have been consumed from one clone but not the other. If you advance one clone far ahead, the buffer grows. This can be memory-intensive, so tee() is best used when the clones are consumed roughly in parallel.

Another pattern is to wrap the iterator in a class that can produce fresh iterators from the underlying data. This is useful when the data source can be re-read, such as a file or a database cursor.

class RepeatableIterable:
    def __init__(self, data_factory):
        self.data_factory = data_factory

    def __iter__(self):
        return self.data_factory()

This lets you create a new iterator on demand by calling the factory function each time __iter__ is invoked.

Common Pitfalls with Exhausted Iterators

A frequent mistake is passing the same iterator to multiple functions that each iterate over it. The first function consumes the iterator, and the second receives an empty iterator.

def sum_and_count(numbers):
    total = sum(numbers)
    count = len(list(numbers))  # numbers is already exhausted
    return total, count

The count will always be zero because sum() consumed the iterator. To fix this, either pass a sequence like a list, or use tee() to create two independent iterators.

Another pitfall is assuming that a generator can be reused after a for loop. Once the loop completes, the generator is exhausted. If you need to loop twice, create a new generator object or use itertools.tee().

Performance and Memory Tradeoffs

Choosing between an iterator and a list affects memory and latency. An iterator computes values lazily, so it uses minimal memory and can start producing results immediately. A list stores all values, which may be necessary if you need random access or multiple passes without re-computation.

itertools.tee() sits between these extremes. It avoids storing the entire sequence, but it does store the difference between the fastest and slowest clone. If one clone is consumed completely before the other starts, tee() will buffer everything, effectively becoming a list in memory. Understanding this behavior helps you decide when to use tee() versus a simple list.

For most small datasets, converting to a list is the simplest and most readable approach. For large or infinite streams, you must design your code to avoid multiple passes, or use tee() with careful consumption patterns. The key is to know whether your data source can produce a fresh iterator; if it cannot, you need to buffer or re-architect.