Python Iterator One-Time Use: Why Iterators Can't Be Reused

Python iterators are one-time use objects: once you've consumed them, they're exhausted and cannot be reused. This behavior, often referred to as the python iterator one time use rule, is by design. It is a direct consequence of how the iterator protocol works, and it has important implications for how you structure loops, functions, and data pipelines.

What Makes an Iterator One-Time Use

An iterator in Python is any object that implements the __next__() method, which returns the next item or raises StopIteration when there are no more items. The for loop calls next() internally until it catches StopIteration. Once that exception is raised, the iterator is permanently exhausted. There is no built-in "rewind" mechanism.

Consider a simple list iterator:

numbers = [1, 2, 3]
it = iter(numbers)
print(next(it))  # 1
print(next(it))  # 2
print(next(it))  # 3
# next(it) would raise StopIteration

After the third next(), it is done. Calling next(it) again raises StopIteration. The iterator does not reset.

The Problem with Reusing an Iterator

The one-time nature becomes a problem when you try to iterate over the same iterator twice. For example:

it = iter([1, 2, 3])
for x in it:
    print(x)
# Second loop produces nothing
for x in it:
    print(x)

The first loop consumes all items. The second loop sees an exhausted iterator and immediately stops. This is a common source of bugs, especially when an iterator is passed to a function that consumes it and then the caller tries to use it again.

Creating a Fresh Iterator for Each Pass

The solution is to create a new iterator for each pass. For iterable objects like lists, tuples, and strings, you can call iter() again. For generators, you call the generator function again to get a fresh generator.

def generate_numbers():
    yield 1
    yield 2
    yield 3

for x in generate_numbers():
    print(x)
# Works again because generate_numbers() returns a new generator
for x in generate_numbers():
    print(x)

The key is to distinguish between an iterable (something you can call iter() on) and an iterator (the result of that call). Lists are iterables, not iterators. Generators are both iterable and iterator, but calling the generator function creates a new generator each time.

When to Materialize an Iterator into a List

If you need to iterate over the same data multiple times and the data set is small, converting the iterator to a list is the simplest approach.

data = list(generate_numbers())
for x in data:
    print(x)
for x in data:
    print(x)

This stores all items in memory, which is fine for small data. For large data, materializing can be expensive in terms of memory. Use this approach when you know the data fits comfortably in memory and you need random access or multiple passes.

Using itertools.tee to Duplicate an Iterator

When you want to avoid materializing the entire sequence but still need multiple iterators, itertools.tee can help. It creates independent iterators from a single source iterator.

from itertools import tee

it = iter([1, 2, 3, 4])
it1, it2 = tee(it, 2)

print(next(it1))  # 1
print(next(it2))  # 1
print(next(it1))  # 2

tee buffers items as they are consumed from the original iterator. The memory cost is proportional to the difference in consumption between the derived iterators. If one iterator is consumed far ahead of the other, the buffer grows. This is a tradeoff between memory and the ability to avoid storing the entire sequence.

Practical Patterns for One-Time Iterators

A common design pattern is to have functions accept iterables rather than iterators. This allows the caller to decide whether to pass a list, a generator, or any other iterable, and the function can call iter() on it each time it needs to iterate.

def process(data):
    for item in data:
        print(item)

# Passing a list works, and the list can be reused later
my_list = [1, 2, 3]
process(my_list)
process(my_list)  # works because my_list is an iterable, not an iterator

If you must pass an iterator, be aware that it will be consumed. If the function needs to iterate multiple times, it should accept an iterable and create its own iterator internally.

Performance and Memory Considerations

The choice between re-creating iterators, materializing to a list, or using tee depends on your data size and access pattern.

• Re-creating an iterator from an iterable is cheap if the underlying data is already in memory (e.g., a list). For generators, re-calling the function may involve re-computing values, which could be expensive if the generator does significant work.
• Materializing to a list uses O(n) memory but gives you random access and the ability to iterate any number of times without recomputation.
• tee uses memory proportional to the consumption difference, which is often much less than O(n) if the iterators are consumed roughly in sync. However, it does not give random access, and it can still be memory-heavy if one iterator lags far behind.

For large data streams, prefer re-creating iterators when possible, or use tee with careful consumption order. Avoid materializing huge datasets into lists unless you have no alternative.