Python itertools tee: Splitting Iterables into Multiple Streams

When you have a generator or any one-shot iterable, you cannot iterate over it twice. The first loop exhausts it, and the second loop sees nothing. Python's itertools.tee provides a way to create multiple independent iterators from a single source, allowing you to traverse the same data multiple times without converting everything into a list. This article explains how python itertools tee works, its memory behavior, and the practical tradeoffs you need to consider before using it.

What itertools.tee Does

itertools.tee takes an iterable and an optional integer n (default 2) and returns a tuple of n independent iterators. Each iterator yields the same sequence of items as the original iterable. The key point is that the iterators are independent: you can consume them at different paces, and each one starts from the beginning.

Under the hood, tee does not copy the data. Instead, it lazily pulls items from the original iterable and stores them in an internal buffer (a deque) until every iterator has consumed them. Once all iterators have passed an item, that item is discarded from the buffer. This design allows tee to work with infinite or large iterables without loading everything into memory at once, provided the iterators are consumed at roughly the same rate.

Basic Usage: Creating Independent Iterators

Here is a minimal example that duplicates a generator into two iterators:

from itertools import tee

def generate_numbers():
    for i in range(5):
        yield i

numbers = generate_numbers()
it1, it2 = tee(numbers)

print(list(it1))  # [0, 1, 2, 3, 4]
print(list(it2))  # [0, 1, 2, 3, 4]

Both it1 and it2 produce the same sequence. Notice that the original generator numbers is consumed by tee; you should not use it directly after calling tee. If you try to iterate over numbers again, it will be exhausted because tee already pulled items from it.

The second argument to tee controls how many iterators you get. For example, tee(data, 3) returns three independent iterators. This is useful when you need to process the same stream in multiple passes, such as computing several aggregations in one run.

How tee Buffers Items in Memory

Because tee is lazy, it does not precompute all items. Instead, it stores only the items that have been produced from the source but not yet consumed by every iterator. The buffer size depends on the consumption pattern of the iterators.

Consider this scenario:

from itertools import tee

data = iter(range(1000))
it1, it2 = tee(data)

# Consume it1 completely
list(it1)
# it2 still hasn't consumed anything
list(it2)

When it1 is fully consumed, tee has pulled all 1000 items from data and stored them in the buffer because it2 hasn't consumed any yet. Only after it2 starts does the buffer shrink. If you consume one iterator far ahead of the other, memory usage grows proportionally to the difference. If both iterators are consumed in lockstep, the buffer stays very small.

This behavior is the central tradeoff of tee. It gives you the flexibility of multiple passes without duplicating the entire dataset, but it can still consume significant memory if the iterators diverge. For infinite iterables, this can become a serious problem if you never consume one of the iterators.

Comparing tee with Converting to a List

The most common alternative to tee is converting the iterable to a list and then iterating over the list multiple times. Both approaches allow multiple passes, but they differ in memory and laziness.

Use tee when the source is a generator or a stream that you cannot re-create, and you need to iterate over it only a few times. Use list() when the dataset is small enough to fit in memory and you need random access or want to avoid the complexity of managing multiple iterator states.

Common Mistakes When Using tee

Several pitfalls can trip up developers new to tee.

Using the original iterable after calling tee. As mentioned, tee consumes the original iterable. If you try to iterate over it directly, you will get an exhausted iterator. Always use the iterators returned by tee.

Assuming tee copies data. tee does not copy the underlying data. It creates references to the same objects. If the items are mutable and you modify them through one iterator, the change will be visible through the other. This is usually what you want, but it can cause unexpected side effects if you are not careful.

Not consuming all iterators. If you create three iterators but only consume two, the third one will hold a reference to the buffer. Even if you discard it, the buffer may not be freed until all iterators are garbage-collected. In long-running processes, this can lead to memory leaks if you repeatedly call tee without consuming all returned iterators.

Using tee with infinite iterables without a consumption plan. If you have an infinite stream and you consume only one iterator, the other will buffer every item produced, causing memory to grow without bound. You must ensure that all iterators are consumed at a similar pace, or you must stop consuming one iterator and discard it (and the buffer) explicitly.

Using tee with Infinite or Large Iterables

tee shines when you need to process an infinite or very large stream in multiple ways. For example, suppose you have a stream of sensor readings and you want to compute both the running average and the maximum value in a single pass. Without tee, you would need to either store all readings or iterate the source twice (if possible). With tee, you can split the stream into two iterators and process them concurrently:

from itertools import tee, count

def sensor_stream():
    for i in count(1):
        yield i * 0.5  # simulated reading

readings = sensor_stream()
avg_it, max_it = tee(readings)

# Process avg_it for running average
running_sum = 0
count = 0
for value in avg_it:
    running_sum += value
    count += 1
    if count == 1000:
        break

# Process max_it for maximum (it will start from the beginning)
max_value = max(max_it)  # This would consume the rest of the stream if infinite!

In this example, consuming max_it fully on an infinite stream would never terminate. Instead, you would need to consume both iterators in parallel, for instance by zipping them or using a loop that advances both. The key is to design your consumption pattern so that the buffer does not grow unboundedly.

A practical pattern is to use zip to consume multiple iterators together:

from itertools import tee, islice

stream = iter(range(1000000))
it1, it2 = tee(stream)

for a, b in zip(it1, it2):
    # process a and b simultaneously
    pass

This keeps the buffer size at one item because both iterators advance at the same rate.

When to Choose tee Over Other Approaches

tee is not always the right tool. If you can re-create the iterable cheaply, iterating twice directly is simpler and avoids any buffer overhead. For example, if your data comes from a function that generates a sequence deterministically, you can call that function twice instead of using tee.

Use tee when:

• The source is a generator that cannot be re-created without side effects or expensive computation.
• You need only a small number of independent passes (typically 2 or 3).
• The iterators will be consumed at roughly the same rate, keeping memory bounded.

Avoid tee when:

• You need random access to the data; use list() instead.
• The data is small and you need to iterate many times; a list is simpler and faster.
• You have an infinite stream and cannot guarantee that all iterators will be consumed; you risk unbounded memory growth.

In practice, tee is a specialized tool that solves a specific problem: splitting a one-shot iterable into multiple independent streams without materializing the entire dataset. Understanding its buffering behavior and memory implications helps you decide when it is the right choice and when a simpler alternative is better.