Python itertools groupby: Grouping Consecutive Items

When you call python itertools groupby, you might expect it to group all equal items across an entire iterable. In practice, it only groups consecutive items that share the same key. This distinction is the most common source of confusion, and it determines when you need to sort your data before grouping.

The function is part of the itertools module in the Python standard library. It returns an iterator of (key, group) pairs, where key is the result of applying a key function to each element, and group is an iterator over the consecutive elements that produced the same key. Because the group is an iterator, you must consume it before moving to the next key, or you will lose data.

How itertools.groupby Works

The signature is itertools.groupby(iterable, key=None). The key function is optional; if omitted, the element itself is used as the key. The function evaluates the key for each element and starts a new group whenever the key changes from the previous element. It does not look ahead or behind, so it cannot know that a non-consec occurrence of the same key should be merged into an earlier group.

This behavior is by design. It allows groupby to operate in a single pass over the input with constant memory overhead, and it works with infinite iterables. The tradeoff is that you must arrange your data so that all items with the same key are contiguous if you want a single group per key.

A Minimal Example with a Sorted List

Consider a list of words and the goal of grouping them by their first letter. Without sorting, groupby will produce separate groups for the same letter if it appears in non-contiguous positions.

from itertools import groupby

words = ["apple", "avocado", "banana", "blueberry", "cherry"]

for letter, group in groupby(words, key=lambda w: w[0]):
    print(letter, list(group))

This outputs:

a ['apple', 'avocado']
b ['banana', 'blueberry']
c ['cherry']

Because the words are already sorted alphabetically, each first letter appears in one contiguous block. If you reorder the list to ["apple", "banana", "avocado", "cherry", "blueberry"], the output changes:

a ['apple']
b ['banana']
a ['avocado']
c ['cherry']
b ['blueberry']

The same letters appear in multiple groups because they are not consecutive. This is the core behavior you must understand before using groupby.

The Key Function and When to Use It

The key argument can be any callable that takes one element and returns a hashable value. Common choices are lambda functions, operator.itemgetter for dictionaries or tuples, and str.lower for case-ins grouping. The key function is evaluated once for each element, so it should be cheap and free of side effects. If the key function raises an exception, the entire iteration fails, and you get no partial groups.

A practical example with a list of dictionaries:

from itertools import groupby
from operator import itemgetter

records = [
    {"department": "sales", "name": "Alice"},
    {"department": "sales", "name": "Bob"},
    {"department": "engineering", "name": "Carol"},
    {"department": "engineering", "name": "Dave"},
]

for dept, group in groupby(records, key=itemgetter("department")):
    print(dept, [r["name"] for r in group])

Here itemgetter("department") extracts the department value as the key. Because the records are already sorted by department, the grouping works as intended. If they were not sorted, you would need to sort first.

The Sorting Requirement: Grouping Consecutive vs. All Items

The most common mistake with python itertools groupby is forgetting to sort the input. Since groupby only groups adjacent elements, you must sort by the same key you intend to group by. The standard pattern is:

from itertools import groupby

sorted_data = sorted(data, key=key_func)
groups = groupby(sorted_data, key=key_func)

Sorting adds O(n log n) time complexity, which is acceptable for most data sets. But if you do not need the groups in sorted order, and you only need to collect all items with the same key, a defaultdict or a regular dictionary may be simpler and more efficient.

Practical Use Cases for groupby

groupby shines when you need to process consecutive runs of data, such as detecting changes in a time series, compressing repeated values, or parsing log streams where events are already grouped by timestamp or severity.

For example, you might compress a run-length encoded list:

from itertools import groupby

data = [1, 1, 2, 2, 2, 3, 3, 1, 1]
runs = [(value, len(list(group))) for value, group in groupby(data)]
print(runs)

This produces [(1, 2), (2, 3), (3, 2), (1, 2)], which is a compact representation of the original sequence. Without sorting, the two runs of 1 remain separate, which is exactly what you want for run-length encoding.

Another common use is grouping log lines by a timestamp prefix when the logs are already in chronological order. You can iterate over the groups and process each batch without loading the entire log into memory.

Performance and Memory Considerations

groupby is lazy: it reads elements from the input one at a time and yields groups as they are discovered. The group iterator itself is lazy as well, so you must consume it before advancing to the next group. This means that if you call list(group) for each group, you are materializing that group into memory. For very large groups, that can be memory-heavy. If you only need to iterate over the group, you can avoid the list conversion.

The time complexity is O(n) for the grouping pass, but the overall cost depends on whether you sort first. Sorting adds O(n log n). For a one-off script with small data, this is irrelevant. For large data, you should measure whether sorting is acceptable or whether a dictionary-based grouping would be faster.

A dictionary approach, such as defaultdict(list), groups all items with the same key regardless of order, but it does not preserve the order of first appearance unless you use an ordered dict. It also requires storing all items, whereas groupby can process groups one at a time. The choice depends on whether you need to process groups incrementally or collect all items for each key at once.

Common Mistakes and How to Avoid Them

One common mistake is consuming a group iterator after moving to the next key. The group iterator is tied to the underlying iterable; once groupby advances, the previous group is exhausted. If you store the group iterator and try to use it later, you will get an empty list. Always consume the group within the loop body.

Another mistake is using a key function that has side effects or depends on mutable state. Since the key is evaluated for each element, any state change will affect subsequent keys and can break the grouping. Keep the key function pure and deterministic.

Empty input is handled gracefully: groupby returns an empty iterator. You do not need a special case for empty iterables, but you should be aware that no groups will be produced.

When to Choose groupby vs. Other Approaches

Use python itertools groupby when you need to process consecutive runs of items, when the input is already sorted or you are willing to sort it, and when you want lazy evaluation to avoid materializing the entire dataset. It is also the right tool when you need to detect transitions or compress runs.

Use a dictionary or defaultdict when you need to group all equal items regardless of order, when the input is unsorted and you do not want the cost of sorting, or when you need random access to groups after construction. The dictionary approach is often simpler for one-off grouping tasks.

For example, if you are counting occurrences of each key, collections.Counter is better than groupby because it does not require sorting and directly gives counts. If you are building a mapping from key to list of values, a defaultdict(list) is more direct. groupby is not a general-purpose grouping tool; it is a tool for grouping consecutive sequences.

A final consideration: groupby works with infinite iterables. You can pass an infinite generator as long as you break out of the loop after a condition. This is impossible with a dictionary approach, which would try to consume the entire iterable. If you are processing a stream, groupby is the only viable option.