itertools.groupby: Aggregate Consecutive Runs Without Losing Records

Reviewed October 9, 2026. Examples target Python 3.12 and use synthetic ordered records.

A report can split one customer's total into several rows even though its code uses a function named groupby. Another report can produce empty groups because it saves iterators and reads them only after moving to the next group. These failures are consequences of a particular streaming contract, rather than random behavior or a defect in Python.

itertools.groupby groups consecutive equal keys. That makes it useful for processing runs, such as adjacent status segments in an event stream, and for aggregating records whose equal keys are already contiguous. This tutorial distinguishes those two uses, checks the ordering assumption, and explains when preserving a group as a list is appropriate. All amounts below are invented integers chosen for transparent arithmetic.

1. Decide whether order carries meaning

Before grouping records, ask whether the goal is to summarize adjacent runs or collect every occurrence of a key. A sequence of statuses such as waiting, running, waiting contains two waiting runs. Merging them would discard the fact that a running segment separated them. In that case, the original order is part of the answer.

For a total by customer, separation is usually incidental rather than meaningful. Records for the same customer must be brought together or accumulated in a mapping. Sorting can make equal keys consecutive, but sorting also changes order and requires its own resources. The choice should follow the intended question rather than the name of a library function.

Specify a deterministic key function. If one stage sorts by normalized customer code but another groups by the original spelling, their definitions of equality do not match. The ordering contract and grouping contract should use the same key rule whenever the goal is one group per key.

2. Demonstrate the difference with interleaved records

The fixture intentionally places customer A on both sides of a B record. Consecutive grouping returns three groups in the original order. Sorting by the grouping key makes the two A records contiguous and produces a single A total.

from itertools import groupby
from operator import itemgetter

records = [
    {"customer": "A", "units": 2},
    {"customer": "B", "units": 5},
    {"customer": "A", "units": 3},
]
key = itemgetter("customer")
runs = [(customer, sum(row["units"] for row in group))
        for customer, group in groupby(records, key=key)]
assert runs == [("A", 2), ("B", 5), ("A", 3)]

ordered = sorted(records, key=key)
totals = [(customer, sum(row["units"] for row in group))
          for customer, group in groupby(ordered, key=key)]
assert totals == [("A", 5), ("B", 5)]
assert sum(total for _, total in totals) == sum(row["units"] for row in records)

statuses = ["waiting", "waiting", "running", "waiting"]
segments = [(status, len(list(group))) for status, group in groupby(statuses)]
assert segments == [("waiting", 2), ("running", 1), ("waiting", 1)]
print(runs, totals, segments, sep="\n")

The two reports answer different questions while preserving the same overall number of units. A reconciled grand total is useful but cannot establish correct grouping. The per-key expected result catches the error that an overall sum would miss.

The table compares contracts, not speed measurements. A streaming interface is useful only if the surrounding reader, grouping logic, and result handling preserve its intended resource behavior.

3. Consume a group before advancing the outer iterator

The group iterator shares the underlying input with the outer groupby iterator. Advancing to another group consumes input that the previous group depended on. Consequently a group object should generally be reduced or materialized while it is the current group.

The next program makes the lifetime visible with explicit next calls. It also creates a safe snapshot immediately, demonstrating the tradeoff between lazy access and retained data. Each snippet is a complete independent program; neither depends on state created by the earlier fixture.

from itertools import groupby

outer = groupby(["A", "A", "B", "B"])
key_a, group_a = next(outer)
assert key_a == "A"
assert next(group_a) == "A"
key_b, group_b = next(outer)
assert key_b == "B"
assert list(group_a) == []
assert list(group_b) == ["B", "B"]

retained = [(key, list(group)) for key, group in groupby(["A", "A", "B"])]
assert retained == [("A", ["A", "A"]), ("B", ["B"])]

def checked_rows(rows):
    previous = None
    seen = False
    for key, units in rows:
        if seen and key < previous:
            raise ValueError("input must be nondecreasing by key")
        if not isinstance(units, int) or isinstance(units, bool) or units < 0:
            raise ValueError("units must be a nonnegative integer")
        previous = key
        seen = True
        yield key, units

good = [("A", 2), ("A", 3), ("B", 5)]
totals = [(key, sum(units for _, units in group))
          for key, group in groupby(checked_rows(good), key=lambda row: row[0])]
assert totals == [("A", 5), ("B", 5)]
rejected = False
try:
    list(checked_rows([("B", 5), ("A", 2)]))
except ValueError:
    rejected = True
assert rejected
print(retained, totals)

The validator enforces a nondecreasing string-key contract and nonnegative integer units for this fixture. It is stronger than the minimum requirement of contiguous equal keys, but simpler to verify incrementally. Rejecting a Boolean explicitly matters because Python otherwise treats it as an integer for an isinstance check.

4. Choose reduction or retention intentionally

If a group only needs a count or sum, consume it with that reduction and discard the individual records. If it needs several summaries, compute them in one pass or materialize the group once. Attempting two independent passes over the same exhausted iterator does not restore its records.

Materializing a group into a list makes later use dependable, but the largest group can be large. Retaining every such list can also retain the complete dataset. A workflow described as streaming should state whether it holds one group, one aggregate per key, or every original row.

Never save the raw group iterators as a substitute for snapshots. Their shared lifetime means they do not represent independent collections waiting for later consumption. If the caller needs independent groups, return materialized values or another explicitly owned representation rather than leaking those temporary iterators across an interface boundary.

5. Understand what sorting changes

Python's sorted returns a new list and preserves the relative order of records with equal keys. That stability is useful when within-key order matters, but it does not make sorting a streaming operation. Feeding it an unbounded or very large iterator can require materializing more data than the workflow intended.

For an upstream database or file that promises key ordering, validate and consume that promise rather than automatically sorting again. The program's comparison rule must agree with the producer's collation and normalization rules. A database ordering textual codes under one collation may not match Python's string comparison in every language or character set.

If arbitrary input must be globally aggregated, a dictionary-based accumulator is often conceptually simpler than sorting first. Its storage grows with distinct keys, so it also has a resource contract. For workloads exceeding that contract, use an appropriate external aggregation system rather than claiming that one standard-library iterator solves all large-data cases.

6. Make partial failure behavior explicit

A validation error late in a stream may occur after earlier groups have already been computed. If the consumer publishes each aggregate immediately, the destination can contain a partial report. If the report must be all-or-nothing, stage results and finalize them only after the entire required input has been validated successfully.

Record counts and aggregate totals help with reconciliation, but they do not replace ordering validation. The first fixture demonstrates that a faulty per-key result can preserve the overall total. Keep both levels of checks when grouping identity matters.

Consider missing or malformed keys at the input boundary. Decide whether to reject them or map them to a documented group before ordering. Mixing None with strings can also make a simple nondecreasing comparison invalid. The demonstration's all-string keys intentionally avoid pretending to cover every possible record schema.

7. Test runs, identities, and lifetimes separately

Include an interleaved repeated key, an empty stream, a single record, and a large contiguous group in your test design. Check per-key expectations rather than only the grand total. If original sequence order matters, verify the sequence of groups as well as their contents.

For interfaces that return groups for later use, test consumption after the producer has advanced. This exposes whether the implementation retained independent records or returned shared iterators. For one-pass reducers, test that a second summary does not silently operate on an already exhausted group.

A robust grouping workflow explains what defines equality, why keys are contiguous when necessary, when each group is consumed, and what remains in memory. Those explanations let another developer distinguish a meaningful run summary from a global aggregate and review either one with confidence.

Primary sources

• Python 3.12 itertools: consecutive grouping, shared group iterators, and consumption requirements.
• Python 3.12 built-in functions: sorted, key functions, and sort stability.
• Python 3.12 Functional Programming HOWTO: iterators, one-pass consumption, and functional reductions.