Grouped Validation: Test New Entities Without Mixing Their Records
Audit participant overlap, choose groups that match deployment, and distinguish row-weighted from entity-weighted evaluation.
Reviewed October 8, 2026. Examples target scikit-learn 1.8.0 and use synthetic entity IDs.
A dataset contains many rows from each participant. Randomly splitting the rows can place the same participant in training and evaluation. A model may then exploit participant-specific patterns rather than generalize to a new participant. A high score under that split answers a different question from performance on people the model has never encountered.
Grouped validation keeps related records together at the split boundary. Its value depends on choosing groups that represent the intended unseen unit: participant, device, household, document, or another source of dependence. This article checks that boundary explicitly and shows why preventing group overlap is only one part of a defensible evaluation.
1. State what will be new at deployment
Predicting another event for a known participant and predicting an event for a new participant are distinct tasks. Historical records from the same person may be legitimately available for the first task, subject to time restrictions. They are not available as that person's training history in the second task.
Choose the split according to that distinction before measuring model quality. A grouped score can be lower than a row-split score because it asks a harder or different question, not because the splitter is defective. Conversely, automatically grouping every dataset by an arbitrary field can create an evaluation that does not match the real deployment population.
Group IDs need to be stable across the full dataset. If the same participant has two identifiers, the split can still overlap their true identity. If unrelated participants share an identifier, records may be grouped unnecessarily. Audit identity resolution and source provenance before relying on a software assertion about IDs as proof of the underlying separation.
2. Inspect ordinary and grouped folds directly
import numpy as np
from sklearn.model_selection import GroupKFold, KFold
groups = np.tile(np.arange(4), 3)
X = np.arange(len(groups)).reshape(-1, 1)
row_overlap = []
for train, test in KFold(n_splits=3).split(X):
overlap = set(groups[train]) & set(groups[test])
row_overlap.append(len(overlap))
assert row_overlap == [4, 4, 4]
seen_test_rows, seen_test_groups, sizes = [], [], []
for fold, (train, test) in enumerate(GroupKFold(n_splits=3).split(X, groups=groups), 1):
training_groups, test_groups = set(groups[train]), set(groups[test])
assert training_groups.isdisjoint(test_groups)
seen_test_rows.extend(test.tolist())
seen_test_groups.extend(test_groups)
sizes.append(len(test))
print(fold, "test groups", sorted(test_groups), "test rows", len(test))
assert sorted(seen_test_rows) == list(range(len(groups)))
assert sorted(seen_test_groups) == [0, 1, 2, 3]
assert sorted(sizes) == [3, 3, 6]
The ordinary folds split twelve rows into blocks of four. Because IDs repeat in each block, every entity appears on both sides of each split. GroupKFold keeps each entity wholly on one side in a fold and assigns every group to a test fold once across the sequence.
The grouped test folds have six, three, and three rows in this versioned fixture. Group separation does not promise identical fold sizes. The number of distinct groups must also support the requested number of folds. Increasing the fold count cannot create more independent participants than the dataset contains.
| Split design | Unit kept together | Question it can support when other assumptions hold |
|---|---|---|
| Ordinary row split | Individual row | New rows under the assumed row-sampling process |
| Entity group split | All rows of one entity | Generalization to held-out entities |
| Chronological split | Earlier versus later observations | Prediction under a specified time boundary |
| Site or source split | Records from one source | Transfer to a held-out source |
These boundaries can be combined, but one does not automatically imply the others. A held-out participant may still come from the same site and time period as training participants.
3. Choose the right grouping level
Dependence may exist at more than one level. Participants can share households, devices can share a site, and sentences can share a document. Holding out a participant while retaining their household's other members may still expose patterns unavailable for a truly new household. The grouping level should follow the strongest boundary relevant to the intended claim.
A composite key is useful only when it preserves the intended relationships. Grouping by participant-plus-session keeps each session together, but allows the same participant's other sessions into training. That may match a new-session task and fail a new-participant task. Explain the unit in ordinary words before encoding it.
Near-duplicate records and derived samples deserve similar attention. Crops from one image or chunks from one document should not be treated as independent original sources when the claim concerns new images or documents. Retain source IDs through preprocessing so the splitter can use them.
4. Weighting changes what the reported score represents
Even with correct separation, an entity contributing many rows can dominate a pooled row-level metric. That may be appropriate if deployment decisions occur per row with similar importance. It can be inappropriate if the claim concerns an equally weighted average participant.
import numpy as np
import pandas as pd
errors = pd.DataFrame({"entity": ["A", "A", "A", "A", "B", "C"],
"absolute_error": [1.0, 1.0, 1.0, 1.0, 10.0, 10.0]})
row_weighted = errors["absolute_error"].mean()
entity_means = errors.groupby("entity")["absolute_error"].mean()
entity_weighted = entity_means.mean()
assert np.isclose(row_weighted, 24 / 6)
assert entity_weighted == 7.0
print("Per row:", row_weighted, "Per entity:", entity_weighted)
The same six error values produce four per row or seven after averaging each entity first. Neither arithmetic result is a contradiction. They assign different weights. State the metric's unit, weighting rule, and denominator so the reader knows which population experience it summarizes.
Likewise, the unweighted mean of fold scores is not always the pooled score over all held-out predictions. For an additive metric such as mean absolute error, fold row counts can explain that difference. For nonlinear metrics, pooling needs more care. Preserve held-out predictions and group identities rather than only a short list of fold scores.
5. Keep class balance, chronology, and preprocessing visible
GroupKFold does not guarantee that each class appears in every fold. A group can contain mostly one class, and a small number of groups can make balance impossible. StratifiedGroupKFold attempts to preserve class proportions while respecting groups, but cannot guarantee a balanced result for arbitrary group structure. Inspect actual fold distributions before fitting.
Chronology remains a separate constraint. If the task predicts future records for new entities, a group split alone may mix earlier and later periods in ways that do not match deployment. Build an explicit combined protocol and audit timestamps or label availability alongside group overlap.
Fit preprocessing and feature selection inside each training fold. A perfect group split does not excuse an imputer or encoder fitted using all rows. Hyperparameter selection also needs the appropriate inner split. Keep a final evaluation boundary when repeated tuning would otherwise make the reported score optimistic.
6. Report the evidence and its limits
Record the group definition, number of independent groups, fold row counts, class distributions, overlap checks, and metric weighting. Describe whether the evaluation concerns new entities, new sites, later time periods, or a specific combination. A count of rows should not substitute for a count of independent groups when discussing uncertainty.
For practice, make one entity ten times larger and inspect the resulting folds. Then compute both row-weighted and entity-weighted errors from the same held-out predictions. Discuss which aligns with the intended use. The fixture in this article checks splitting mechanics; a real generalization claim requires representative groups, a meaningful outcome, and an appropriate uncertainty analysis.
Sources and scope
- scikit-learn 1.8: grouped cross-validation.
- scikit-learn 1.8: GroupKFold.
- scikit-learn 1.8: StratifiedGroupKFold.
- scikit-learn 1.8: common pitfalls.
All identities and errors are synthetic. The examples do not estimate the performance of a deployed model or establish independence among real participants.