Pandas Categoricals: Treat the Vocabulary as a Schema, Not a Guess

Reviewed October 8, 2026. Examples were designed for pandas 2.2.3 and use synthetic labels.

A report expects three priority levels: low, medium, and high. A new file includes urgent. Converting the column to a declared categorical dtype makes that label disappear into a missing marker. The conversion succeeds, but the report has lost the difference between an absent response and a response outside its vocabulary. Successful casting is not the same as successful validation.

Categorical data is useful when a column has a defined set of labels, sometimes with a meaningful order. The value lies in making that vocabulary visible across files and calculations. This article explains how to preserve invalid input, choose aggregation behavior explicitly, and avoid interpreting internal category codes as measurements.

1. Define the vocabulary before reading its codes

Using astype("category") infers categories from the data currently present. Two batches can therefore infer different vocabularies. A batch containing only low and medium priorities says nothing about whether high is a valid level that simply did not occur. An explicit CategoricalDtype can represent the shared schema independently of the observations.

Ordering is another schema decision. Low, medium, and high have a domain order that differs from alphabetic order. Declaring ordered=True with that sequence makes categorical sorting follow it. For an unordered category such as a device type, inventing an order can imply comparisons that the domain does not support.

Codes are implementation identifiers for labels. A code of two does not mean twice the priority of a code of one, and missing values use a special code. Feeding codes into arithmetic or a distance-based model imposes numerical structure that was not established by declaring categories. Use an encoding chosen for the downstream model and the domain.

2. Keep original absence separate from invalid labels

import pandas as pd

raw = pd.Series(["low", "medium", "urgent", None], dtype="string")
schema = pd.CategoricalDtype(["low", "medium", "high"], ordered=True)
original_missing = raw.isna()
invalid = raw.notna() & ~raw.isin(schema.categories)
categorical = raw.astype(schema)
audit = pd.DataFrame({"raw": raw, "original_missing": original_missing,
                      "invalid_label": invalid, "priority": categorical})
assert original_missing.sum() == 1
assert invalid.sum() == 1
assert categorical.isna().sum() == 2
assert categorical.cat.codes.tolist() == [0, 1, -1, -1]
assert audit.loc[invalid, "raw"].tolist() == ["urgent"]
print(audit.to_string(index=False))

The categorical column alone cannot explain its two missing entries. The raw column and audit flags retain that explanation. A strict ingestion workflow could reject the batch whenever invalid.any() is true. A tolerant workflow could quarantine those rows while keeping valid ones. Both policies are clearer than silently adding every unfamiliar label to the vocabulary.

Text cleanup also needs a policy before validation. Stripping surrounding whitespace may be appropriate; collapsing case or merging spelling variants may not be. Record the original value and the normalization rule so a change in the input cannot masquerade as a change in the underlying phenomenon. Categories are a destination for validated labels, not a replacement for that work.

3. Decide whether unobserved categories belong in the report

In pandas 2.2.3, categorical groupby behavior depends on observed. Setting it explicitly avoids depending on a default that can change across versions. observed=True reports categories that occur; observed=False can include declared categories with no observations. dropna addresses a separate question: whether missing group keys appear.

import pandas as pd

schema = pd.CategoricalDtype(["low", "medium", "high"], ordered=True)
frame = pd.DataFrame({"priority": pd.Series(
    ["low", "medium", None], dtype=schema)})
observed = frame.groupby("priority", observed=True, dropna=False).size()
complete = frame.groupby("priority", observed=False, dropna=False).size()
assert "high" not in observed.index
assert complete.loc["high"] == 0
assert int(observed.sum()) == int(complete.sum()) == len(frame)
assert observed.iloc[-1] == complete.iloc[-1] == 1
print("Observed categories:\n", observed)
print("Declared vocabulary:\n", complete)

The missing row remains visible because dropna=False. The declared but unobserved high category receives a count of zero only in the second result. That zero means no rows in this fixture belong to that category. It does not imply a measured quantity of zero, complete survey coverage, or the impossibility of a high-priority observation.

If multiple categorical keys are grouped, including unobserved combinations can create more result rows than the observed data suggests. Check the intended output grain and size before interpreting a rectangular report as evidence that every combination was measured.

4. Preserve category semantics across batches

Apply the same dtype to compatible batches before comparing them. Confirm that categories, ordering, and normalization rules match. A schema revision that introduces a new priority is a change to the data contract; record its version and how older batches are interpreted. Do not silently rewrite historical labels to make a chart appear consistent.

Operations that remove categories can also produce new missing values when removed labels are still used. Renaming and removing therefore deserve tests against actual values, not only tests of the category list. A code review should ask whether the operation changes a vocabulary, changes values, or both.

Merging or concatenating data with incompatible categorical definitions may change the resulting dtype. Inspect the result instead of assuming category metadata survives every operation. Reapply a validated common schema when that is the desired contract, while preserving any labels that fail it. The correct behavior depends on the operation and compatibility of its inputs.

5. Memory savings need a representative measurement

Serialization is another boundary to test. A CSV export contains the displayed values but does not carry the complete categorical dtype, its unused levels, or its ordering metadata. Reloading that CSV and inferring categories can therefore produce a different vocabulary. Keep the schema in a versioned configuration or another documented metadata artifact and reapply it after validating reloaded values. A round-trip test should compare both values and the declared dtype when preserving categorical semantics is a requirement.

Categories can save storage when many rows repeat a limited vocabulary, but the benefit depends on row count, category count, and representation. The category dictionary itself has a cost. A column in which almost every value is distinct may not gain the same advantage as a column with three repeated labels.

This article reports no memory ratio because its tiny fixture cannot establish one for a real workload. If memory motivates a conversion, measure the representative column before and after with an appropriate memory inspection method. Include the vocabulary and the actual backend, and separately verify that semantics remain correct. A smaller representation that erases invalid-label evidence is not an acceptable improvement.

6. Build a category review checklist

A reviewable workflow states the allowed labels, whether they are ordered, how raw text is normalized, and what happens to unfamiliar values. Its report states whether unobserved levels and missing keys are included. It also reconciles group sizes to the accepted input rows rather than assuming the output is complete because every expected category appears.

For practice, add a second unknown label and an empty string to the first example. Decide whether empty text counts as absent or invalid under your schema. Then add a valid high observation and verify that the zero category count changes to one without changing the declared order. These exercises separate vocabulary design, observation counts, and missing-value policy.

Sources and scope

• pandas 2.2.3: categorical data.
• pandas 2.2.3: DataFrame.groupby parameters.
• pandas 2.2.3: CategoricalDtype.

All labels and counts are synthetic. The examples demonstrate schema behavior, not the distribution of priorities in a real organization.