Scikit-Learn Preprocessing Pipelines: Keep Learned State Inside the Evaluation

Reviewed October 8, 2026. Examples target pandas 2.2.3 and scikit-learn 1.8.0 with synthetic data.

A model needs a numeric matrix, but a useful table contains numbers, category labels, missing values, and columns that should never be features. Preparing that matrix outside the evaluation can accidentally let held-out rows influence means, imputations, and category vocabularies. A prediction pipeline needs to carry the fitted preprocessing state along with the estimator.

ColumnTransformer applies different operations to selected columns and combines their outputs. Pipeline connects preprocessing to a final estimator. Together they can make the training boundary easier to maintain, provided the whole pipeline is fitted inside that boundary. They do not independently choose a correct split, validate every raw input, or prove reliability on unfamiliar categories.

1. Distinguish learned operations from fixed rules

A fixed conversion such as changing a documented unit does not estimate a parameter from the dataset. A median imputer, standard scaler, and inferred category encoder do. Their fitted state belongs to training data in an inductive evaluation. Using all rows to estimate that state exposes the training process to information about the held-out distribution.

The boundary matters even when no target values are used. Test-set numeric values can affect a fitted mean; test categories can affect the output vocabulary. For ordinary evaluation of deployment on unseen rows, fit those operations using training rows only. If a different transductive protocol intentionally uses unlabeled evaluation data, it needs explicit justification and reporting rather than being an accidental implementation detail.

Start with a raw schema: required columns, types, units, missing markers, and allowed feature fields. A pipeline handles the declared transformations; it is not a replacement for validating that an unexpected string has not entered a numeric column or that an ID has not been included as a predictor.

2. Inspect the transformation with a hand-checkable fixture

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

train = pd.DataFrame({"value": [1.0, 3.0], "kind": ["A", "B"]})
new = pd.DataFrame({"value": [5.0], "kind": ["C"]})
prepare = ColumnTransformer([
    ("num", StandardScaler(), ["value"]),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["kind"]),
], remainder="drop")
prepare.fit(train)
transformed = prepare.transform(new)
assert np.allclose(transformed, [[3.0, 0.0, 0.0]])
assert prepare.named_transformers_["num"].mean_.tolist() == [2.0]
assert prepare.get_feature_names_out().tolist() == [
    "num__value", "cat__kind_A", "cat__kind_B"]
print(prepare.get_feature_names_out(), transformed)

The training mean is two and the scaler's standard deviation is one for this fixture, so five becomes three. Category C was not seen during fitting. The chosen ignore policy gives zeros for both known-category columns; it does not invent a learned effect for C. Transforming the new row does not update the training mean.

Column selection and output naming are important review tools. Default dropping of unspecified columns can be intentional, or can hide a forgotten feature. Conversely, passing through every remaining column can accidentally include identifiers or targets. Define the feature set rather than treating remainder behavior as a convenience setting.

3. Put the estimator behind the same training boundary

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

X = pd.DataFrame({"value": [1.0, 2.0, np.nan, 4.0, 5.0, 6.0],
                  "kind": ["A", "B", "A", "B", "A", "B"]})
y = np.array([0, 0, 0, 1, 1, 1])
numeric = Pipeline([("impute", SimpleImputer(strategy="median")),
                    ("scale", StandardScaler())])
preprocess = ColumnTransformer([
    ("num", numeric, ["value"]),
    ("cat", OneHotEncoder(handle_unknown="ignore", sparse_output=False), ["kind"]),
])
model = Pipeline([("prepare", preprocess),
                  ("classify", LogisticRegression(max_iter=200))])
cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=7)
checked = cross_validate(model, X, y, cv=cv, scoring="accuracy",
                         return_estimator=True, error_score="raise")
assert len(checked["estimator"]) == 3
for fitted, (training, testing) in zip(checked["estimator"], cv.split(X, y)):
    imputer = fitted.named_steps["prepare"].named_transformers_["num"].named_steps["impute"]
    expected = np.nanmedian(X.iloc[training]["value"].to_numpy())
    assert np.isclose(imputer.statistics_[0], expected)
model.fit(X, y)
new = pd.DataFrame({"value": [8.0], "kind": ["C"]})
assert model.predict(new).shape == (1,)
print("Each fold retained its own training-only imputation state.")

The example checks fitted imputation state in each fold rather than advertising a score from six rows as predictive evidence. The final fit on all six rows illustrates the separate step after the construction check. A real project needs an evaluation design and dataset large enough to support its intended claim.

4. Unknown categories need operational handling

Missing-category adaptation is a separate decision from unknown-category handling. Decide how raw None, empty text, and numeric missing markers are represented before they reach the categorical branch. If missingness becomes a category or an imputed label, document that policy and monitor its frequency. A category can be known to the encoder yet still indicate an upstream collection problem. Do not merge absence and unfamiliar supplied text merely because both would otherwise complicate prediction.

OneHotEncoder's default raises for unknown categories. Ignoring them keeps transformation executable, but can mask a shift in the input vocabulary unless unknown occurrences are monitored separately. A model trained on A and B may behave poorly on C even though predict returns a value.

An infrequent-category policy can map unknown labels into an infrequent bucket only when such a bucket was learned. If none exists, that setting can behave like ignoring. Inspect fitted categories and infrequent groups before promising a special fallback representation.

Dropping a reference category also changes interpretation: an all-zero encoding can represent the dropped category as well as an ignored unknown. That ambiguity may be acceptable under a documented model design, but it should be understood. Encoder settings are part of the feature contract, not merely formatting preferences.

5. Pipelines do not fix the evaluation question

A pipeline fitted inside cross-validation avoids a common preprocessing leak. It does not prevent the same person from appearing in both folds, enforce chronology, or remove a feature measured after the target. Choose row, group, and time boundaries according to the deployment task before interpreting scores.

Hyperparameter selection also belongs inside the appropriate evaluation design. Trying many configurations against one held-out set and then reporting the best result from that same set changes what the score represents. Keep a final evaluation boundary or use a suitable nested design when tuning claims require it.

6. Review the complete prediction contract

Inspect output dimensions as well as values. A learned category vocabulary can change across retraining runs, and an all-missing numeric training column can require an explicit imputer policy. A downstream component expecting fixed positions must receive the fitted feature names and compatible model version, rather than a freshly assembled matrix that happens to have the same width. Save the preprocessing and estimator together using a persistence approach appropriate to the application and its trust boundary.

Dense output is used in these tiny examples for inspection. One-hot representations with many categories can be large, and sparse output may be more suitable when the final estimator supports it. Confirm the actual output representation and downstream compatibility before changing the setting. The six-row demonstration makes no memory or speed claim about a large encoded table; those properties need representative measurement in the intended environment.

Record the raw schema, selected features, missing-value adapter, learned transformations, estimator parameters, and library versions. Test required columns, reordered columns under the supported interface, missing values, unknown categories, and extreme numeric inputs. Ensure output feature names remain interpretable for inspection and downstream reporting.

For practice, replace the ignore policy with the default and confirm that category C raises. Then add an explicit raw-input audit that reports unfamiliar categories before prediction. A robust pipeline combines retained training state with visible input checks; successful matrix construction alone is a narrower achievement.

Sources and scope

• scikit-learn 1.8: pipelines and composite estimators.
• scikit-learn 1.8: OneHotEncoder.
• scikit-learn 1.8: common pitfalls and leakage.
• scikit-learn 1.8: ColumnTransformer.

The fixtures verify mechanics and fitted-state boundaries. They are not a classifier benchmark or evidence of performance on unseen populations.