TF-IDF Text Features: Treat Vocabulary and Weights as Learned State

Reviewed October 9, 2026. Examples target scikit-learn 1.8.0 and use a synthetic English corpus.

A text classifier receives a new message containing only words it has never encountered. The preprocessing step still returns a matrix with the expected number of columns, but that message becomes a zero vector. The pipeline ran successfully while losing every observed token. Matrix shape alone is therefore a weak test of whether a text representation is useful.

TF-IDF combines within-document term counts with corpus-dependent inverse document frequencies. Its vocabulary, weights, tokenization policy, and normalization together define a learned representation. This guide makes that state inspectable, derives a small calculation, and explains why evaluation documents should not silently influence the representation used to assess a model.

1. Define what counts as a document and token

The document unit might be a message, paragraph, support ticket, or whole article. Splitting one article into ten documents changes document frequencies even if the words are unchanged. Deduplicated versus repeated documents also change the corpus statistics. Choose the unit and record how documents were selected before interpreting any weight as evidence of importance.

Tokenization establishes which text units can become features. The default word analyzer lowercases text and uses a token pattern that generally selects alphanumeric word tokens with at least two characters. A one-character product code may disappear. Punctuation, case, accents, and language-specific segmentation can matter, so inspect actual analyzer output for representative domain strings.

TF-IDF is not inherently an understanding of meaning. It represents observed textual features under a chosen extraction rule. Synonyms can occupy different columns, and the same word can mean different things in different documents. It remains a useful, inspectable baseline when its feature contract matches the task and its evaluation is honest.

2. Trace the smoothed IDF calculation

With smoothing enabled, scikit-learn uses an inverse document frequency of log((1 + n) / (1 + df)) + 1, where n is the training document count and df counts documents containing the term. The extra constant means a term present in every document still has IDF one. Term repetition affects term frequency, while presence across documents affects document frequency.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

documents = ["red red blue", "blue green", "blue"]
vectorizer = TfidfVectorizer(norm=None, smooth_idf=True)
matrix = vectorizer.fit_transform(documents)
names = vectorizer.get_feature_names_out().tolist()
weights = dict(zip(names, vectorizer.idf_))
assert names == ["blue", "green", "red"]
assert np.isclose(weights["blue"], 1.0)
assert np.isclose(weights["red"], np.log(4 / 2) + 1)
red_column = vectorizer.vocabulary_["red"]
assert np.isclose(matrix[0, red_column], 2 * weights["red"])
assert matrix.shape == (3, 3)
print(names, matrix.toarray())

Normalization is disabled to expose the raw product of count and IDF. With the default L2 normalization, each nonzero document vector is rescaled to unit Euclidean length. Its final component is then relative to the other components in the document. A larger raw count need not translate into the same proportional change after normalization.

The tiny dense printout is for inspection only. Text matrices are usually sparse, and converting a large corpus with toarray() can require substantial memory. A production diagnostic should inspect selected rows or features without materializing the entire dense matrix. The example makes no memory or speed claim for real datasets.

3. Freeze vocabulary when transforming new documents

Calling fit_transform learns state; calling transform uses existing state. Applying fit_transform independently to training and test corpora can produce different columns and weights, even if the resulting matrix widths happen to match. A classifier coefficient attached to one token cannot safely be applied to another token occupying the same column position.

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer()
training = vectorizer.fit_transform(["red blue", "blue green"])
before = vectorizer.get_feature_names_out().copy()
new = vectorizer.transform(["violet unknown", "red violet", ""])
assert np.array_equal(before, vectorizer.get_feature_names_out())
assert new.shape == (3, training.shape[1])
assert new[0].nnz == 0
assert new[1].nnz == 1
assert new[2].nnz == 0
analyzer = vectorizer.build_analyzer()
assert analyzer("A blue, RED!") == ["blue", "red"]
assert np.isclose(np.linalg.norm(new[1].toarray()), 1.0)
print(new.toarray())

Unseen terms are ignored under the learned vocabulary. The unknown-only and empty documents both yield zero vectors in this fixture, though their source meanings differ. Keep diagnostics about the original text so the system can distinguish an empty submission from a nonempty message whose tokens were all excluded. Neither case should be inferred solely from a model score.

4. Keep vocabulary fitting inside the evaluation boundary

Learning vocabulary and document frequencies from all available text before splitting train and test lets evaluation documents influence preprocessing. The process may not use their labels, but it still changes feature selection and weighting using information outside the training boundary. For ordinary inductive evaluation, fit text preprocessing on the training partition only.

A pipeline containing a vectorizer and estimator helps cross-validation fit the representation separately within each training fold. It does not automatically choose a valid split. Repeated authors, duplicate documents, or later messages from the same thread can still connect training and validation observations. Use group or chronological boundaries when those relationships define the deployment question.

There are intentional transductive settings where an unlabeled evaluation corpus is available during training. Such a design should be stated explicitly and evaluated under that assumption. It should not be silently described as a model's performance on completely unseen future documents. The evaluation contract determines which information is permitted.

5. Audit filtering and empty representations

Vocabulary filtering is a modeling choice rather than a universal cleanup recipe. An integer min_df represents a document-count threshold; a fractional value expresses a corpus proportion. On a small corpus, an apparently modest threshold can remove almost everything. Inspect retained and excluded features, especially for minority categories or specialized terminology.

Stop-word removal deserves the same scrutiny. A generic list can remove words that distinguish the task, while leaving domain-specific boilerplate untouched. Negation is particularly sensitive: deleting a word that changes the meaning of a phrase can damage a classifier even if the vocabulary becomes smaller. Test representative phrases rather than assuming fewer columns always improve quality.

Report the share of documents with zero retained features, broken down by source, time, or relevant language. A rising zero-vector rate can indicate drift, a new input format, or a tokenizer mismatch. It is not by itself proof that predictions are wrong, but it is a useful signal that the representation is discarding input information.

6. Interpret feature weights within their context

An IDF weight describes rarity in the fitted corpus, not truth, causal influence, or universal linguistic importance. A rare typo can have high IDF. A common word can be highly predictive once combined with other features. Interpreting a classifier also requires its coefficients, class definitions, regularization, and the evaluation data, not just a ranked list of IDF values.

With L2-normalized vectors, a dot product can express cosine similarity for nonzero vectors. A zero vector needs explicit treatment because it has no ordinary cosine direction. Retrieval systems should define what happens when a query has no retained features rather than quietly treating a zero similarity score as meaningful evidence that no relevant document exists.

If a fixed vocabulary is supplied, the feature columns can remain stable while IDF is still learned from the fitted corpus. Freezing names alone does not freeze the entire representation. Save the fitted vectorizer state and preprocessing configuration together, and validate that inference uses the same analyzer and feature ordering as training.

7. Build an accountable text-feature baseline

Record corpus selection, document unit, deduplication, language assumptions, vectorizer parameters, library version, and fitted feature names. Retain a small set of representative strings with expected analyzer outputs and expected zero-vector behavior. These checks make accidental preprocessing changes visible before they silently alter model inputs.

The synthetic examples verify the smoothed formula, repeated counts, frozen vocabulary, and ignored unknown terms. They do not establish accuracy for sentiment analysis, search, or any other real application. Compare against an appropriate baseline on a leakage-resistant split, report uncertainty or variation where relevant, and inspect actual errors. A useful TF-IDF workflow is one whose representation and evidence can both be explained.

Sources and further reading

• scikit-learn 1.8 TfidfVectorizer reference: tokenization, vocabulary, filtering, and fitted state.
• scikit-learn 1.8 TfidfTransformer reference: weighting formula and normalization.
• scikit-learn 1.8 common pitfalls: training-only preprocessing and pipeline evaluation. Corpus auditing recommendations are workflow design conclusions.