Word and Character N-Grams: Design the Text Representation Before Training

Reviewed October 9, 2026. Examples use Python 3.12 and scikit-learn 1.8.0. All documents are synthetic fixtures.

A text classifier does not receive language directly. It receives the features produced by a particular analyzer. If that analyzer deletes a meaningful token, breaks an identifier, or represents a phrase only as separate words, the estimator cannot recover the missing evidence merely by being more sophisticated. Representation choices belong in the experiment plan, alongside the classifier and evaluation split.

This guide examines word and character n-grams by looking at their actual outputs. The goal is to make feature behavior observable before measuring prediction quality. A handful of invented messages can verify tokenization contracts; they cannot establish that one representation will work better for real users, languages, or writing styles.

1. Begin with a prediction question

Suppose a system routes short support messages to an account team or a shipping team. Words such as account and parcel could be useful, but actual messages may contain abbreviations, misspellings, product codes, and phrases whose meaning depends on local order. Decide which variations should preserve the route and which should change it. That decision provides concrete examples for testing an analyzer.

A word unigram records an individual token. A word bigram records two adjacent tokens produced by the analyzer. Character n-grams record short spans of characters. These definitions describe feature construction, not linguistic understanding. A bigram can represent a local distinction such as not delivered without giving a model a general ability to understand negation, reference, sarcasm, or an entire conversation.

Create a small inspection set before training: a normal message, a spelling variation, punctuation, a single-letter code, a phrase with reordered words, and a blank input. Write down what information each case must preserve. Such a fixture is especially useful when cleaning rules are changed by a different team from the team that evaluates the classifier.

2. Inspect the analyzer instead of guessing

The CountVectorizer reference documents the default word token pattern, n-gram range, and analyzer options. In the following fixture, default tokenization omits the single-letter token A and lowercases the remaining words. Changing the token pattern is an explicit decision to keep single-character word tokens.

from sklearn.feature_extraction.text import CountVectorizer

text = "A parcel is not delivered!"
default = CountVectorizer(ngram_range=(1, 2))
features = default.build_analyzer()(text)
assert "a" not in features
assert "not delivered" in features
assert "parcel is" in features
keep_single = CountVectorizer(token_pattern=r"(?u)\b\w+\b")
assert "a" in keep_single.build_analyzer()(text)
word = CountVectorizer(ngram_range=(2, 2))
assert word.build_analyzer()("not delivered") == ["not delivered"]
assert word.build_analyzer()("delivered not") == ["delivered not"]
print(features)

These assertions establish exactly what this configuration emits. They do not show that retaining A is desirable in every corpus. It might be a product identifier, a language token, or an uninformative article. The representation needs a domain-specific contract, rather than a cleaning rule chosen solely because it reduces vocabulary size.

Inspecting features also exposes interactions. Removing stop words before creating word bigrams changes adjacency: two words that were separated in the original sentence can become neighbors in the token stream. If a custom callable analyzer is supplied, inspect its own output rather than assuming every built-in preprocessing parameter still controls it. Save these inspections next to the model configuration so that later readers can reproduce the intended behavior.

3. Compare boundaries, not just feature counts

Character features can preserve overlapping pieces of a word when its spelling changes. That is a plausible reason to test them for noisy short messages; it is not a guarantee of robustness. A small typo may alter several overlapping spans, and unrelated strings can share many short spans. Character representations can also consume considerable space when the range is broad.

The char analyzer can include spans across spaces. The char_wb analyzer builds spans within word boundaries and pads word edges with spaces. Those padded spans are actual feature values, so trimming them during debugging can conceal a meaningful distinction. Treat printed representations with visible quotes as preferable to an unquoted list when investigating boundary behavior.

The table is a design comparison, not a ranking. The feature extraction guide provides the underlying bag-of-features context. The reader should assess each option against examples from the actual writing conditions expected at deployment.

4. Test unseen text with a frozen vocabulary

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer

training = ["parcel delayed", "account locked"]
word = CountVectorizer().fit(training)
unknown = word.transform(["completely novel"])
assert unknown.nnz == 0
char = CountVectorizer(analyzer="char_wb", ngram_range=(3, 3))
char.fit(training)
names = set(char.get_feature_names_out())
assert " pa" in names and "el " in names
changed = char.transform(["parcels delayed"])
assert changed.nnz > 0
assert changed.shape[1] == len(names)
cross = CountVectorizer(analyzer="char", ngram_range=(3, 3))
assert "a b" in cross.build_analyzer()("a b")
bounded = CountVectorizer(analyzer="char_wb", ngram_range=(3, 3))
assert "a b" not in bounded.build_analyzer()("a b")
assert np.all(changed.data >= 0)
print("Known character spans:", changed.nnz)

The all-zero word row means none of the analyzed terms are in this fitted vocabulary. It does not mean the message contains no information. A downstream classifier may still return a confident label because of its intercept or other learned behavior. Monitor zero rows, empty messages, and input lengths as representation diagnostics, while respecting privacy rules for storing actual text.

The character fixture retains known spans after adding a letter to parcel. That mechanical observation is much narrower than a claim about classification accuracy. Whether those spans help the route decision depends on training examples, labels, competing features, regularization, and evaluation data. The shape assertion verifies that new inputs use the same feature columns as training inputs.

5. Separate token construction from weighting

Counts and TF-IDF weighting answer different questions from the choice of word or character analyzer. A token construction rule determines what can become a feature. A weighting rule determines how the resulting features contribute numerically. A phrase that is never emitted cannot acquire a useful weight through TF-IDF.

The TfidfVectorizer reference documents these configuration layers. When comparing weighting choices, hold the analyzer fixed first. When comparing analyzers, record weighting and normalization explicitly. Changing both at once may be a valid experiment, but the result cannot isolate which change explains the difference.

Vocabulary and document-frequency information must be learned inside each training fold. Put the vectorizer and classifier in a pipeline, then pass the complete pipeline into cross-validation. Even when labels are not consulted, learning vocabulary from the evaluation set uses information about that set and changes the experiment. Freeze the selected representation before reporting a final test result.

6. Design the comparison around deployment

Use the same validation examples and scoring definition for candidate representations. Check whether messages from the same conversation, author, template, or organization appear on both sides of the split. Shared templates can make a text classifier look effective even when it cannot handle genuinely new writing. If the product predicts future messages, consider the order in which templates and vocabulary become available.

Evaluate relevant slices: long versus short messages, noisy versus clean input, language groups with adequate support, and cases where a human would request clarification. A single aggregate metric can hide representation failures affecting a small but important slice. Keep slice definitions fixed before comparison, and report their sample counts alongside the scores.

Finally, retain the analyzer configuration, package version, training vocabulary, label definitions, and a small non-sensitive regression fixture. Re-run the fixture when preprocessing changes. The outcome of this guide is a testable representation decision: inspect the emitted evidence, compare candidates under the intended split, and document the situations where the selected representation remains uncertain.

Sources and scope

The three linked scikit-learn 1.8 references support the analyzer and weighting APIs. The fixtures, inspection checklist, comparison table, and diagram are original instructional material. They demonstrate behavior on invented text and make no claims about production accuracy, benchmark superiority, or universal language coverage.