Unicode Normalization in Python: Compare Text Without Losing Meaning
Understand NFC, NFKC, and casefold in Python. Compare Unicode text, preserve original values, audit collisions, and separate normalization from decoding.
Checked on October 3, 2026. Examples were executed with Python 3.12.14, whose unicodedata reports Unicode Database 15.0.0. They use deliberately chosen strings, not a survey of real names or languages.
Two customer names look identical on a screen, yet an equality comparison says they differ. A search index misses one spelling. A developer then applies an aggressive cleaning function to every field, and previously distinct product codes collapse into the same key.
Both problems can arise from text normalization. The task is to choose which differences your application is allowed to ignore. Visual resemblance, equivalent Unicode representations, case-insensitive matching, and business identity are separate relationships.
Short answer: decode bytes using the correct encoding first. Use NFC when canonical equivalence is the intended comparison rule. Use NFKC only when compatibility distinctions may be removed. Add casefold when the application deliberately ignores case. Preserve the original text and inspect collisions before using transformed strings as unique identifiers.
Understand the layer that is failing
An encoding converts text to bytes and back. Normalization transforms an already decoded Unicode string. Changing NFC to NFKC cannot reliably repair a string that was decoded using the wrong encoding.
Python's Unicode HOWTO distinguishes Unicode code points from encoded bytes. For example, a UTF-8 byte sequence for an accented character must be decoded as UTF-8 before any comparison policy is applied. A replacement character introduced during lossy decoding may mean the original information is already unavailable.
| Symptom | Likely layer to inspect | Useful first action |
|---|---|---|
| Accented text becomes sequences such as “Ô plus another character | Encoding or decoding | Check the producer's encoding and inspect original bytes |
| Identical-looking accented strings compare unequal | Canonical representation | Inspect code points and try NFC on both strings |
| Fullwidth letters fail a deliberately width-insensitive search | Compatibility policy | Evaluate NFKC and its collision effects |
| “Straße” and “STRASSE” should match in a general caseless search | Case policy | Evaluate casefold, followed by normalization |
| A Latin letter resembles a Cyrillic letter | Script or confusable policy | Inspect code points; normalization alone is insufficient |
The correct diagnosis matters because each action loses or retains different information. Re-encoding a broken display string repeatedly can conceal the original decoding mistake. Removing every non-ASCII character can appear to improve matching while discarding meaningful distinctions across names and languages.
Canonical equivalence: one accent, two representations
The string “café” can end with the single code point U+00E9, or with U+0065 followed by combining acute accent U+0301. Those representations are canonically equivalent, but their Python code-point sequences are different.
import unicodedata as ud
composed = "caf\u00e9"
decomposed = "cafe\u0301"
assert composed != decomposed
assert len(composed) == 4 and len(decomposed) == 5
assert ud.normalize("NFC", composed) == ud.normalize("NFC", decomposed)
assert ud.normalize("NFD", composed) == decomposed
for value in (composed, decomposed):
print([f"U+{ord(char):04X}" for char in value])
print("Unicode Database:", ud.unidata_version)
The output exposes the sequence rather than relying on a font. len counts code points in a Python string; it is not a general count of user-perceived characters or a measure of screen width. Combining sequences and some emoji make those quantities diverge. Unicode's text segmentation specification defines grapheme-cluster boundaries for applications that need a closer model of user-perceived characters.
The Unicode normalization specification defines four forms. NFD uses canonical decomposition; NFC adds canonical composition. NFKD and NFKC additionally use compatibility decomposition. These are rules about Unicode representation, not a promise that every similar-looking string will become equal.
Compatibility normalization makes a stronger choice
Compatibility distinctions include some presentation forms, width variants, ligatures, and numeric symbols. Whether they should survive depends on the field. A search box may intentionally treat fullwidth “Ａ” as “A”; a catalog recording the exact original label may need to retain the difference.
The following outcomes were calculated with Python's normalization functions:
| Original value | NFC | NFKC | Consequence in this example |
|---|---|---|---|
| U+0065 plus U+0301: e with combining acute accent | é | é | Canonical representations converge |
| Ａ, fullwidth capital A | Ａ | A | Width distinction is removed by NFKC |
| ①, circled digit one | ① | 1 | Circled presentation is removed by NFKC |
| fi, the fi ligature | fi | fi | One compatibility character becomes two code points |
| ², superscript two | ² | 2 | Superscript distinction is removed by NFKC |
These transformations explain why NFKC can be useful and why it should not overwrite original text indiscriminately. In a mathematical expression, changing a superscript to an ordinary digit alters how the notation appears. In a search index, that transformation might be an accepted way to improve retrieval. Store a display value and a comparison key separately.
Normalization does not remove accents in general. NFC makes the representations of an accented letter consistent; it does not turn “é” into “e.” An accent-insensitive search requires an additional, explicit policy. That policy can collapse different words or names and should be evaluated for the languages in your dataset.
Casefold is a comparison policy, not translation
Python documents str.casefold as a more thorough caseless transformation than lower. In the example below, the German sharp s becomes “ss” under case folding. Ordinary lowercasing does not produce that same key.
import unicodedata as ud
def search_key(text):
if not isinstance(text, str):
raise TypeError("A search key requires a decoded string")
# An application-specific, compatibility-insensitive, caseless key.
normalized = ud.normalize("NFKC", text)
return ud.normalize("NFKC", normalized.casefold())
assert "Stra\u00dfe".lower() != "STRASSE".lower()
assert search_key("Stra\u00dfe") == search_key("STRASSE") == "strasse"
assert search_key("\uff21") == search_key("A") == "a"
assert search_key("caf\u00e9") == search_key("cafe\u0301")
assert search_key("caf\u00e9") != search_key("cafe")
assert search_key("a") != search_key("\u0430") # Latin vs Cyrillic
for text in ["Stra\u00dfe", "\uff21", "cafe\u0301"]:
assert search_key(search_key(text)) == search_key(text)
print("Comparison examples passed")
This function defines a policy for a search field. It is not an implementation of every Unicode identifier profile or the dedicated NFKC_Casefold mapping discussed in Unicode UAX #31. In particular, that dedicated mapping has additional behavior involving default-ignorable characters. Do not label a short normalization recipe as standards conformance without implementing the named standard.
The second normalization keeps the folded result normalized. The explicit examples also show what the policy does not do: it retains the accent and does not merge a Latin “a” with the Cyrillic letter U+0430. General Unicode case folding also does not replace locale-specific collation rules. If a product needs language-sensitive ordering or search, evaluate an appropriate collation system rather than assuming one string key settles every linguistic requirement.
Audit collisions before enforcing uniqueness
A transformation can be individually sensible and still unsuitable as an account ID. “Ａ,” “A,” and “a” all produce “a” under the example policy. If those strings identify three existing records, a normalized unique constraint would fail or a careless import could merge the records.
from collections import defaultdict
import unicodedata as ud
def search_key(text):
return ud.normalize("NFKC", ud.normalize("NFKC", text).casefold())
records = [
("record-1", "A"),
("record-2", "\uff21"),
("record-3", "a"),
("record-4", "caf\u00e9"),
("record-5", "cafe\u0301"),
]
groups = defaultdict(list)
for record_id, original in records:
groups[search_key(original)].append((record_id, original))
collisions = {key: rows for key, rows in groups.items() if len(rows) > 1}
assert sorted(len(rows) for rows in collisions.values()) == [2, 3]
for key, rows in collisions.items():
print(repr(key), rows)
The audit groups records, not just distinct original spellings. Two records with exactly the same spelling can still require investigation if the key is meant to be unique. Conversely, a search index can legitimately keep many records under one comparison key and return all of them.
Use stable record IDs to preserve identity. Retain the original value, derived key, transformation policy name, and software or Unicode version used. If you later change the policy, regenerate keys and compare collision groups before migrating production constraints.
Similar appearance does not establish identity
The Latin and Cyrillic letters in the previous test can appear very similar in some fonts. Their code points remain different after the chosen normalization. Unicode's security mechanisms specification treats confusable detection and script restrictions as separate mechanisms.
For an ordinary article search, returning similar spellings may improve usability. For a username registry, the same behavior raises a different design question: which names may coexist and how should users distinguish them? Make that policy explicit rather than automatically rewriting all visually similar characters into one script.
Even a collision-free normalized key does not prove that two records refer to the same person. Two people can share a name; one person can use several names. Text equivalence is only one input to record matching. Combine it with appropriate stable identifiers and domain evidence.
A practical workflow for imports and search
First, capture the source and decode strictly using its declared encoding. Investigate decoding errors instead of silently deleting bytes. Second, keep the original decoded field. Third, derive the comparison key using the rule appropriate to that field. Fourth, audit collisions and sample both matches and nonmatches.
Finally, test the entire path through storage, import, and search. Database collation rules may differ from your Python key function. A value that survives Python unchanged may compare equal under a database collation, or the reverse. Store and compare the derived key consistently if that key defines your application policy.
Before joining transformed IDs, read Pandas merge validation. Key normalization and merge validation solve different parts of the same workflow: one defines equality, while the other checks how many records each equality match creates.
Sources and reproducibility
- Python 3.12 Unicode HOWTO: encoding and Unicode strings.
- Python unicodedata: normalization API and database version.
- Python str.casefold: caseless transformation.
- Unicode UAX #15: canonical and compatibility normalization.
- Unicode UAX #29: text segmentation.
- Unicode UAX #31: identifier profiles and NFKC_Casefold.
- Unicode UTS #39: identifier and confusable mechanisms.
All three Python blocks are standalone. The examples and collision groups are original exercises checked locally. They establish behavior for the published strings, not suitability for every language or identifier system.