Text as bytes: Unicode, encodings and normalization

A tokenizer receives encoded text, not the glyphs on your screen. A silent decoding or normalization change can alter document identity before BPE even starts.

explain

You can now

Explain why code-point, UTF-8 byte and token counts differ, including combining marks and control characters.
Defend a normalization and invalid-byte policy, and state which distinctions it cannot preserve.
Predict whether a round trip returns the original text or only its normalized form.

Covers

Unicode code points, byte encodings and invalid-byte policy
Normalization choices and distinctions they erase
Character, byte and token counts

Study, in this order

3 steps

CS336 Assignment 1: Unicode and encodings
Stanford CS336 Assignment 1 - tokenizer contract

Sections 2.1 and 2.2, including Problems (unicode1) and (unicode2) (pp. 3 to 5); Section 2.6.2, Decoding text (p. 11)

Done
Canonical and compatibility normalization
Python unicodedata - normalization forms

unicodedata.normalize: NFC, NFD, NFKC and NFKD, with the cedilla and Roman numeral examples; unicodedata.unidata_version

Done
Normalization and pre-tokenization
Hugging Face LLM Course - tokenizer translation

Normalization, Pre-tokenization and SentencePiece: normalize_str and the repeated-space examples

Depth · Stop before Algorithm overview. Use the library calls to inspect behavior, not to replace your later blank-file implementation.

Done

Focus on

Python len(str) counts code points, not visible grapheme clusters. UTF-8 uses one to four bytes per Unicode scalar value; token counts depend on the tokenizer.
NFC merges canonically equivalent spellings. NFKC also collapses compatibility distinctions; lowercasing and accent stripping are separate choices.
Rejecting malformed UTF-8, replacing it and preserving raw bytes are different contracts. The CS336 text decoder uses replacement, not a reversible arbitrary-byte interface.
Record the encoding, error policy, normalization form and Unicode database version before comparing counts or content hashes.

Reproduce closed-book

Without notes, make a count table for 'é', 'e\u0301', '牛' and chr(0): code points and UTF-8 bytes before and after NFC. Explain why the first two become identical under NFC, but a token count still needs a named tokenizer.

Traps

Decoding one byte at a time. A UTF-8 character can span several bytes, so concatenate token bytes before decoding.
Promising raw-text round trips after lowercasing or normalization. Lost distinctions cannot be recovered from token IDs alone.
Using errors='ignore' as cleanup. It deletes malformed bytes and can make distinct raw documents look identical.

Practice

Hugging Face LLM Course: cased versus uncased normalization

Try it out! in Normalization: pass 'Héllò hôw are ü?' to bert-base-cased and bert-base-uncased normalizers

Write down the erased distinctions. Then check the count table from Reproduce with Python and state an explicit invalid-byte policy for your corpus.

Self-test

Why can 'é' and 'e\u0301' look identical but have different lengths?
What happens to the Roman numeral 'Ⅰ' under NFC and NFKC?
Does complete byte vocabulary coverage make replacement decoding reversible for malformed input?
Is chr(0) an empty string or a text terminator in Python?

Completion criterion

Stop when

Your count table agrees with Python, and you can predict the cased/uncased outputs and explain exactly what your decoding and normalization policy loses.

Mark passed, then open 1.2 Only mark this when you can do it without notes.

Leads to

1.2 Byte-level BPE: training, merges and tie-breaking Use bytes as the complete base vocabulary.
1.6 Provenance, deduplication and contamination Keep raw identity separate from normalized identity.

Resources for this unit

Stanford CS336 Assignment 1 - tokenizer contract
Stanford Universityrequiredassessment
Read the Assignment 1 tokenizer handout and use the named unicode1, unicode2, train_bpe, tokenizer and tokenizer_experiments specifications plus public tests as an acceptance contract. Implement independently in your own repository and do not copy or modify course solutions.
Hugging Face LLM Course - tokenizer translation
Hugging Faceoptionalreference
After the blank-file tokenizer passes, use the tokenizer chapters to map vocabulary, special tokens, fast tokenizers and batching into library APIs. Do not replace the deterministic implementation gate with a pretrained tokenizer call.
Python unicodedata - normalization forms
Python Software Foundationreferencereference
Read unicodedata.normalize for canonical and compatibility equivalence, and unidata_version for recording the normalization database version.

Also in this section

LLMs from Scratch companion - tokenization and data loading
Sebastian Raschkaprimaryimplementation
Work through the ch02 main notebook, 05_bpe-from-scratch and 04_bonus_dataloader-intuition. Rebuild byte-level BPE, serialization and packed windows in blank files, then add multilingual, control, whitespace and chunk-boundary properties.
Modern LLM Notebook - tokenization and data experiments
Walking Labssupportimplementation
Run English notebooks 01-tokenizer-basics and 02-bpe-tokenizer. Use only the quality, deduplication and mixture examples from 15-data-engineering. Change the corpus and tie cases, then verify every quantitative claim independently.
Tokenizer and Corpus Pipeline - reduced foundation slice
Roadmap-Finalsupportproject
Use the byte, identity, split, deduplication, contamination, packing, manifest and restart correctness requirements. Keep the corpus small and stop before distributed processing or internet-scale filtering.
Hugging Face Datasets - streaming, sharding and resume
Hugging Facereferencereference
Read Shard, Interleave, Save a dataset checkpoint and resume iteration, and Export. Use tiny local data and inspect the warning that shuffle buffers are lost on resume; do not download web-scale corpora or upload to the Hub.