Unit 1.5 · about 4 h
294 / 324From documents to training examples: packing and loss masks
Packing changes which tokens become context and which predictions enter the loss. An EOS token marks a boundary, but it does not enforce attention isolation or remove an unwanted target.
- derive
- implement
You can now
- Derive shifted inputs, targets and valid-loss positions for packed documents, including EOS and padding.
- Implement packing with document lineage, a declared truncation/tail policy and separate attention and loss masks.
- Keep split ownership fixed before fitting a tokenizer or packing, so evaluation text cannot become training context.
Covers
- Documents, boundary tokens, truncation and packing
- Causal loss masks across packed document boundaries
- Split-before-fit and split-before-pack discipline
Study, in this order
3 steps
- Shifted targets and sliding windowsLLMs from Scratch companion - tokenization and data loading
Section 2.4: EOS between independent sources; Section 2.6, Data sampling with a sliding window:
GPTDatasetV1and stride/context-length examplesDepth · Stop before Section 2.7. The single-text loader is a baseline; it does not supply document isolation or split-safe corpus construction.
- Data loader intuition with integersLLMs from Scratch companion - tokenization and data loading
Data sampling with a sliding window with number data:
GPTDatasetV1, thenmax_length=4withstride=1andstride=4Depth · Stop before Finally, a data loader with shuffling. Track indices and the last omitted tail rather than running the full number file.
- Split ownership and packed boundary contractTokenizer and Corpus Pipeline - reduced foundation slice
Core requirements: split assignment before leakage-causing transforms and deterministic sample/pack construction with document-boundary/loss-mask policy; Architecture constraints: evaluation data cannot influence vocabulary/filters/mixture; Correctness requirements: fixed split ownership and packed masks
Depth · Apply these requirements to a few local documents. Do not implement the full project or distributed corpus processing.
Focus on
- Declare whether packed documents may attend to earlier documents. Continuous-stream training and isolated-document training are different objectives; neither policy is created by EOS alone.
- For isolated documents, attention allows \(j \le i\) only within the same document. The loss mask separately excludes padding targets and cross-document next-token transitions.
- Attach EOS to the document it ends. Predicting that EOS is valid; predicting the next document's first token from that EOS is masked under the isolated policy.
- Split raw documents or duplicate groups before fitting and packing. Record token spans back to document IDs, and account for every retained, padded or deliberately dropped tail token.
Reproduce closed-book
EOS=9 into \(z=[11,12,9,21,22,9]\). For \(x=z[:-1]\) and \(y=z[1:]\), derive \(y=[12,9,21,22,9]\) and isolated-document loss mask \([1,1,0,1,1]\). Build the lower-triangular same-document attention mask for \(x\) and explain which prediction it does not remove by itself.Traps
- Masking EOS as if it were padding. The last content token should usually learn to predict its own document's EOS.
- Using only a block-diagonal attention mask. The shifted label after one document's EOS can still be the next document's first token unless loss is masked too.
- Splitting after packing or overlapping-window construction. The same document can enter both splits, even when their packed rows differ.
Practice
The section mini-project covers this: split-before-fit and split-before-pack, document-span lineage, EOS/tail handling, and independently checked shifted loss and attention masks.
Self-test
-
In Reproduce, which target is excluded and why?
Target 21 at \(x\) position 2 is excluded: its context token is \(A\)'s EOS, but 21 belongs to \(B\). Predictions of either document's EOS remain valid. -
Does masking a loss position stop that token from being used as context?
No. Loss masking removes a prediction from the objective. Attention masking controls which input tokens can influence later representations. -
With ten tokens, context length four and stride four, how many complete shifted windows does the notebook loader produce?
Two, starting at 0 and 4. Each needs five stream tokens to make four input-target pairs; the remaining suffix cannot form another complete window. -
Why split before fitting the tokenizer if no evaluation examples enter model training?
Fitting on evaluation text still uses held-out distribution information to choose the representation. Fit on training data unless a separate, declared evaluation protocol permits otherwise.
Completion criterion
Stop when
Your packer matches the hand-derived inputs, targets and both masks on two documents, padding and an overlength document, with no document ID shared across splits.
Mark passed, then open 1.6 Only mark this when you can do it without notes.Leads to
- 3.1 The next-token objective: shifts, masks and perplexity Use the valid-target mask in the next-token loss.
- 2.2 Scaled dot-product attention from scratch Enforce causal and document visibility in attention.
Comes after
- 1.4 A tokenizer you can trust: round trips, identity and streamingBuild training examples from a stable token stream.
Resources for this unit
- LLMs from Scratch companion - tokenization and data loading Work through the ch02 main notebook, 05_bpe-from-scratch and 04_bonus_dataloader-intuition. Rebuild byte-level BPE, serialization and packed windows in blank files, then add multilingual, control, whitespace and chunk-boundary properties.
- Tokenizer and Corpus Pipeline - reduced foundation slice Use the byte, identity, split, deduplication, contamination, packing, manifest and restart correctness requirements. Keep the corpus small and stop before distributed processing or internet-scale filtering.
Also in this section
- Stanford CS336 Assignment 1 - tokenizer contract Read the Assignment 1 tokenizer handout and use the named unicode1, unicode2, train_bpe, tokenizer and tokenizer_experiments specifications plus public tests as an acceptance contract. Implement independently in your own repository and do not copy or modify course solutions.
- Modern LLM Notebook - tokenization and data experiments Run English notebooks 01-tokenizer-basics and 02-bpe-tokenizer. Use only the quality, deduplication and mixture examples from 15-data-engineering. Change the corpus and tie cases, then verify every quantitative claim independently.
- Hugging Face LLM Course - tokenizer translation After the blank-file tokenizer passes, use the tokenizer chapters to map vocabulary, special tokens, fast tokenizers and batching into library APIs. Do not replace the deterministic implementation gate with a pretrained tokenizer call.
- Python unicodedata - normalization forms Read
unicodedata.normalizefor canonical and compatibility equivalence, andunidata_versionfor recording the normalization database version. - Hugging Face Datasets - streaming, sharding and resume Read Shard, Interleave, Save a dataset checkpoint and resume iteration, and Export. Use tiny local data and inspect the warning that shuffle buffers are lost on resume; do not download web-scale corpora or upload to the Hub.