stage 07 · Core
LLM Foundations
Build the tokenizer-to-cached-generation path, test every causal and state boundary, and connect model quality claims to evaluation and arithmetic.
5 sections 35 units about 135 h 0/35
Sections
In dependency order
Capstone
Tokenizer-to-Cached-Generation Evidence Pack (8 to 12 h)
Keep one small corpus, tokenizer identity, model configuration and evaluation manifest across three artifacts. The result proves the whole causal-language-model mechanism while leaving large-scale training and serving to their dedicated tracks.
Milestone deliverables
- Tokenizer and corpus audit pack - deterministic byte-safe BPE, serialized identity, adversarial properties, split-before-pack pipeline, provenance manifest, deduplication report, contamination probe and reproducible restart
- Tiny decoder stack - blank-file embeddings, RoPE, RMSNorm, MHA plus GQA or MQA, SwiGLU, causal loss, AdamW, complete checkpoint state, four decoding policies and differential, causality, mask and resume tests
- Evaluation, arithmetic and cached-generation dossier - versioned task and parser, raw outputs, slices and uncertainty, evaluator failure case, parameter, FLOP and memory worksheet, full-prefix cache equivalence and separate prefill and decode measurements
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| One identity across artifacts | All three artifacts use one tokenizer hash, one model config and one evaluation manifest. | One command checks the hashes and fails if any artifact uses a different one. |
| Tokenizer and corpus audit pack | The section 1 mini-project meets Pass on every criterion. | It meets Strong on at least half its criteria, including reproducibility. |
| Tiny decoder stack | The section 2 and 3 mini-projects meet Pass, and their CS336 tests and your own tests pass in one run. | Resume is exact on CPU, and the attention oracle compares gradients in float64. |
| Evaluation, arithmetic and cached-generation dossier | The section 4 and 5 mini-projects meet Pass. | Every estimate sits next to a measurement, and cache equivalence covers MHA, GQA and MQA. |
| Rebuild from a clean clone | A README lists the commands that rebuild each artifact. | One script rebuilds every reported number from a fresh clone on a CPU in under an hour. |
| Honest limits | The dossier states what the tiny model and small corpus cannot show. | Each limit names the later track that addresses it. |
Capstone hours · 8 to 12
Begin the first section