Unit 3.7 · about 3 h
310 / 324Stopping rules and reproducible generation
Unbounded generation never terminates cleanly, and unseeded sampling cannot be debugged. Stopping, seeds and validation turn demos into experiments that can be rerun and compared.
- implement
You can now
- Implement EOS, stop-sequence, maximum-length and finished-batch handling in a
generateloop. - Make sampling exactly reproducible from a recorded seed and validate the outputs.
- Compare prompts and chat templates while holding decoding fixed, and state each choice in config.
Covers
- EOS, stop sequences, maximum length and finished batches
- Sampling RNG, reproducibility and output validation
- Prompt and chat-template comparison
Study, in this order
5 steps
- Raschka `gpt_generate.py`: bounded generation with `eos_id`LLMs from Scratch companion - training and generation
generate(): context-window slicing of the running sequence, themax_new_tokensloop, theeos_idearly break and appending each sampled token - Modern LLM Notebook 20-generation: complete pipeline and orderModern LLM Notebook - configuration, loss and generation
Section 9 (complete
generatewith penalty, temperature, top-\(k\) and top-\(p\),eos_idand seed) and the Summary execution order: penalty, temperature, top-\(k\) and top-\(p\), sample, theneos - PyTorch reproducibility notes: seeding for exact rerunsPyTorch saving, loading and reproducibility references
Controlling sources of randomness (seed torch, Python and other libraries together); the Raschka scripts set
torch.manual_seed(123)before generation - Modern LLM Notebook 20-generation and 10-training-loss: templatesModern LLM Notebook - configuration, loss and generation
Notebook 20 Section 8 (chat templates stitch turns into model text, wrong template degrades quality,
apply_chat_template) with notebook 10 Section 2 (templates decide which tokens the loss sees) - Hugging Face LLM Course: inference translationHugging Face LLM Course - library translation
Chapter 1, Inference with LLMs (chapter1/8): map generation configuration and stopping behavior into library APIs
Depth · Open only after the blank-file loop passes fixed-seed checks; the from-scratch tests stay the authority.
Focus on
- Check EOS every step and cap length regardless. EOS is model-controlled and may never arrive.
- Seed every RNG in the path and record the seeds with the output. One unseeded source breaks reruns.
- Validation is cheap and explicit: EOS present, length within cap, template matching training, decoded text well-formed.
- Hold decoding fixed when comparing prompts or templates, or policy differences will masquerade as prompt effects.
Reproduce closed-book
generate loop with max_new_tokens, eos_id, a stop-sequence check and seed control that returns identical tokens on every rerun.Traps
- Stopping on EOS alone. A model that never emits EOS generates until the budget or the wallet runs out.
- Seeding torch but not the Python RNG or sampler state. Multinomial draws still differ and the mismatch looks like a model bug.
- Grading a chat model with the wrong template and blaming the weights. Format mismatch degrades answers without changing any parameter.
Practice
Modern LLM Notebook 09-model-config Exercise 1: compare generation configs
Exercise 1: open two model generation_config files, write three differences with explanations for temperature, top_p, top_k, max_new_tokens and stopping entries
Then encode your own loop's stopping and sampling choices as one such config and demonstrate the EOS plus max-length stops with a fixed seed.
Self-test
-
Why does the loop need both
eos_idandmax_new_tokens?EOS is emitted by the model and may never come, so it cannot bound cost. The cap bounds tokens, latency and cost on every call no matter what the model does. -
Same seed, different outputs on rerun. Name two suspects.
An unseeded RNG such as Pythonrandomor a sampler without generator control, and a nondeterministic kernel or op ordering. Both must be fixed before blaming the policy code. -
What failure does
apply_chat_templateprevent?Train and serve format mismatch. The template wraps turns exactly as training saw them; a hand-rolled prompt can silently use the wrong markers and degrade quality. -
A run uses
temperature0.6,top_p0.9,top_k50 and repetition penalty 1.1. In what order do they apply?Penalty first, then temperature, then top-\(k\) and top-\(p\) truncation, then sampling, then the EOS check, as the notebook summary lists. The order matters: temperature changes the probabilities, so applying it before or after top-\(p\) changes which tokens the nucleus keeps. Record the order in the config.
Completion criterion
Stop when
Your loop stops on EOS and on the cap, reruns bit-identically from a recorded seed, and every stopping and template choice is stated in one config.
Mark passed, then open 4.1 Only mark this when you can do it without notes.Leads to
- 5.1 Why cache: recomputation in autoregressive decoding Keep the same stopping and sampling contract while removing recomputation.
- 4.2 Task contracts and automatic metrics Feed validated, reproducibly generated outputs into task metrics.
Comes after
- 3.5 Checkpoints that really resumeExtend determinism from training into generation.
- 3.6 Decoding policies: from greedy to nucleus samplingBound the policies with stopping rules and exact reproducibility.
Resources for this unit
- LLMs from Scratch companion - training and generation Use the ch05 training and generation code for the small pretraining loop. Add full checkpoint state, explicit loss masks and resume-equivalence tests that the base example does not supply.
- Modern LLM Notebook - configuration, loss and generation Use English notebooks 09-model-config, 10-training-loss and 20-generation to visualize configuration ownership, shifted loss and decoding policies. Change edge cases and cross-check probability calculations against your own implementation.
- PyTorch saving, loading and reproducibility references Use the state-dict and reproducibility notes to enumerate all state needed for the supported resume guarantee. Test model, optimizer, schedule, scaler, data order and RNG state across an interruption.
- Hugging Face LLM Course - library translation Only after the blank-file loop passes, map tokenization, model configuration, data collation, generation configuration and stopping behavior into Transformers APIs. Keep the from-scratch tests as the authority.
Also in this section
- Stanford CS336 Assignment 1 - training and decoding contract Use cross_entropy, adamw, adamw_accounting, learning_rate_schedule, gradient_clipping, data_loading, checkpointing, training_together and decoding specifications plus public tests. Debug on CPU and implement independently in your own repository.
- PyTorch Automatic Mixed Precision recipe Use the
autocast,GradScaler, unscale-before-clip, scaler checkpointing and inf-NaN troubleshooting sections to add mixed precision to the blank-file loop. Keep the CS336 clipping contract as the authority for the clipping rule.