Unit 5.6 · about 4 h
323 / 324Measuring time to first token and inter-token latency
One blended tokens-per-second number hides whether the prompt or the generation is slow. Split timing shows which phase to fix and whether the fix worked.
- measure
You can now
- Measure TTFT with prefill latency and decode ITL with throughput separately, using warmup and explicit synchronization.
- Compare regimes across batch, context and generated length and name the bottleneck in each.
- Diagnose misleading latency from async timing, missing warmup, tokenization in the timed region, output mismatch or phase aggregation.
Covers
- Time to first token and prefill latency
- Inter-token latency and decode throughput
- Batch, context and generated-length regime changes
- Warmup, synchronization and phase-specific timing
Study, in this order
3 steps
- Modern LLM Notebook 21 Section 9: split prefill and decode timingModern LLM Notebook - KV cache and inference acceleration
Section 9 code cell (50-token prefill timed once for TTFT, then 20 single-token decodes timed separately with mean ITL and the per-step plot)
- Raschka `gpt_with_kv_cache.py` `main()`: synchronized timing patternLLMs from Scratch companion - KV cache and GQA
main()(torch.cuda.synchronizebefore and after the run,time.timearound 200-token generation fromHello, I am, tokens/sec plusmax_memory_allocated)Depth · Read for the synchronization pattern only.
main()times the whole run at once, so add your own warmup and phase split from the focus points. - CS336 Lecture 10: metrics and batch trade-offsStanford CS336 - Lecture 10 Inference
landscape(TTFT, latency in seconds per token and throughput in tokens per second, training parallel versus inference sequential) andthroughput_and_latency(Llama 2 13B on H100 at batch 1, 64 and 256 with smaller batches favoring latency and larger favoring throughput, TTFT as a function of prefill)
Focus on
- TTFT comes from the prefill call plus the first sample. ITL is the mean over steady decode steps. Never average the two phases together.
- Warm up first and discard the cold runs: allocation, compilation and caches make the first iteration much slower than the rest.
- CUDA launches are async, so synchronize around each timed region. Without it the clock measures launch speed, not work.
- Fix everything else before comparing: same prompt, batch, lengths and seeds, tokenization outside the clock, and identical outputs on both sides.
Reproduce closed-book
Traps
- Timing tokenization with the model. Encode and decode outside the clock or tokenizer speed masquerades as model speed.
- Claiming a speedup with different outputs. Different lengths or sampling paths do different work, so verify identical tokens first.
- Reporting one blended rate. A fast prefill can hide a slow decode and misdirect the next optimization.
Practice
Raschka `ch04` KV-cache timing: uncached versus cached 200-token runs
main() in gpt_with_kv_cache.py versus gpt_ch04.py (Hello, I am prompt with 200 new tokens, synchronize-then-time, tokens/sec and max_memory_allocated; README table of 27 versus 144 versus 166 tokens/sec on M4 CPU)
Your numbers will differ by hardware. The checks are identical outputs, then your own phase-split repeat with warmup. The mini-project does the full two-regime matrix.
Self-test
-
Where do TTFT and ITL come from in the Modern 21 Section 9 cell?
The cell times one 50-token prefill call (the TTFT proxy; a full TTFT adds the first sample) and takes ITL as the mean of the 20 one-token calls. Mixing them hides the regime difference. -
CUDA timing without synchronization shows impossibly fast decodes. Why?
Kernel launches are async, so the clock measures launch instead of work. Synchronize before and after each region as the Raschkamain()does. -
The first timed run is 3x slower than repeats. What do you do?
Discard it as warmup (cold allocation, compilation and caches) and report the median or mean over repeats with fixed prompt, batch and lengths. -
The cached run is faster but its tokens differ from the baseline. Is that a valid speedup?
No. Verify identical tokens or logits first at a fixed seed. Different outputs mean different work or a correctness bug, so fix that before claiming anything.
Completion criterion
Stop when
You report separate TTFT and ITL with warmup and synchronization at two regimes, with identical outputs on both sides.
Mark passed, then open 5.7 Only mark this when you can do it without notes.Leads to
- 5.7 Beyond one cache: prefix sharing, paging and speculation Place sharing and speculation after solid measurements.
Comes after
- 5.3 Proving the cache correctTime prefill and decode separately once equivalence holds.
- 5.4 KV memory arithmeticCheck the arithmetic against measured memory and latency.
- 5.5 Two regimes: prefill and decodeMeasure the two regimes you just analyzed.
Resources for this unit
- LLMs from Scratch companion - KV cache and GQA Use ch04/03_kv-cache and ch04/04_gqa with the chapter model code. Add explicit per-layer ownership, position offsets, variable lengths, EOS compaction and MHA-versus-GQA cache arithmetic to the runnable comparison.
- Modern LLM Notebook - KV cache and inference acceleration Use the KV-cache portions of English notebooks 11-mla-kv-cache and 21-inference-acceleration. Skip MLA and advanced acceleration as core work. Recalculate cache sizes and verify timing after warmup and explicit synchronization.
- Stanford CS336 - Lecture 10 Inference Use the Spring 2026 inference lecture to connect KV arithmetic, phase behavior and batching vocabulary. Keep kernels, distributed serving and expensive performance experiments for the later tracks.
Also in this section
- How to Scale Your Model - inference chapter After cache equivalence and phase measurement pass, read the inference chapter for arithmetic intensity, parallelism and production-scale reasoning. Treat it as the handoff to inference and performance tracks, not required foundation work.
- AI Performance Engineering - inference-engine bridge Use Sections 4 and 5 only after this gate to choose the next inference, KV-cache, scheduling, quantization or distributed-serving resource. Do not treat its GPU and kernel material as unfinished LLM Foundations work.
- No Magic - micro KV or paged-cache simulation Use one small CPU simulation such as microkv.py or micropaged.py only to visualize state ownership. Do not present Python timing as GPU or serving-performance evidence.