Memory, arithmetic intensity and reconciling estimates

1. Modern LLM Notebook Appendix B: activations, inference, training and checkpointing

   Modern LLM Notebook - evaluation and FLOP accounting

   Section 2 (attention \(6BSD\) plus \(BHS^2\), FFN \(2BSD\) plus \(2BSF\) to \(8BSD\), per-layer \(14BSD\) plus \(BNS^2\) with the \(S\)4096 dominant-term check), Section 5 (FP16 weights \(2P\), KV \(B\) times \(S\) times \(K\) times \(d_h\) times \(L\) times 2 times bytes with 7B scenarios and weights, KV and activations regimes), Section 6 (AdamW ledger with FP32 master, BF16 copy, \(m, v\) and grads plus BF16 activations and A100 fit) and Section 7 (checkpoint trade to \(O(\sqrt{L})\) with torch.utils.checkpoint API)

   Done
2. CS336 Lecture 2: intensity, roofline and checkpointing

   Stanford CS336 - resource accounting, inference and evaluation

   arithmetic_intensity_relu through arithmetic_intensity_matmul with H100 accelerator intensity and roofline_plots(), activation_checkpointing() with the \(O(\sqrt{L})\) rule, tensors_memory() with dtypes and the 2.3GB example, and optimizer() memory plus erees and adamcasson links

   Done
3. CS336 Lecture 10: inference intensity and throughput

   Stanford CS336 - resource accounting, inference and evaluation

   review_of_arithmetic_intensity() with \(2BDF\) over \(2BD\) plus \(2DF\) plus \(2BF\) to intensity \(B\) and the H100 295 bound, arithmetic_intensity_of_inference() with MLP \(6BTDF\) and attention \(4BSTD\) to \(BT\) and \(ST/(S + T)\) plus prefill \(S/2\) and generation below 1, and throughput_and_latency() with Llama 2 13B at \(B\)1, \(B\)64 and \(B\)256 plus the TTFT note

   Done
4. LLM Foundations Checkpoint: debug and measure contract

   LLM Foundations Checkpoint - evaluation and arithmetic slice

   Debug and measure bullets: reconcile analytical parameters, FLOPs, state and KV with profiles and separate prefill and decode across batch and context regimes

   Done

Focus on

• State the precision for every term. Mixed bf16 params and grads with fp32 master and moments is 12 bytes without master and 16 to 18 with it, so measure to confirm which ledger your loop keeps.
• Compare arithmetic intensity to accelerator intensity (H100 about 295). Below it you are memory-bound no matter how good the kernels are.
• Prefill grows with \(B\) times \(T\) for MLP and \(S/2\) for attention, while generation sits at \(B\) for MLP and below 1 for attention. Batching rescues MLP generation, not attention.
• Expect measured memory above the formula. Name allocator caching, fragmentation, graph, workspace and framework overhead before doubting the math.

Reproduce closed-book

Traps

• Quoting the formula as the fit result. torch.cuda.memory_allocated includes caching and fragmentation, so report both numbers and explain the gap.
• Assuming a bigger batch fixes generation attention. Its intensity stays below 1 because each sequence owns its KV, while batching only helps the MLP part.
• Comparing quality at unequal compute. Fix prompts, decoding, budgets and evaluator versions, then report quality with compute ranges.

Practice

Self-test

1. What is the Exercise 2 training total for the 13B config?

   Show answer Got it Missed

2. When is an \(XW\) matmul compute-bound on an H100?

   Show answer Got it Missed

3. How do prefill and generation intensities differ for MLP versus attention?

   Show answer Got it Missed

4. Measured memory is 10 percent above the formula. Name the suspects and one isolation per suspect.

   Show answer Got it Missed

Leads to

• 5.4 KV memory arithmetic Apply KV math to cache capacity and layout design.
• 5.5 Two regimes: prefill and decode Carry intensity regimes into separate prefill and decode measurement.

Comes after

• 4.4 Uncertainty, robustness and honest claimsCarry interval discipline into quality-compute comparisons with uncertainty ranges.
• 4.6 Counting parameters and FLOPsTurn FLOP counts into memory budgets and intensity regimes.

Resources for this unit

• Stanford CS336 - resource accounting, inference and evaluation
  Stanford Universityprimarycourse
  Use Spring 2026 Lectures 2, 10 and 12 for FLOPs, memory, arithmetic intensity, inference and evaluation. Reproduce the small calculations locally. Assignment 3 IsoFLOPs analysis is optional; do not require its hosted training API or costly experiments.
• Modern LLM Notebook - evaluation and FLOP accounting
  Walking Labsrequiredimplementation
  Use English notebook 25-evaluation and Appendix B FLOP and memory accounting. Build a 20-example evaluator with two slices, raw outputs and bootstrap intervals, and deliberately create one parser, prompt or judge failure.
• LLM Foundations Checkpoint - evaluation and arithmetic slice
  Roadmap-Finalsupportassessment
  Use the contamination, evaluator-bias, parameter, FLOP, activation, state and KV requirements as the assessment contract. Keep raw outputs and distinguish a verified task result from a broad model capability claim.