The modern decoder block: residuals, RMSNorm and SwiGLU

1. Normalization, gating and norm placement

   Modern LLM Notebook - decoder construction path

   Component Upgrades: LayerNorm and RMSNorm hand calculations, RMSNorm full implementation, The SwiGLU gating mechanism, Full SwiGLU FFN implementation, and Post-Norm versus Pre-Norm comparison

   Depth · Read the formulas and code. Skip the eight-layer gradient demonstration; a single random example is not a general stability proof.

   Done
2. GELU FFN baseline

   LLMs from Scratch companion - attention and GPT construction

   Section 4.3, Implementing a feed forward network with GELU activations

   Done
3. Initialization and modern block specifications

   Stanford CS336 Assignment 1 - model contract

   Section 3.3.1, Parameter Initialization, printed p. 17; section 3.4 through 3.4.2, printed pp. 19 to 22; section 3.5 through problem (transformer_block), printed p. 26

   Done
4. Depth-scaled residual initialization

   nanoGPT - residual projection initialization

   MLP and Block classes; GPT.__init__ from self.apply(self._init_weights) through the c_proj.weight initialization loop; GPT._init_weights

   Depth · Use this as a contrasting GPT-2 recipe, not a replacement for the CS336 test architecture. Skip pretrained loading, optimizer setup and generation.

   Done

Focus on

• Write \(y=x+\operatorname{Attn}(\operatorname{RMSNorm}(x))\), then \(z=y+\operatorname{FFN}(\operatorname{RMSNorm}(y))\). Normalize the branch input, not the residual sum; final stack normalization comes later.
• LayerNorm subtracts the feature mean and divides by feature standard deviation. RMSNorm divides by \(\sqrt{\operatorname{mean}(x^2)+\mathrm{eps}}\), without centering, and uses one learned gain per feature.
• SwiGLU is \(W_{\mathrm{down}}(\operatorname{SiLU}(W_{\mathrm{gate}} x) \odot (W_{\mathrm{up}} x))\). Its gate is not bounded to \([0, 1]\). Three bias-free matrices cost \(3DF\) parameters; a two-matrix GELU FFN costs \(2DF\) at the same hidden width.
• CS336 specifies truncated-normal component weights and unit RMSNorm gains. nanoGPT instead uses base \(\mathrm{std}=0.02\) and residual-projection \(\mathrm{std}=0.02/\sqrt{2L}\); that scales initial branch weights, not the residual identity path.

Reproduce closed-book

Traps

• Normalizing across tokens instead of the feature axis. That couples positions and can leak future information even when attention is causal.
• Squaring low-precision activations before upcasting. Overflow has already happened; accumulate RMS statistics in float32 first.
• Copying the notebook's explanation that a SwiGLU gate is a probability. SiLU can be negative or exceed one; the sigmoid factor alone is bounded.

Practice

Self-test

1. For \(x=[1, 3, 5, 7]\), unit gains and negligible \(\epsilon\), how do LayerNorm and RMSNorm differ?

   Show answer Got it Missed

2. Why does SwiGLU often use \(F\) near \(8D/3\) instead of \(4D\)?

   Show answer Got it Missed

3. What is the clean gradient path supplied by \(x+f(\operatorname{norm}(x))\)?

   Show answer Got it Missed

4. In nanoGPT, why are both attention and MLP c_proj weights scaled with layer count?

   Show answer Got it Missed

Leads to

• 2.7 Assembling and verifying a decoder Stack these blocks and add the final norm and vocabulary head.
• 3.4 Keeping training numerically healthy Track activation and gradient behavior during training.

Comes after

• 2.5 RoPE: positions as rotationsInsert rotary attention into the pre-norm residual block.

Resources for this unit

• Modern LLM Notebook - decoder construction path
  Walking Labsprimarycourse
  Work through English notebooks 03-embedding, 04-position-encoding, 05-transformer-block, 06-mini-gpt and the selected normalization, RoPE, GQA and SwiGLU comparisons in 08-gpt2-to-modern-models. Re-derive shapes and rerun with changed dimensions.
• LLMs from Scratch companion - attention and GPT construction
  Sebastian Raschkarequiredimplementation
  Use the ch03 attention notebooks and ch04 model code. Rebuild the model without a high-level Transformer module and compare one forward and backward path with an independent small reference.
• Stanford CS336 Assignment 1 - model contract
  Stanford Universityrequiredassessment
  Use the named linear, embedding, rmsnorm, positionwise_feedforward, rope, softmax, scaled_dot_product_attention, multihead_self_attention, transformer_block, transformer_lm and transformer_accounting specifications and public tests. Keep implementation independent and respect the course AI and honor-code boundary.
• nanoGPT - residual projection initialization
  Andrej Karpathysupportimplementation
  Read model.py: MLP, Block, GPT.__init__'s special scaled initialization of c_proj.weight, and GPT._init_weights. Contrast the depth-dependent residual initialization with CS336's component initialization; do not adopt the training or generation code.