pre-ln-eliminates-warmup-requirement
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-1.md
Created 2026-06-21T09:50:11+00:00
Pre-LN (layer normalization before attention/FFN sublayers) stabilizes training and eliminates the need for learning rate warmup, unlike the original post-LN design from 2017.
Summary
Placing layer normalization before the attention and feed-forward blocks, rather than after them, keeps the network's activations from blowing up during early training. In practice, this means you can start training at full learning rate right away without a careful warmup schedule, making the setup simpler and less fragile than the original 2017 Transformer recipe required.
Dependents
These beliefs depend on this one:
- IN post-transformer-refinements-stabilize-without-changing-fundamentals — Post-2017 architectural refinements — Pre-LN normalization (eliminating warmup), SwiGLU activation, RMSNorm, grouped-query attention — all improve training stability and inference efficiency without altering the fundamental self-attention mechanism, representing evolutionary optimization of a settled architecture.