pre-ln-vs-post-ln-transformer

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture.md

Created 2026-06-21T09:50:10+00:00

Pre-LN Transformers apply layer normalization before attention and feedforward layers, which stabilizes training and removes the need for learning rate warmup, unlike the original post-LN design.

Summary

Using layer normalization before the attention and feedforward blocks (rather than after them) makes transformer training much more stable from the start, so you can skip the gradual learning-rate warmup schedule that the original design required. In practical terms, this means the system can be trained at full learning rate immediately without the extra tuning overhead, reducing the risk of early divergence.