layernorm-degrades-gd-alignment-but-may-improve-ood
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s0-ix-chunk-1.md
Created 2026-08-24T17:11:05+00:00
LayerNorm degrades gradient descent alignment in self-attention layers because its averaging operation loses the input/target structural distinction needed for the dot-product construction, but may improve out-of-distribution generalization to larger-scale inputs and targets via implicit data normalization.
Summary
Adding LayerNorm to self-attention layers blurs the structural difference between what is being attended to and what is doing the attending, which weakens how cleanly gradient descent can optimize the scoring between them. The tradeoff is that this same averaging acts as a built-in stabilizer, potentially making the model more robust when it encounters inputs or targets far larger or more different than anything it saw during training.