ffn-layers-contain-most-transformer-parameters
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-2.md
Created 2026-06-21T09:50:11+00:00
Feed-forward layers (not attention layers) contain the majority of parameters in a transformer model.
Summary
In a transformer, the bulk of its learned weights live in the feed-forward blocks rather than in the attention heads. This matters because most of the model's capacity and stored knowledge sits in those dense linear layers, so decisions about compression, pruning, or targeted modification should focus there first.
Dependents
These beliefs depend on this one:
- IN ffn-dimensions-follow-fixed-ratios-across-architectures — The transformer's feed-forward layers contain the majority of model parameters. A common dimensioning convention sets FFN size to 4x the hidden size (as seen in GPT-2 and BERT), and BERT specifically uses a fixed ratio of H/64 for attention heads — though these ratios are architectural conventions of specific models rather than universal structural constants across all architectures.