ffn-size-typically-4x-embedding-size

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-2.md

Created 2026-06-21T09:50:11+00:00

The feedforward network intermediate size (d_ffn) is typically 4 times the embedding dimension (d_emb), as used in GPT-2 and BERT.

Summary

In standard transformer models like GPT-2 and BERT, the internal feedforward layers are sized to be roughly four times wider than the model's main hidden representation. This 4x ratio is a widely followed design convention that directly determines how much compute and memory each layer consumes relative to the model's core size, so it anchors architecture decisions and resource planning.

Dependents

These beliefs depend on this one:

IN ffn-dimensions-follow-fixed-ratios-across-architectures — The transformer's feed-forward layers contain the majority of model parameters. A common dimensioning convention sets FFN size to 4x the hidden size (as seen in GPT-2 and BERT), and BERT specifically uses a fixed ratio of H/64 for attention heads — though these ratios are architectural conventions of specific models rather than universal structural constants across all architectures.