ffn-dimensions-follow-fixed-ratios-across-architectures

IN derived (depth 1)

Created 2026-06-21T13:06:41+00:00 · Reviewed 2026-06-21T14:41:08+00:00

The transformer's feed-forward layers contain the majority of model parameters. A common dimensioning convention sets FFN size to 4x the hidden size (as seen in GPT-2 and BERT), and BERT specifically uses a fixed ratio of H/64 for attention heads — though these ratios are architectural conventions of specific models rather than universal structural constants across all architectures.

Summary

In many popular transformer designs like GPT-2 and BERT, the feed-forward layers are set to about four times the hidden width, and BERT further ties its number of attention heads to that same width by a fixed fraction. These are deliberate design choices made by those specific model families, not universal laws of transformer construction, so they cannot be assumed to hold in every architecture you encounter.

Justifications

SL — Three independent architectural observations combine into a single structural invariant across transformer families

Antecedents (all must be IN):

IN ffn-layers-contain-most-transformer-parameters — Feed-forward layers (not attention layers) contain the majority of parameters in a transformer model.
IN ffn-size-typically-4x-embedding-size — The feedforward network intermediate size (d_ffn) is typically 4 times the embedding dimension (d_emb), as used in GPT-2 and BERT.
IN bert-attention-heads-h-div-64-ffn-4h — BERT's number of attention heads equals H/64 and feed-forward layer size equals 4H, where H is the hidden size — these are fixed ratios, not free parameters