bert-attention-heads-h-div-64-ffn-4h
IN premise — entries/2026/06/21/wiki-BERT_language_model-chunk-1.md
Created 2026-06-21T09:50:09+00:00
BERT's number of attention heads equals H/64 and feed-forward layer size equals 4H, where H is the hidden size — these are fixed ratios, not free parameters
Summary
In BERT, you can't independently choose the number of attention heads or the size of the feed-forward layer; both are locked to the hidden dimension at fixed ratios (heads at H divided by 64, feed-forward at 4 times H). This means scaling the model up or down is a single-knob decision, and any design choice that treats those components as free parameters will not match how BERT is actually constructed.
Dependents
These beliefs depend on this one:
- IN ffn-dimensions-follow-fixed-ratios-across-architectures — The transformer's feed-forward layers contain the majority of model parameters. A common dimensioning convention sets FFN size to 4x the hidden size (as seen in GPT-2 and BERT), and BERT specifically uses a fixed ratio of H/64 for attention heads — though these ratios are architectural conventions of specific models rather than universal structural constants across all architectures.