ffn-intermediate-4x-embedding

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-3.md

Created 2026-06-21T09:55:55+00:00

The feedforward network intermediate size is conventionally 4x the embedding size (d_ffn = 4 * d_emb) in GPT-2 and BERT families.