gpt2-768-dim-12-heads-64-dhead

IN premise — summaries/2026-08-24/wiki-Transformer_deep_learning_architecture-chunk-2-chunk-1.md

Created 2026-08-24T17:11:24+00:00

GPT-2 (124M parameters) uses d_emb=768, n_heads=12, d_head=64, satisfying the constraint d_emb = n_heads × d_head (12 × 64 = 768), making W_O a square 768×768 matrix.

Summary

GPT-2 (the 124M version) splits its 768-dimensional internal representation evenly into 12 attention heads, each working on a 64-dimensional slice, which keeps every projection matrix square. This is a foundational architectural fact: all the other layers — feed-forward networks, layer norms, residual connections — are built to operate in this same 768-wide space, so getting these numbers right anchors the entire model's geometry.