llama3-405b-architectural-parameters

IN premise — summaries/2026/08/24/wiki-LLaMA-chunk-1.md

Created 2026-08-24T17:11:14+00:00

Llama 3 405B has 126 layers, 16,384 model dimension, 53,248 FFN dimension, 128 attention heads, 8 key/value heads (GQA), RoPE(θ=500,000), SwiGLU activation, and 128,000 vocabulary size.

Summary

This is the concrete spec sheet for the Llama 3 405B model, recording how wide its internal processing stages are, how it tracks position to handle long contexts, and which efficiency shortcuts it uses when generating text. It matters because every downstream claim about the model's capabilities, compute cost, or training requirements ultimately depends on these structural numbers being accurate.