llama-arch-swiglu-rope-rmsnorm

IN premise — entries/2026/06/21/wiki-LLaMA-chunk-1.md

Created 2026-06-21T09:50:09+00:00

Llama uses SwiGLU activation (not GeLU), rotary positional embeddings/RoPE (not absolute), and RMSNorm (not LayerNorm), differing from GPT-3's architecture

Summary

Llama's internal building blocks were chosen to be different from GPT-3's in three specific ways: how it sharpens signals between layers, how it encodes word order, and how it stabilizes inputs. This means the two models are not structurally interchangeable, and anyone trying to reimplement, fine-tune, or extend Llama has to work with these specific components rather than assuming the older GPT-3 defaults will apply.

Dependents

These beliefs depend on this one:

IN llama-optimized-architecture-for-data-scaling-strategy — Llama's architecture adopts proven refinements from prior independent research (SwiGLU from Shazeer 2020, RoPE from Su et al. 2021, RMSNorm from Zhang & Sennrich 2019) rather than introducing novel components, while its training data volume scaled dramatically from 1.4T to 40T tokens across four generations — a pattern consistent with concentrating effort on data scaling over architectural novelty.
IN post-transformer-refinements-stabilize-without-changing-fundamentals — Post-2017 architectural refinements — Pre-LN normalization (eliminating warmup), SwiGLU activation, RMSNorm, grouped-query attention — all improve training stability and inference efficiency without altering the fundamental self-attention mechanism, representing evolutionary optimization of a settled architecture.