llama3-rope-theta-500k-vocab-128k

IN premise — entries/2026/06/21/wiki-LLaMA-chunk-1.md

Created 2026-06-21T09:50:09+00:00

Llama 3 uses RoPE with θ=500,000 for positional embeddings and has a vocabulary size of 128,000

Summary

Llama 3 is built around two key design choices: it encodes word position using a rotary scheme tuned for long-range context, and it recognizes 128,000 distinct text units rather than the smaller vocabularies of earlier models. These are the baseline facts any analysis of the model's length handling, token efficiency, or multilingual behavior depends on.

Dependents

These beliefs depend on this one:

IN positional-encoding-is-a-diverse-unsettled-design-choice — Multiple competing positional encoding strategies (absolute sinusoidal, RoPE, ALiBi) exist because self-attention is inherently permutation-invariant — each approach trades off different properties.