LLaMA

38 beliefs (38 IN, 0 OUT)

This topic covers Meta's Llama family of large language models, tracing their development from the original 65B-parameter LLaMA announced on 24 February 2023 (llama-first-release-24-feb-2023-65b) through six successive generations to Llama 4 in April 2025 (llama-timeline-six-generations-feb-2023-apr-2025), with a successor, Muse Spark, released by Meta Superintelligence Labs in April 2026 (muse-spark-llama-successor-april-2026). The family matters as the most widely studied open-weight model line in the NLP literature, serving as a substrate for interpretability work, deployment research, and licensing debate. Architectural scale grew steadily: training tokens went from 1.4T in Llama 1 to 15T in Llama 3 to 40T in Llama 4 Scout (llama-training-data-sizes-progression), context windows expanded from 2K to 10M tokens (llama-context-window-progression), and Llama 4 introduced Mixture-of-Experts with 17B active parameters spread across 16 (Scout) or 128 (Maverick) experts (llama4-moe-scout-maverick-params). Llama 4 also became the first natively multimodal entry in the line, whereas Llama 3.2 had bolted on a separate vision encoder (llama-4-first-natively-multimodal-in-line). Code Llama, a fine-tuned variant, was trained on 500B general code tokens plus 20B long-context tokens, with a Python-specialized branch on 100B Python-only tokens (code-llama-training-token-recipe).

On the technical side, the beliefs capture several design choices that distinguish Llama from earlier transformer families: RMSNorm replaces LayerNorm for computational simplicity (rmsnorm-llama-alternative-to-layernorm), RoPE with theta=500,000 handles positional encoding in Llama 3 (llama3-rope-theta-500k-vocab-128k), and the vocabulary expanded from 32K SentencePiece tokens in Llama 2 (llama2-7b-architecture-specs) to 128K in Llama 3 (llama3-rope-theta-500k-vocab-128k). Llama 3.1 introduced a 400B MoE model (llama-3-1-400b-moe-model-jul-2024), and Llama 4's MoE design is further contextualized by the claim that compute routing operates at both intra-model (expert activation per token) and inter-model (task-level model selection) granularities (compute-routing-operates-at-multiple-architectural-granularities). Deployment research builds on this: frontier-model serving benefits from a stacked optimization approach combining MoE sparsity, speculative decoding, PagedAttention KV-cache management, and Flash Attention tiling (production-deployment-requires-full-optimization-stack), while llama.cpp provides a CPU-only inference path with the GGUF quantization format (llama-cpp-gguf-cpu-inference). Llama also appears as an experimental substrate in cognitive and interpretability studies, including bilingual-pair filtering constrained to single LLaMA-2 BPE tokens for translation measurement (bilingual-pair-single-token-filter-llama2-vocab), single-layer cGD simulations in Shen et al. (cgd-single-random-middle-layer), validation of the Linear Representation Hypothesis across 27 concepts on LLaMA-2 7B (lrh-validated-on-llama2-7b-27-concepts), and a finding that primacy bias in list processing emerges only at around 13B parameters (llama2-primacy-bias-threshold-13b).

The licensing and distribution story is a recurring thread. Llama 2 (July 2023) was the first generation to permit commercial use, but only for entities under 700 million daily active users and under an acceptable-use policy (llama-2-first-commercial-use-jul-2023, llama2-license-700m-dau-restriction). Llama 3.1 formalized a two-document model pairing a Community License Agreement with a separate Acceptable Use Policy (llama-3-1-two-document-licensing-model). The Free Software Foundation classified Llama 3.1 as nonfree software in January 2025, citing jurisdictional trade-regulation enforcement and restrictions on popular applications (fsf-classified-llama31-nonfree-jan-2025), and the Open Source Initiative's October 2024 AI definition requires training-data disclosure that Meta does not provide (osi-ai-definition-requires-training-data-disclosure). The consensus across these beliefs is that Llama is best described as "source-available" or "open-weight" rather than open-source in the OSI or FSF sense (llama-not-open-source-osi-fsf). Distribution runs through the official site at llama.com and the meta-llama Hugging Face organization (llama-official-site-and-distribution-channels). The March 2023 BitTorrent leak of Llama 1 weights via 4chan, followed by a GitHub DMCA takedown at Meta's request, is recorded across three beliefs (llama-march-2023-4chan-leak-dmca-takedown, llama1-4chan-leak-github-dmca, llama1-weights-leaked-bittorrent-march-2023) and is noted as a catalyst for widespread adoption that Simon Willison called the LLM "Stable Diffusion moment." All beliefs in this topic are currently IN; none have been retracted.

IN bilingual-pair-single-token-filter-llama2-vocab
Bilingual translation pairs for the measurement notion are restricted to top-1 mutual correspondences in a publicly available lexicon that are also single tokens in LLaMA-2's BPE vocabulary; English⇒French yields 209 contexts for Y⁰ and 231 for Y¹.
IN cgd-single-random-middle-layer
In the Shen et al. experiments, continuous gradient descent (cGD) is simulated by optimizing a single randomly selected middle layer of LLaMA while holding the rest of the model frozen.
IN code-llama-500b-code-tokens
Code Llama is fine-tuned Llama 2 on 500B code tokens + 20B long-context tokens, with a separate Python-specialized variant trained on 100B Python-only tokens
IN code-llama-training-token-recipe
Code Llama was fine-tuned on Llama 2 using 500B tokens of general code, then 20B long-context tokens, then 5B instruction-following tokens; the Python variant used 100B Python-only tokens followed by long-context tokens.
IN compute-routing-operates-at-multiple-architectural-granularities
Compute routing in LLMs operates at two distinct architectural granularities: intra-model routing (MoE activating expert subnetwork subsets per token, as in Llama 4) and inter-model routing (GPT-5 selecting between fast and reasoning models per task), demonstrating that conditional computation is a general efficiency principle applicable at both the parameter level and the model-selection level.
IN fsf-classified-llama31-nonfree-jan-2025
The Free Software Foundation classified Llama 3.1's license as nonfree software in January 2025, specifically citing its acceptable use policy, restrictions on users with popular applications, and enforcement of trade regulations outside the user's jurisdiction.
IN llama-2-first-commercial-use-jul-2023
Llama 2 (July 2023) was the first Llama generation to permit commercial use, subject to user-count thresholds and an acceptable-use policy.
IN llama-3-1-400b-moe-model-jul-2024
Llama 3.1 (July 2024) introduced a 400B-parameter Mixture-of-Experts model.
IN llama-3-1-two-document-licensing-model
Llama 3.1 uses a two-document licensing model consisting of a Community License Agreement plus a separate Acceptable Use Policy, creating more restrictions than a standard single open-source license.
IN llama-3-trained-approximately-15-trillion-tokens
Llama 3 was trained on approximately 15 trillion tokens.
IN llama-4-first-natively-multimodal-in-line
Llama 4 (April 2025) is the first natively multimodal (text + vision) model in the Llama family; Llama 3.2 (September 2024) was the first to add vision via a separate vision encoder.
IN llama-4-moe-17b-active-128-experts
Llama 4 (April 2025) uses a Mixture-of-Experts architecture; the model identifier Llama-4-Maverick-17B-128E denotes 17B active parameters across 128 experts.
IN llama-context-window-progression
Llama context windows grew from 2K (Llama 1) to 4K (Llama 2) to 128K (Llama 3.1) to 10M (Llama 4 Scout)
IN llama-cpp-gguf-cpu-inference
llama.cpp is a C++ reimplementation of Llama inference enabling CPU-only execution, and introduced the GGUF binary format for quantized model storage with support for multiple quantization types.
IN llama-first-release-24-feb-2023-65b
Meta's Llama (LLaMA) was first announced on 24 February 2023 as a 65B-parameter model.
IN llama-march-2023-4chan-leak-dmca-takedown
In March 2023, LLaMA model weights were shared publicly via 4chan before Meta's controlled release, and GitHub issued a DMCA takedown at Meta's request.
IN llama-not-open-source-osi-fsf
Llama is not open-source by OSI or FSF standards; the FSF classified Llama 3.1 as nonfree software in January 2025; it is more accurately described as 'source-available' or 'open-weight'
IN llama-official-site-and-distribution-channels
Llama's official website is https://www.llama.com/ and model weights are distributed via the Hugging Face meta-llama organization; the Wikidata entry is Q116894231.
IN llama-timeline-six-generations-feb-2023-apr-2025
The Llama family timeline: LLaMA (Feb 2023) → LLaMA-2 (Jul 2023) → Llama 3 (Apr 2024) → Llama 3.1 (Jul 2024) → Llama 3.2 (Sep 2024) → Llama 4 (Apr 2025).
IN llama-training-data-sizes-progression
Llama training data sizes: 1.4T tokens (Llama 1) → 2T (Llama 2) → 15T (Llama 3) → 40T (Llama 4 Scout)
IN llama-training-token-volumes-per-generation
Llama training data volumes were approximately: Llama 1 ≈ 1.4T tokens, Llama 2 = 2T tokens, Llama 3 = 15T tokens, Llama 3.1 = 15T tokens (July 2024), and Llama 4 = 22T–40T tokens.
IN llama1-4chan-leak-github-dmca
Llama 1 model weights leaked via 4chan in March 2023 before official release, and GitHub issued a DMCA takedown at Meta's request to remove the unauthorized copies.
IN llama1-weights-leaked-bittorrent-march-2023
Llama 1 weights were leaked via BitTorrent on March 3, 2023, catalyzing widespread adoption (called LLMs' 'Stable Diffusion moment' by Simon Willison)
IN llama2-7b-architecture-specs
LLaMA-2 7B (meta-llama/Llama-2-7b-hf) uses a 32,000-token SentencePiece vocabulary with 4,096 embedding dimension, pre-trained on 2 trillion tokens (90% English).
IN llama2-license-700m-dau-restriction
Llama 2's license blocks entities with >700 million daily active users and prohibits using outputs to improve other LLMs
IN llama2-primacy-bias-threshold-13b
Llama-2 7B shows only recency bias (no primacy bias), while Llama-2 13B and 70B show both primacy and recency bias, indicating primacy bias requires approximately ≥13B parameters.
IN llama3-rope-theta-500k-vocab-128k
Llama 3 uses RoPE with θ=500,000 for positional embeddings and has a vocabulary size of 128,000
IN llama4-moe-scout-maverick-params
Llama 4 uses mixture-of-experts: Scout has 109B total / 17B active parameters with 16 experts; Maverick has 400B total / 17B active with 128 experts
IN llama4-scout-10m-context-window
Llama 4 Scout has a 10 million token context window, the largest in the Llama family
IN llama4-scout-maverick-specific-parameters
Llama 4 Scout has 17B active parameters, 16 experts, 109B total parameters, and 10M context window; Llama 4 Maverick has 17B active parameters, 128 experts, 400B total parameters, and 1M context window.
IN lrh-validated-on-llama2-7b-27-concepts
The Linear Representation Hypothesis was empirically validated on LLaMA-2 7B (decoder-only Transformer, 32K vocabulary) across 27 concepts comprising 22 from BATS 3.0, 4 language pairs (EN→FR, FR→DE, FR→ES, DE→ES), and 1 frequency pair, with code at github.com/KihoPark/linear_rep_geometry.
IN maverick-codistilled-from-behemoth
Llama 4 Maverick was codistilled from the unreleased Behemoth model (~2T total parameters), while Scout was trained from scratch
IN mistral-llama-weekday-month-accuracy
Mistral 7B and Llama 3 8B achieve approximately 29–31/49 on natural-language Weekdays tasks and 125–143/144 on Months tasks, yet trivial accuracy on raw modular-addition prompts.
IN muse-spark-llama-successor-april-2026
Muse Spark was released by Meta Superintelligence Labs in April 2026 as Llama's successor
IN osaid-october-2024-training-data-disclosure
The Open Source AI Definition (OSAID), published by OSI in October 2024, requires open-source AI to disclose training data details, which Meta does not do for Llama
IN osi-ai-definition-requires-training-data-disclosure
The Open Source Initiative's AI open-source definition (published 28 October 2024) specifically requires training data disclosure as a criterion, which Meta's Llama models do not meet.
IN production-deployment-requires-full-optimization-stack
Frontier model deployment benefits from optimization across multiple levels simultaneously: architectural (MoE decoupling parameters from compute, as in Llama 4), algorithmic (speculative decoding), memory management (PagedAttention's KV cache paging), and compute-aware (Flash Attention's IO-aware tiling), suggesting these layers form a complementary deployment stack.
IN rmsnorm-llama-alternative-to-layernorm
RMSNorm is used in Llama as a computationally simpler alternative to LayerNorm.