context-windows-expanded-10000x-in-seven-years
IN derived (depth 1)
Created 2026-06-21T09:54:53+00:00 · Reviewed 2026-06-21T14:41:08+00:00
Context windows expanded approximately 10,000x from GPT-2's 1,024 tokens (2019) to Llama 4 Scout's 10 million tokens (2025), with Claude 2.1's 200K window as a major intermediate milestone in late 2023.
Summary
AI models' working memory has grown from about a page and a half of text to roughly 30,000 pages in just six or seven years, meaning a single model can now hold an entire book, a full codebase, or months of conversation in active consideration at once. For any system that reasons over information, this collapse of the context bottleneck changes what is practically possible: the constraint of "the model will forget what you said twenty pages ago" is no longer the design driver it once was.
Justifications
SL — Three context window measurements across model families chart an exponential growth trajectory
Antecedents (all must be IN):
- IN gpt2-context-window-1k-tokens — GPT-2 had 12 attention heads and a 1,024-token context window
- IN claude-2-1-200k-context-window — Claude 2.1 introduced a 200,000-token context window (approximately 500 pages).
- IN llama4-scout-10m-context-window — Llama 4 Scout has a 10 million token context window, the largest in the Llama family
Dependents
These beliefs depend on this one:
- IN context-expansion-compounds-governance-gap — The 10,000x context window expansion over seven years occurred alongside the governance gap created by weight availability: as model weights became increasingly accessible despite unresolved governance frameworks, released models also gained substantially larger context windows — suggesting that governance challenges may involve both growing numbers of accessible models and increasing capability per model, though the specific security implications of larger context windows require independent evidence beyond the expansion trend itself.
- IN context-expansion-enabled-by-efficiency-breakthroughs — The 10,000x context window expansion (1K to 10M tokens over seven years) was only achievable because efficiency research (Flash Attention, PagedAttention, linear attention variants) driven by quadratic attention costs made such scales computationally tractable.
- IN inference-memory-management-recapitulates-os-evolution — The approximately 10,000x context window expansion (from GPT-2's 1,024 tokens to Llama 4 Scout's 10 million) coincided with inference memory management evolving from basic KV caching (storing key/value vectors to avoid recomputation) to virtual-memory-style paging (PagedAttention partitioning KV cache into non-contiguous blocks), a progression that parallels the operating system's evolution from simple buffers to demand-paged virtual memory.
- IN memory-evolving-from-wider-windows-to-persistent-state — LLM memory capability is evolving along two distinct axes: horizontal expansion (10,000x context window growth from 1K to 10M tokens over seven years) and temporal persistence (Dreaming consolidating memory between sessions) — suggesting the next frontier is not how much a model can process at once but what it retains across interactions.