context-expansion-enabled-by-efficiency-breakthroughs

IN derived (depth 2)

Created 2026-06-21T09:57:37+00:00 · Reviewed 2026-06-21T14:41:08+00:00

The 10,000x context window expansion (1K to 10M tokens over seven years) was only achievable because efficiency research (Flash Attention, PagedAttention, linear attention variants) driven by quadratic attention costs made such scales computationally tractable.

Justifications

SL — context expansion is the demand-side consequence of efficiency supply

Antecedents (all must be IN):

IN context-windows-expanded-10000x-in-seven-years — Context windows expanded approximately 10,000x from GPT-2's 1,024 tokens (2019) to Llama 4 Scout's 10 million tokens (2025), with Claude 2.1's 200K window as a major intermediate milestone in late 2023.
IN quadratic-attention-drives-efficiency-research — Self-attention's O(n²) complexity is the root constraint driving both context window limitations and a proliferation of efficiency techniques (Flash Attention, sparse/linear variants, Performers).

Dependents

These beliefs depend on this one:

IN context-expansion-enabled-agentic-application-paradigm — The 10,000x context window expansion — made possible by efficiency breakthroughs addressing quadratic attention — created the prerequisite substrate for agentic applications, as demonstrated by Claude's evolution from a chatbot to CLI coding agent, GUI office automation, and visual design tool, all of which require processing large, stateful contexts.
IN prompt-fragility-compounds-with-context-expansion — The dual architectural limitation of prompt control — sensitivity (40%+ accuracy shifts from formatting) and injection (inability to distinguish instructions from input) — compounds with the 10,000x context window expansion enabled by efficiency breakthroughs: larger contexts create proportionally larger attack surfaces for injection attempts, make sensitivity effects harder to diagnose across expanded input, and increase the probability that adversarial content co-occurs with legitimate instructions in the same context window.
OUT rag-reliable-for-knowledge-augmentation — RAG provides reliable knowledge augmentation — reducing hallucinations and keeping responses current via external retrieval at the prompting layer, with practical value scaling alongside context window expansion that enables richer retrieval payloads.