decoder-only-scaling-requires-comprehensive-efficiency-stack
IN derived (depth 3)
Created 2026-06-21T09:57:37+00:00 · Reviewed 2026-06-21T14:41:08+00:00
The decoder-only architecture won the scaling race through data volume advantage, but its quadratic attention cost means this dominance is only practically sustainable through the comprehensive efficiency stack (Flash Attention, PagedAttention, speculative decoding, MoE).
Justifications
SL — decoder-only dominance creates the very scaling demand that drives the efficiency stack
Antecedents (all must be IN):
- IN decoder-only-dominance-driven-by-data-scaling-advantage — Decoder-only architectures became dominant in the scaling era, coinciding with evidence that data volume matters more than parameter count for model performance. Autoregressive next-token prediction's ability to leverage massive unsupervised text corpora may have been a contributing factor in this dominance.
- IN quadratic-attention-spawned-comprehensive-efficiency-stack — Self-attention's O(n²) complexity is a primary driver of efficiency research, motivating both sub-quadratic algorithmic alternatives (sparse/linear variants, Performers) and a multi-layer practical optimization stack spanning algorithmic (speculative decoding), memory (PagedAttention), and compute (FlashAttention) approaches.
Dependents
These beliefs depend on this one:
- IN complete-llm-paradigm-requires-efficiency-to-be-practical — The full NLP paradigm shift culminating in decoder-only dominance only became a practical deployment paradigm because the comprehensive efficiency stack (Flash Attention, speculative decoding, MoE, PagedAttention) addressed attention's quadratic scaling constraint — without efficiency breakthroughs, decoder-only dominance would be a theoretical victory unusable at frontier scale.
- IN training-and-deployment-optimization-diverge-at-every-level — LLM training and deployment require fundamentally divergent optimization strategies: training prioritizes data volume over parameters (validated by both Chinchilla theory and compression evidence), while deployment requires a comprehensive efficiency stack to manage quadratic attention costs — meaning optimal LLM development demands different expertise and infrastructure at each lifecycle stage.