quadratic-attention-spawned-comprehensive-efficiency-stack
IN derived (depth 2)
Created 2026-06-21T09:54:53+00:00 · Reviewed 2026-06-21T14:41:08+00:00
Self-attention's O(n²) complexity is a primary driver of efficiency research, motivating both sub-quadratic algorithmic alternatives (sparse/linear variants, Performers) and a multi-layer practical optimization stack spanning algorithmic (speculative decoding), memory (PagedAttention), and compute (FlashAttention) approaches.
Justifications
SL — The theoretical constraint (depth-1) and the practical response (depth-1) combine into a complete picture of efficiency as a field-wide effort
Antecedents (all must be IN):
- IN quadratic-attention-drives-efficiency-research — Self-attention's O(n²) complexity is the root constraint driving both context window limitations and a proliferation of efficiency techniques (Flash Attention, sparse/linear variants, Performers).
- IN inference-optimization-is-multi-layer-discipline — LLM inference optimization operates at multiple layers simultaneously: algorithmic (speculative decoding with draft models), memory management (PagedAttention's KV cache paging), compute (Flash Attention's IO-aware tiling), and caching (KV reuse across tokens).
Dependents
These beliefs depend on this one:
- IN attention-universality-makes-efficiency-existential — Attention's validated status as a universal computation primitive — the sole mechanism underlying all frontier language, protein, chess, and RL models — transforms its O(n²) complexity from a performance concern into an existential constraint: the technique that everything depends on is the one most expensive to scale.
- IN decoder-only-scaling-requires-comprehensive-efficiency-stack — The decoder-only architecture won the scaling race through data volume advantage, but its quadratic attention cost means this dominance is only practically sustainable through the comprehensive efficiency stack (Flash Attention, PagedAttention, speculative decoding, MoE).