ssm-architecturally-validates-transformer-quadratic-limitation
IN derived (depth 1)
Created 2026-06-21T14:03:57+00:00 · Reviewed 2026-06-21T15:37:01+00:00
State space models (Mamba, RWKV) achieving competitive performance with linear complexity architecturally validates that the Transformer's quadratic attention cost is a genuine limitation, not merely a theoretical concern — alternative architectures prove that sequence modeling does not inherently require quadratic computation.
Justifications
SL — Three premises independently establish that (1) SSMs offer linear complexity, (2) Mamba competes with Transformers, and (3) Transformers have quadratic cost — together they validate the limitation as practically real, not merely asymptotic.
Antecedents (all must be IN):
- IN state-space-models-linear-complexity-alternative — State-space models (e.g., Mamba) have emerged as alternatives to transformers that handle long sequences with linear rather than quadratic complexity.
- IN mamba-state-space-model-competes-rnns-transformers — Mamba is a state-space model architecture that competes with both RNNs and Transformers for sequence modeling tasks
- IN transformer-2017-quadratic-context — The Transformer architecture (2017, 'Attention Is All You Need') uses self-attention with quadratic computation cost in context window size and became the basis for GPT, Gemini, Grok, DeepSeek, and Qwen