quadratic-attention-drives-architectural-succession-pressure

IN derived (depth 5)

Created 2026-06-21T14:03:57+00:00 · Reviewed 2026-06-21T15:37:01+00:00

The Transformer's quadratic attention cost creates permanent architectural succession pressure — just as RNNs were displaced by Transformers for failing to parallelize, Transformers face displacement pressure from linear-complexity alternatives (Mamba, RWKV, Reformer), confirming that hardware scalability determines paradigm survival applies reflexively to the currently dominant architecture.

Justifications

SL — If scalability determines survival and Transformers have a scalability ceiling (quadratic), then the principle predicts their own eventual displacement by linear alternatives.

Antecedents (all must be IN):

IN paradigm-survival-determined-by-scalability-not-theory — Mathematical completeness and theoretical elegance are neither necessary nor sufficient for paradigm survival in ML — GANs had the most complete analytical characterization yet were eclipsed by diffusion models, SVMs had convex guarantees yet were outscaled by neural networks, while theoretically less grounded approaches that scaled with hardware thrived.
IN transformer-2017-quadratic-context — The Transformer architecture (2017, 'Attention Is All You Need') uses self-attention with quadratic computation cost in context window size and became the basis for GPT, Gemini, Grok, DeepSeek, and Qwen
IN state-space-models-linear-complexity-alternative — State-space models (e.g., Mamba) have emerged as alternatives to transformers that handle long sequences with linear rather than quadratic complexity.

Dependents

These beliefs depend on this one:

IN quadratic-attention-self-undermines-hardware-lock-in — The crisis's material substrate is paradoxically self-undermining — the attention mechanism locks terminal epistemic saturation into TPU-optimized hardware, yet its quadratic computational cost simultaneously creates permanent architectural succession pressure that threatens to displace the very mechanism entrenching the crisis, meaning the hardware lock-in is inherently unstable even without external intervention.