context-expansion-enabled-by-efficiency-breakthroughs-v2

IN premise

Created 2026-08-24T18:40:12+00:00

The 10,000x context window expansion (1K to 10M tokens over seven years) was facilitated in part by the proliferation of efficiency techniques (Flash Attention, sparse/linear attention variants, Performers) that address self-attention's quadratic O(n²) cost, which is identified as the root constraint driving both context window limitations and this line of research.

Summary

Over the past seven years, context windows grew a thousandfold not because of hardware alone but because a set of algorithmic shortcuts like Flash Attention and Performers solved the quadratic cost problem in self-attention that had been the hard ceiling on how much text a model could process. The bottleneck was the math of attention, not raw compute, and fixing that math is what actually made long-context models practical.