transformer-replaces-recurrence-with-parallel-attention

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture.md

Created 2026-06-21T09:50:10+00:00

Transformers replaced recurrent mechanisms (RNN/LSTM) with parallel multi-head self-attention, eliminating sequential bottlenecks but introducing O(n²) computation cost in context window size.

Summary

The core design move in modern large language models is trading the step-by-step, one-token-at-a-time processing of earlier architectures for a scheme where every position can look at every other position at once. This makes training far faster and more parallelizable, but it also means the computational cost balloons quadratically as the context window grows, which is the fundamental reason long-context inference and training remain so expensive.

Dependents

These beliefs depend on this one:

IN transformer-superseded-lstm-via-parallelism — The Transformer ended LSTM's 20-year dominance (1997–2017) by replacing sequential recurrence with parallel self-attention, enabling massive scaling.