lstm-transformer-parallelism-tradeoff

IN derived (depth 1)

Created 2026-06-21T09:59:01+00:00 · Reviewed 2026-06-21T15:37:01+00:00

The LSTM-to-Transformer transition traded the ability to learn dependencies across unlimited timesteps for massively parallel training, a sequential-depth-for-parallel-breadth exchange.

Justifications

SL — LSTM's unbounded sequential memory exchanged for Transformer's bounded but parallel attention

Antecedents (all must be IN):

IN lstm-learns-over-thousands-millions-timesteps — LSTM can learn dependencies across thousands or millions of time steps by allowing error to flow backward through unlimited virtual layers.
IN transformer-no-recurrence-enables-parallelism — Transformers eliminate sequential recurrence found in RNNs/LSTMs, enabling significantly more parallelism and shorter training times.
IN transformer-2017-quadratic-context — The Transformer architecture (2017, 'Attention Is All You Need') uses self-attention with quadratic computation cost in context window size and became the basis for GPT, Gemini, Grok, DeepSeek, and Qwen

Dependents

These beliefs depend on this one:

IN sequence-modeling-complete-evolutionary-arc — The arc of sequence modeling shows three generations — RNNs, LSTMs, Transformers — connected by successive solutions to predecessor limitations (vanishing gradients, then sequential bottleneck), with the LSTM-to-Transformer transition trading the ability to learn dependencies across unlimited timesteps for massively parallel training. Attention served as a primary bridging innovation, first augmenting RNN encoder-decoders to solve the fixed-size bottleneck, then enabling the replacement of recurrence entirely.