sequence-modeling-three-generations

IN derived (depth 1)

Created 2026-06-21T09:59:01+00:00 · Reviewed 2026-06-21T15:37:01+00:00

Sequence modeling evolved through three generations — RNNs, LSTMs, Transformers — each solving a specific limitation of its predecessor (vanishing gradients for LSTMs, sequential bottleneck for Transformers).

Justifications

SL — Each generation addressed the key technical bottleneck of the previous one

Antecedents (all must be IN):

IN lstm-designed-to-solve-vanishing-gradient — LSTM was designed specifically to solve the vanishing gradient problem using forget gates as the key mechanism.
IN transformer-replaced-recurrence-with-self-attention-2017 — The Transformer (Vaswani et al. 2017, 'Attention is All You Need') replaced recurrence with self-attention and became dominant for sequence modeling.
IN encoder-decoder-rnn-precursor-to-transformer — Encoder-decoder RNNs were state-of-the-art for neural machine translation from 2014–2017 and were a direct precursor to the Transformer architecture.

Dependents

These beliefs depend on this one:

IN sequence-modeling-complete-evolutionary-arc — The arc of sequence modeling shows three generations — RNNs, LSTMs, Transformers — connected by successive solutions to predecessor limitations (vanishing gradients, then sequential bottleneck), with the LSTM-to-Transformer transition trading the ability to learn dependencies across unlimited timesteps for massively parallel training. Attention served as a primary bridging innovation, first augmenting RNN encoder-decoders to solve the fixed-size bottleneck, then enabling the replacement of recurrence entirely.