lstm-1997-vanishing-gradient
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture.md
Created 2026-06-21T09:50:10+00:00
LSTM (Hochreiter & Schmidhuber, 1997) solved the vanishing gradient problem in RNNs using multiplicative gating units and was the dominant sequence modeling architecture until Transformers in 2017.
Summary
Before LSTMs, recurrent networks could only usefully look back a few steps because signal strength collapsed over time, making them bad at tracking long dependencies in text or speech. This gating trick kept information flowing through long sequences and became the standard approach for sequence modeling for twenty years, which is why the arrival of Transformers in 2017 represented a genuine paradigm shift rather than an incremental improvement.