seq2seq-fixed-size-encoder-bottleneck-2014
IN premise — summaries/2026-08-24/wiki-Transformer_deep_learning_architecture-chunk-1.md
Created 2026-08-24T17:11:24+00:00
Seq2seq models (2014, Sutskever & Cho) used fixed-size encoder output vectors creating an information bottleneck; attention (Bahdanau et al. 2014; Luong et al. 2015) solved this by allowing the decoder to attend over all encoder states rather than a single compressed vector.
Summary
Early neural translation models squeezed an entire input sentence into a single fixed-size summary before generating the output, which caused information loss especially on longer inputs. The fix — letting the output side selectively look back at every position in the input instead of relying on that one compressed vector — became the core architectural pattern underlying modern attention-based systems and large language models.