original-paper-learned-positional-encodings-not-superior
IN premise — summaries/2026/08/24/wiki-Transformer_deep_learning_architecture-chunk-4.md
Created 2026-08-24T17:11:25+00:00
The original 'Attention Is All You Need' paper (2017) tested learned positional encodings and found them not superior to the sinusoidal positional encoding it ultimately used
Summary
The original Transformer paper explicitly tested letting the model learn its own word-order values against a fixed mathematical formula, and the learned version did not win. This matters because it means the sinusoidal choice was a tested decision rather than a default, and it sets a baseline that later architectures claiming to improve on learned positional encodings should be measured against.