rope-dot-products-relative-position-only

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-4.md

Created 2026-06-21T09:50:11+00:00

RoPE (Rotary Positional Embedding) encodes position by rotating pairs of embedding dimensions; its key property is that dot products depend only on relative position, not absolute position.

Summary

In transformer attention, the model compares token pairs by computing dot products, and RoPE's design ensures that comparison captures only how far apart two tokens are, not where they sit in the absolute sequence. This matters because it lets the system generalize across sequence lengths and treat any pair of tokens the same way regardless of where they appear, which is the foundation for why modern LLMs can attend coherently over long contexts.