liu-2023-position-encodings-tested

IN premise — summaries/2026/08/24/liu-2023-lost-in-middle-s2-multi-document-question-answering.md

Created 2026-08-25T02:58:08+00:00

The U-shaped positional bias is observed across models using ALiBi (MPT-30B), condensed RoPE (LongChat-13B), and learned/rotary embeddings (GPT-3.5-Turbo, Claude-1.3), indicating the effect is architecture-general.

Summary

The finding that the U-shaped middle-of-sequence weakness shows up whether a model uses ALiBi, RoPE, or learned position embeddings tells us this bias is not a bug of any single encoding design. It points to something deeper about how transformer architectures process text, meaning the problem likely can't be fixed simply by swapping in a different positional encoding scheme.