llm-order-sensitivity-popularity-bias

Summary

When an LLM is asked to pull an answer from a set of documents, its choice shifts depending on which document appears first or last and which topics are mentioned most often across the set. This matters because the model is not neutrally weighing evidence; it is quietly steered by presentation order and by whichever answer is backed by the most text, so any system relying on it for retrieval or fact-checking has to actively control for these distortions rather than assume the output reflects the true consensus of the sources.