rlhf-easy-to-judge-assumption-undermined-by-prompt-sensitivity

IN derived (depth 1)

Created 2026-06-21T13:28:05+00:00 · Reviewed 2026-06-21T14:41:08+00:00

RLHF's foundational motivation — that model outputs are "hard to specify but easy to judge" — is structurally undermined by prompt sensitivity: if formatting alone shifts accuracy by 40%+, human preference judgments evaluate a presentation-dependent signal rather than stable model capability, making the "easy to judge" assumption unreliable at precisely the point where it matters most.

Summary

The core idea behind RLHF — that humans can reliably pick the better of two model outputs even when they can't write explicit rules for what "good" means — falls apart when small formatting changes like reordering examples can swing accuracy by over 40 points. This means human preference comparisons are often reacting to presentation quirks rather than stable underlying capability, which makes the reward signal RLHF is built on fundamentally noisy and untrustworthy.

Justifications

SL — 40%+ accuracy shifts from formatting confound the human judgments RLHF depends on

Antecedents (all must be IN):

IN rlhf-hard-to-specify-easy-to-judge — RLHF is motivated by tasks that are hard to specify but easy to judge — humans can compare outputs more easily than they can write explicit reward functions
IN prompt-sensitivity-40-percent-accuracy-shift — LLM performance is highly sensitive to prompt design, with accuracy shifts of over 40 percentage points from minor changes such as reordering examples, and up to 76 accuracy points difference across formatting changes