rlhf-preference-evaluation-reliable-at-scale

OUT derived (depth 1)

Created 2026-06-21T13:28:06+00:00

RLHF preference evaluation scales reliably: the foundational "easy to judge" property is amplified by K-wise ranking efficiency (asymptotically more efficient than decomposing into pairwise comparisons), enabling robust preference data collection at scale.

Justifications

SL — K-wise efficiency scales RLHF evaluation, but only if model outputs are stable enough to judge reliably

Antecedents (all must be IN):

IN rlhf-hard-to-specify-easy-to-judge — RLHF is motivated by tasks that are hard to specify but easy to judge — humans can compare outputs more easily than they can write explicit reward functions
IN rlhf-kwise-more-efficient-than-pairwise — K-wise comparisons (ranking K responses simultaneously) are asymptotically more efficient than decomposing into pairwise comparisons; all C(K,2) pairs from each ranking are used as a single training batch.

Unless (any of these IN defeats this justification):

IN prompt-sensitivity-40-percent-accuracy-shift — LLM performance is highly sensitive to prompt design, with accuracy shifts of over 40 percentage points from minor changes such as reordering examples, and up to 76 accuracy points difference across formatting changes