rlhf-hard-to-specify-easy-to-judge

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-1.md

Created 2026-06-21T09:50:10+00:00

RLHF is motivated by tasks that are hard to specify but easy to judge — humans can compare outputs more easily than they can write explicit reward functions

Summary

For many tasks, writing a precise rule for what "good" looks like is nearly impossible, but glancing at two examples and picking the better one is effortless. This gap is the core reason RLHF exists: it sidesteps the need for explicit specifications and instead trains on the comparisons humans can actually make, which means the whole approach is only as strong as the quality of those pairwise judgments.

Dependents

These beliefs depend on this one:

IN rlhf-easy-to-judge-assumption-undermined-by-prompt-sensitivity — RLHF's foundational motivation — that model outputs are "hard to specify but easy to judge" — is structurally undermined by prompt sensitivity: if formatting alone shifts accuracy by 40%+, human preference judgments evaluate a presentation-dependent signal rather than stable model capability, making the "easy to judge" assumption unreliable at precisely the point where it matters most.
OUT rlhf-preference-evaluation-reliable-at-scale — RLHF preference evaluation scales reliably: the foundational "easy to judge" property is amplified by K-wise ranking efficiency (asymptotically more efficient than decomposing into pairwise comparisons), enabling robust preference data collection at scale.
IN rlhf-transferred-from-games-via-universal-evaluation-property — RLHF transferred from Atari/robotics (Christiano 2017) to NLP because its motivating property — tasks that are hard to specify but easy to judge — is domain-independent, enabling the same preference-comparison framework to align game bots and language models alike.