rlhf-reward-model-bradley-terry-loss
IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-2.md
Created 2026-06-21T09:50:10+00:00
The RLHF reward model uses a Bradley-Terry cross-entropy loss over human preference comparisons, and when K responses are ranked, all C(K,2) pairwise comparisons form a single training batch
Summary
When the reward model is trained on a set of ranked responses, it carves out every possible head-to-head comparison from that ranking and trains on them all in one go, rather than just comparing adjacent pairs. This means each set of rankings yields far more training signal than the number of responses might suggest, and the model learns a smooth, holistic ranking function instead of a string of isolated yes-or-no judgments.
Dependents
These beliefs depend on this one:
- IN rlhf-has-complete-mathematical-pipeline — RLHF has a fully specified mathematical pipeline: Bradley-Terry loss trains the reward model from comparisons, KL divergence constrains policy drift, and PPO's clipped surrogate optimizes the policy — each component addressing a distinct failure mode.