rlhf-reward-model-normalization-mean-zero

IN premise — summaries/2026/08/24/wiki-Reinforcement_learning_from_human_feedback-chunk-2.md

Created 2026-08-24T17:11:23+00:00

RLHF reward model outputs are normalized post-training so reference completions have mean reward of 0, achieved by setting the bias to the mean reward across the training set

Summary

After the reward model finishes training, its output is re-centered so that the baseline completions it was asked to judge score exactly zero on average, meaning the numbers it produces are really just "how much better or worse than the reference" rather than some absolute grade. This matters because it gives the downstream reinforcement-learning step a clean, relative signal, so the policy only needs to push toward positive scores without having to interpret what a raw magnitude actually means.