rlhf-reward-model-scalar-regression-head

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-2.md

Created 2026-06-21T09:50:10+00:00

The RLHF reward model replaces the pre-trained LM's final layer with a randomly initialized regression head that outputs a single scalar score for any prompt-response pair — it is a regression head, not classification

Summary

In the RLHF setup, the reward model's job is to look at a prompt and a proposed response and spit out one number on a continuous scale — not a label, not a next-word prediction, just a single quality score. This matters because it frames the whole feedback signal as a measurable magnitude rather than a category, which is why the training targets are numeric and why the model can meaningfully rank responses against each other.

Dependents

These beliefs depend on this one:

IN rlhf-reward-model-is-modified-pretrained-lm — The RLHF reward model is architecturally a pretrained language model with only its final layer replaced by a scalar regression head, sharing ancestry with the policy model it will later train.