rlhf-reward-model-is-modified-pretrained-lm

IN derived (depth 1)

Created 2026-06-21T09:52:14+00:00 · Reviewed 2026-06-21T14:41:08+00:00

The RLHF reward model is architecturally a pretrained language model with only its final layer replaced by a scalar regression head, sharing ancestry with the policy model it will later train.

Summary

The reward model in RLHF is not a separate kind of system; it is the same pretrained language model that the policy model starts from, with only the final layer swapped out to output a single score instead of generating text. Because both models descend from the same base, the judge and the student share deep structural understanding of language, which means they inherit the same blind spots and the reward signal is essentially one close cousin grading another.

Justifications

SL — Minimal architectural modification — same pretrained LM, one layer changed — gives the reward model linguistic competence it inherits rather than learns

Antecedents (all must be IN):

  • IN rlhf-reward-model-architecture — The RLHF reward model is initialized from a pretrained language model with its final layer replaced by a regression head that outputs a scalar reward score.
  • IN rlhf-reward-model-scalar-regression-head — The RLHF reward model replaces the pre-trained LM's final layer with a randomly initialized regression head that outputs a single scalar score for any prompt-response pair — it is a regression head, not classification
  • IN rlhf-two-models-same-pretrained-ancestor — RLHF trains two models (reward model and policy model), both initialized from the same pre-trained autoregressive language model

Dependents

These beliefs depend on this one: