rlhf-reward-model-architecture
IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback.md
Created 2026-06-21T09:50:10+00:00
The RLHF reward model is initialized from a pretrained language model with its final layer replaced by a regression head that outputs a scalar reward score.
Summary
The reward model used to grade responses is built by taking a pretrained language model and swapping its final output layer for a single-number scorer, rather than designing a separate system from scratch. This means the model's sense of what makes a response good is rooted in the same linguistic understanding that generated the text, but compressed into one scalar value that drives the entire training loop.
Dependents
These beliefs depend on this one:
- IN rlhf-reward-model-is-modified-pretrained-lm — The RLHF reward model is architecturally a pretrained language model with only its final layer replaced by a scalar regression head, sharing ancestry with the policy model it will later train.