rlhf-two-models-same-pretrained-ancestor

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-2.md

Created 2026-06-21T09:50:10+00:00

RLHF trains two models (reward model and policy model), both initialized from the same pre-trained autoregressive language model

Summary

In RLHF, the model that scores responses and the model that generates them both start from the same base language model, so they share the same underlying understanding of language. This matters because any biases or limitations baked into that base model are inherited by both sides of the training loop, meaning the alignment process is constrained by whatever the original model already believed about how language works.

Dependents

These beliefs depend on this one:

IN rlhf-reward-model-is-modified-pretrained-lm — The RLHF reward model is architecturally a pretrained language model with only its final layer replaced by a scalar regression head, sharing ancestry with the policy model it will later train.
IN rlhf-shared-ancestry-creates-reward-model-evaluation-blindspot — RLHF's architecture — two models initialized from the same pretrained ancestor — means the reward model inherits the same memorization patterns, biases, and knowledge gaps as the policy it evaluates, creating a systematic evaluation blindspot where the judge cannot detect vulnerabilities it shares with the defendant.