rlhf-shared-ancestry-creates-reward-model-evaluation-blindspot

IN derived (depth 3)

Created 2026-06-21T13:28:05+00:00 · Reviewed 2026-06-21T14:41:08+00:00

RLHF's architecture — two models initialized from the same pretrained ancestor — means the reward model inherits the same memorization patterns, biases, and knowledge gaps as the policy it evaluates, creating a systematic evaluation blindspot where the judge cannot detect vulnerabilities it shares with the defendant.

Justifications

SL — Shared pretrained ancestry makes reward model blind to inherited vulnerabilities in the policy

Antecedents (all must be IN):

IN rlhf-two-models-same-pretrained-ancestor — RLHF trains two models (reward model and policy model), both initialized from the same pre-trained autoregressive language model
IN memorization-is-dual-use-capability-and-vulnerability — Training data memorization exhibits dual-use characteristics: the same retention mechanism that contributes to model knowledge also creates an attack surface for deliberate data poisoning, as memorization rates serve as a quantitative proxy for poisoning vulnerability. GPT-2's early demonstration of both measurable memorization (1-7% exact duplicates) and capability-related safety concerns suggests this tension scales with model capability, though the evidence characterizes the pattern at one scale rather than confirming it as a universal structural property.