reward-model-alignment-resilient-against-base-poisoning

OUT derived (depth 4)

Created 2026-06-21T11:44:45+00:00

The RLHF reward model — grounded in the surviving pretrain-finetune paradigm's three-dimensional resilience and backed by a complete mathematical pipeline (Bradley-Terry, KL penalty, PPO) — provides resilient alignment signals that reliably encode human preferences.

Justifications

SL — Reward model inherits pretrain-finetune resilience unless base model memorization creates a poisoning vector into alignment itself

Antecedents (all must be IN):

IN reward-model-embodies-surviving-pretrain-finetune-paradigm — The RLHF reward model — a pretrained language model with its final layer replaced by a regression head — is itself an instance of BERT's surviving pretrain-finetune paradigm, meaning the decoder-only era's core alignment component is architecturally defined by an encoder-era methodology that outlived the architecture it was designed for.
IN rlhf-has-complete-mathematical-pipeline — RLHF has a fully specified mathematical pipeline: Bradley-Terry loss trains the reward model from comparisons, KL divergence constrains policy drift, and PPO's clipped surrogate optimizes the policy — each component addressing a distinct failure mode.

Unless (any of these IN defeats this justification):

IN memorization-creates-attack-surface-for-training-poisoning — LLM training data memorization (measured at 1-7% exact duplicates in GPT-2) is a plausible mechanism through which deliberate data poisoning (grooming) could materialize in model outputs — if poisoned content is memorized, it may be retrievable at inference time, suggesting that memorization rates could serve as one indicator of susceptibility to poisoning attacks.