memorization-creates-attack-surface-for-training-poisoning

IN derived (depth 1)

Created 2026-06-21T10:20:46+00:00 · Reviewed 2026-06-21T14:41:08+00:00

LLM training data memorization (measured at 1-7% exact duplicates in GPT-2) is a plausible mechanism through which deliberate data poisoning (grooming) could materialize in model outputs — if poisoned content is memorized, it may be retrievable at inference time, suggesting that memorization rates could serve as one indicator of susceptibility to poisoning attacks.

Summary

Because language models retain small but real slices of their training data in verbatim form, an attacker who floods the web with targeted content could potentially embed it in a model in a way that gets pulled back out later when prompted. This means the measured memorization rate is not just a data-leakage stat but a rough gauge of how exposed a model is to deliberate poisoning through public channels.

Justifications

SL — Memorization is the mechanism that converts training-time poisoning into inference-time output corruption

Antecedents (all must be IN):

IN gpt2-memorization-1-to-7-pct-exact-duplicates — GPT-2 models were measured to contain 1% to approximately 7% exact duplicate outputs from training data, and repetition prompting can trigger memorized excerpts.
IN llm-grooming-deliberate-training-data-poisoning — LLM grooming is the deliberate mass-publishing of web content to bias LLM training data and outputs, a term coined by the American Sunlight Project in 2025 (e.g., the Pravda network).

Dependents

These beliefs depend on this one:

OUT constitutional-ai-scales-beyond-rlhf-complexity — Constitutional AI provides a complete alignment path that bypasses RLHF's irreducible theoretical complexity (non-Markovian optimal policies, divergent online/offline formulations) by using written principles and AI-generated feedback instead of per-example human preferences — but only if training data memorization does not create attack surfaces that corrupt the base model's capacity to follow constitutional principles faithfully.
IN memorization-is-dual-use-capability-and-vulnerability — Training data memorization exhibits dual-use characteristics: the same retention mechanism that contributes to model knowledge also creates an attack surface for deliberate data poisoning, as memorization rates serve as a quantitative proxy for poisoning vulnerability. GPT-2's early demonstration of both measurable memorization (1-7% exact duplicates) and capability-related safety concerns suggests this tension scales with model capability, though the evidence characterizes the pattern at one scale rather than confirming it as a universal structural property.
OUT reward-model-alignment-resilient-against-base-poisoning — The RLHF reward model — grounded in the surviving pretrain-finetune paradigm's three-dimensional resilience and backed by a complete mathematical pipeline (Bradley-Terry, KL penalty, PPO) — provides resilient alignment signals that reliably encode human preferences.