pretrain-finetune-resilience-propagates-vulnerability-across-pipeline
IN derived (depth 5)
Created 2026-06-21T11:44:45+00:00 · Reviewed 2026-06-21T14:41:08+00:00
The pretrain-finetune paradigm's demonstrated resilience across three dimensions (production validation, architectural survival, and RLHF embedding) coexists with memorization's dual-use characteristics (contributing to model knowledge while creating extraction and poisoning attack surfaces). Because the paradigm persists and propagates across the model pipeline, any memorization-related vulnerabilities present within it may persist as well — though this connection is an inference from the co-occurrence of these two independently established properties rather than a directly demonstrated causal relationship.
Justifications
SL — The paradigm that makes the pipeline resilient also makes its vulnerabilities equally resilient
Antecedents (all must be IN):
- IN pretrain-finetune-is-most-resilient-nlp-contribution — The pretrain-finetune paradigm demonstrates unique resilience across three independent dimensions — production validation at web scale (every Google Search query), survival as methodology across architectural transitions (BERT's encoder-only design gave way to decoder-only dominance), and embedding in the RLHF reward model as a living instance — making it arguably the most durable single methodological contribution in NLP history.
- IN memorization-is-dual-use-capability-and-vulnerability — Training data memorization exhibits dual-use characteristics: the same retention mechanism that contributes to model knowledge also creates an attack surface for deliberate data poisoning, as memorization rates serve as a quantitative proxy for poisoning vulnerability. GPT-2's early demonstration of both measurable memorization (1-7% exact duplicates) and capability-related safety concerns suggests this tension scales with model capability, though the evidence characterizes the pattern at one scale rather than confirming it as a universal structural property.
Dependents
These beliefs depend on this one:
- IN paradigm-resilience-propagates-vulnerability-via-information-theoretic-identity — The pretrain-finetune paradigm's propagation of vulnerability across the training pipeline is grounded in information-theoretic identity: capability and vulnerability are not merely co-located but informationally inseparable (compression quality equals memorization capability), meaning the paradigm's three-dimensional resilience (production validation, architectural survival, RLHF embedding) cannot be preserved while excising the vulnerability it carries.
- IN reward-model-inherits-vulnerability-from-paradigm-it-evaluates — The RLHF reward model — itself an instance of the surviving pretrain-finetune paradigm — may inherit the paradigm's vulnerability propagation characteristics: memorization as a dual-use property could flow from pretraining through the reward model into alignment scoring decisions, potentially creating a circular dependency where the judge inherits the defendant's flaws. However, this connection is inferred from the co-occurrence of paradigm resilience and memorization's dual-use nature rather than directly demonstrated.