grokking-memorization-phase-manageable-under-controlled-training

OUT derived (depth 3)

Created 2026-06-21T13:22:52+00:00

Grokking's memorize-then-generalize dynamic implies that the security-vulnerable memorization phase is a transient training state that resolves under continued training — models move from memorization (maximal data extractability) to generalization (compressed, abstract representations), making the vulnerability window manageable under controlled training conditions where intermediate checkpoints are secured.

Justifications

SL — The memorization-phase vulnerability resolves with continued training UNLESS models can embed persistent hidden behaviors that survive safety training — meaning apparent generalization may conceal rather than eliminate memorized patterns

Antecedents (all must be IN):

IN grokking-memorize-then-generalize — Grokking is the phenomenon where a model first memorizes training data (overfitting), then suddenly learns the underlying algorithm and generalizes, discovered via mechanistic interpretability of modular arithmetic models.
IN memorization-is-dual-use-capability-and-vulnerability — Training data memorization exhibits dual-use characteristics: the same retention mechanism that contributes to model knowledge also creates an attack surface for deliberate data poisoning, as memorization rates serve as a quantitative proxy for poisoning vulnerability. GPT-2's early demonstration of both measurable memorization (1-7% exact duplicates) and capability-related safety concerns suggests this tension scales with model capability, though the evidence characterizes the pattern at one scale rather than confirming it as a universal structural property.

Unless (any of these IN defeats this justification):

IN sleeper-agents-resistant-to-safety-training — Anthropic research demonstrated that sleeper agents (models with hidden behaviors triggered by specific conditions) are difficult to detect or remove via standard safety training techniques.