dual-use-properties-make-security-scaling-superlinear

IN derived (depth 5)

Created 2026-06-21T11:09:27+00:00 · Reviewed 2026-06-21T14:41:08+00:00

LLM security challenges tend to scale with capability growth because dual-use properties like memorization — which serves as both a knowledge source and an attack vector — mean that mechanisms driving capability improvements can simultaneously deepen vulnerabilities. Evidence from training data memorization patterns and the expansion of multiple independent security surfaces (poisoning, prompt injection, architectural vulnerabilities) suggests that security surfaces don't merely grow alongside capabilities but may become structurally harder to defend, though this pattern is characterized at limited scales rather than confirmed as a universal structural law.

Summary

Because the very features that make language models useful, such as retaining and recalling training data, are the same features that create attack surfaces like data poisoning and prompt injection, security problems compound rather than simply grow in proportion to capability. This means LLM safety is not a one-time engineering fix but a persistent, multi-front challenge where every capability gain can deepen the attack surface and make defenses structurally harder to maintain.

Justifications

SL — When the mechanisms creating capability ARE the vulnerability surface, defending against attacks means constraining the very properties that make the system useful

Antecedents (all must be IN):

IN memorization-is-dual-use-capability-and-vulnerability — Training data memorization exhibits dual-use characteristics: the same retention mechanism that contributes to model knowledge also creates an attack surface for deliberate data poisoning, as memorization rates serve as a quantitative proxy for poisoning vulnerability. GPT-2's early demonstration of both measurable memorization (1-7% exact duplicates) and capability-related safety concerns suggests this tension scales with model capability, though the evidence characterizes the pattern at one scale rather than confirming it as a universal structural property.
IN security-surfaces-expand-with-capability-scaling — The early evidence of dual-scaling tensions between capabilities and risks (as illustrated by GPT-2's memorization and misuse concerns) compounds the challenge posed by three independent security surfaces — training data poisoning, prompt injection, and architectural vulnerabilities — since architectural vulnerabilities in particular appear fundamental rather than solvable by scale alone, suggesting that LLM security may be a persistently difficult problem rather than one that straightforward engineering progress will resolve.

Dependents

These beliefs depend on this one:

IN attention-universality-couples-capability-and-vulnerability-expansion — Attention's modality-agnostic universality — extending through tokenization to process any domain's data as token sequences — suggests a structural coupling between capability expansion and vulnerability expansion: each new modality enabled by the universal computation primitive may simultaneously create new attack surfaces. Evidence from dual-use properties like memorization indicates these security surfaces may scale superlinearly with capability, though this pattern is characterized at limited scales rather than confirmed as a universal structural law.
IN security-expertise-gap-widens-with-each-capability-advance — Each LLM capability advance simultaneously creates new dual-use security surfaces that scale superlinearly and requires new operational expertise to defend them — expertise that can only be acquired through direct experience with the new capability's failure modes — creating a security expertise gap that widens with every generation.