compound-risk-manageable-through-craft-self-correction

OUT derived (depth 9)

Created 2026-06-21T11:28:04+00:00

The compound risk from adoption acceleration pushing continuous agents into production could be managed through the craft discipline's empirical self-correction — deployment feedback naturally concentrating practitioner attention on the most dangerous failure modes first, as the same experiential learning that characterizes the field's knowledge accumulation would surface and patch vulnerabilities through production observation.

Justifications

SL — Craft self-correction manages compound risks only if threats are empirically detectable

Antecedents (all must be IN):

IN adoption-and-persistence-create-compound-risk-multiplier — The adoption flywheel and persistent memory create a compound risk multiplier: adoption acceleration pushes continuous agents into production at a rate that outpaces security expertise development, while persistent memory adds cross-session attack surfaces that compound with each deployment cycle.
IN llm-field-is-fundamentally-craft-discipline — The LLM field is fundamentally a craft discipline: both its most valuable structural properties (cross-boundary innovation, parameter redundancy) and its deepest barriers (tacit deployment knowledge, experiential prerequisites) are discovered and transmitted empirically, not through formal theory — meaning neither mastery nor failure modes are accessible through documentation alone.

Unless (any of these IN defeats this justification):

IN sleeper-agents-resistant-to-safety-training — Anthropic research demonstrated that sleeper agents (models with hidden behaviors triggered by specific conditions) are difficult to detect or remove via standard safety training techniques.