craft-discipline-nature-makes-safety-assurance-fundamentally-informal
IN derived (depth 9)
Created 2026-06-21T11:14:06+00:00 · Reviewed 2026-06-21T14:41:08+00:00
The LLM field's identity as a craft discipline — where both its most valuable properties and its accessibility barriers are empirical rather than formal — means safety assurance is fundamentally informal: security challenges that compound across all maturity dimensions cannot be formally verified in a field that discovers its own properties only through practice.
Justifications
SL — Craft disciplines produce reliable artifacts but cannot formally prove they are safe — and security compounds make informal assurance perpetually incomplete
Antecedents (all must be IN):
- IN llm-field-is-fundamentally-craft-discipline — The LLM field is fundamentally a craft discipline: both its most valuable structural properties (cross-boundary innovation, parameter redundancy) and its deepest barriers (tacit deployment knowledge, experiential prerequisites) are discovered and transmitted empirically, not through formal theory — meaning neither mastery nor failure modes are accessible through documentation alone.
- IN security-challenge-compounds-across-all-maturity-dimensions — LLM security is uniquely difficult because challenges compound across every dimension of the field's maturity: attack surfaces expand with capability scaling, defense-in-depth strategies are simultaneously necessitated and bounded by theoretical gaps, and the empirical nature of the field means security expertise — like all operational expertise — is irreducibly experiential rather than formally transferable.
Dependents
These beliefs depend on this one:
- IN alignment-diversification-compensates-for-craft-formal-verification-deficit — Alignment diversification into multiple independent paradigms serves as a practical substitute for formal safety verification: since the craft discipline fundamentally cannot provide formal guarantees for any single alignment approach (safety assurance is inherently informal), and RLHF's irreducible theoretical complexity (non-Markovian optimal policies, online/offline divergence) rules out complete formal specification even for the best-understood approach, having multiple independent alignment paths provides probabilistic coverage that no single formally unverifiable approach can offer alone.
- IN formal-verification-impossible-given-ai-completeness — The craft discipline's inability to achieve formal safety verification may reflect a fundamental impossibility rather than a maturity gap: NLP's classification as AI-complete implies that formally verifying NLP system behavior requires solving the full AI problem, explaining why safety assurance remains fundamentally informal despite decades of engineering maturity and massive investment.
- IN safety-deficit-is-structural-and-widening — The LLM field faces a structural and widening safety deficit: safety assurance is fundamentally informal (craft-based, not formally verifiable) while the security expertise gap widens with each capability advance, so the field's safety capacity cannot keep pace with growing demand.
- OUT safety-formalization-could-follow-pretrain-finetune-trajectory — The craft discipline's fundamentally informal safety assurance could formalize through the same mechanism that made pretrain-finetune resilient — repeated production validation eventually crystallizing tacit safety knowledge into standardized methodology, following the paradigm's proven three-dimensional resilience pattern.