craft-methodology-unfalsifiably-self-consistent

IN derived (depth 11)

Created 2026-06-21T13:10:22+00:00 · Reviewed 2026-06-21T14:41:08+00:00

The LLM field's craft methodology is unfalsifiably self-consistent: its most rigorous quantitative results (scaling laws, information-theoretic constants) are themselves empirically discovered rather than theoretically derived, and NLP's AI-completeness ensures no formal methodology can replace empiricism — the craft validates itself by the only epistemology demonstrably available, creating a closed epistemic loop with no external reference point.

Justifications

SL — Scaling laws are empirically discovered and AI-completeness prevents formal alternatives — the craft methodology has no external falsification criterion

Antecedents (all must be IN):

IN scaling-evidence-is-itself-empirical-validating-craft-methodology — Key scaling relationships in LLM research — such as power-law relationships between performance and resources (Kaplan et al., 2020) and Chinchilla's information-theoretic grounding of compute-optimal scaling — were discovered through empirical observation rather than first-principles derivation. That these foundational quantitative regularities emerged from empirical methods is consistent with the field's broader character as a craft discipline where core knowledge is discovered and transmitted experientially.
IN formal-verification-impossible-given-ai-completeness — The craft discipline's inability to achieve formal safety verification may reflect a fundamental impossibility rather than a maturity gap: NLP's classification as AI-complete implies that formally verifying NLP system behavior requires solving the full AI problem, explaining why safety assurance remains fundamentally informal despite decades of engineering maturity and massive investment.

Dependents

These beliefs depend on this one:

IN field-epistemic-closure-prevents-independent-safety-validation — The LLM field exhibits complete epistemic closure that prevents independent safety validation: its craft methodology is unfalsifiably self-consistent (the strongest quantitative evidence — scaling laws, information-theoretic constants — validates the empirical approach that generated it), AND its primary safety evaluation mechanism is doubly circular and vulnerable (alignment is produced by the methodology it compensates for, evaluated by a reward model inheriting the paradigm's vulnerabilities) — meaning neither the methodology nor its safety assurances admit external validation from within the field's own epistemic framework.