scaling-evidence-is-itself-empirical-validating-craft-methodology

IN derived (depth 9)

Created 2026-06-21T13:01:36+00:00 · Reviewed 2026-06-21T14:41:08+00:00

Key scaling relationships in LLM research — such as power-law relationships between performance and resources (Kaplan et al., 2020) and Chinchilla's information-theoretic grounding of compute-optimal scaling — were discovered through empirical observation rather than first-principles derivation. That these foundational quantitative regularities emerged from empirical methods is consistent with the field's broader character as a craft discipline where core knowledge is discovered and transmitted experientially.

Justifications

SL — Scaling laws are craft-discovered regularities — the field's laws validate its own methodology

Antecedents (all must be IN):

IN scaling-laws-power-law-relationships — Neural scaling laws describe empirical power-law relationships between model performance and model size, dataset size, and compute (Kaplan et al., 2020).
IN chinchilla-grounds-scaling-in-information-theory — Chinchilla research established that language model quality directly measures data compression capability (compressing ImageNet to 43% vs PNG's 58%), grounding compute-optimal scaling laws in information-theoretic foundations rather than purely empirical curve-fitting.
IN llm-field-is-fundamentally-craft-discipline — The LLM field is fundamentally a craft discipline: both its most valuable structural properties (cross-boundary innovation, parameter redundancy) and its deepest barriers (tacit deployment knowledge, experiential prerequisites) are discovered and transmitted empirically, not through formal theory — meaning neither mastery nor failure modes are accessible through documentation alone.

Dependents

These beliefs depend on this one:

IN craft-methodology-unfalsifiably-self-consistent — The LLM field's craft methodology is unfalsifiably self-consistent: its most rigorous quantitative results (scaling laws, information-theoretic constants) are themselves empirically discovered rather than theoretically derived, and NLP's AI-completeness ensures no formal methodology can replace empiricism — the craft validates itself by the only epistemology demonstrably available, creating a closed epistemic loop with no external reference point.