scaling-evidence-is-itself-empirical-validating-craft-methodology-v2

IN premise

Created 2026-08-24T18:10:15+00:00

Scaling laws in LLM research are substantially empirical in character: Kaplan et al. (2020) describe power-law relationships between performance and resources as empirical regularities, while Chinchilla extends the picture by grounding compute-optimal scaling in information-theoretic foundations rather than purely empirical curve-fitting. That the field's core quantitative regularities are largely established through empirical methods—even where theoretical grounding is subsequently provided—is consistent with the broader character of the LLM field as a craft discipline in which key knowledge is discovered and transmitted experientially rather than through formal theory alone.

Summary

The field's key quantitative rules about how models improve with more data and compute were largely discovered by watching what happened, not by deriving them from first principles, which means they function more like the hard-won knowledge of a craft tradition than like theorems in a formal science. For the system, this implies that these scaling relationships should be treated as well-supported but revisable empirical findings rather than foundational truths, and that confidence in them rests on the quality and breadth of observation rather than on logical necessity.