formal-understanding-insufficient-across-llm-stack
IN derived (depth 4)
Created 2026-06-21T10:12:39+00:00 · Reviewed 2026-06-21T14:41:08+00:00
Formal theoretical understanding consistently proves insufficient across the entire LLM stack: RLHF's complete mathematical specification fails without dozens of engineering details, prompting's irreducible sensitivity resists formal analysis, the capacity bottleneck inverts between pretraining and alignment stages — and the inversion means that even a correct scaling theory for one stage actively misleads for the next.
Justifications
SL — Theory-practice gaps at the pipeline level (depth-3) compound with stage-specific scaling inversions (depth-3)
Antecedents (all must be IN):
- IN mathematical-completeness-fails-to-guarantee-practical-reliability — RLHF and prompting both illustrate cases where formal or mathematical specification proves insufficient for practical reliability: RLHF has a fully specified mathematical pipeline yet naive implementations fail without dozens of engineering details (motivating simpler alternatives like DPO/IPO/KTO), while prompting exhibits irreducible sensitivity and architectural injection vulnerabilities rooted in the model's inability to formally parse prompt structure. These two examples suggest that in at least some core LLM techniques, mathematical completeness or formal specification does not guarantee practical reliability.
- IN capacity-bottleneck-inverts-between-pretraining-and-alignment — The transformer's known over-parameterization at the pretraining level (compression to 60% of parameters with 95% performance retained) combined with alignment's inverse priority (scaling reward model size matters more than data volume) reveals a systematic inversion: data volume is the bottleneck for pretraining while model capacity is the bottleneck for alignment.
Dependents
These beliefs depend on this one:
- IN defense-in-depth-required-and-bounded-by-theory-gap — The five-dimensional defense requirement for LLM reliability is both necessitated and bounded by insufficient formal understanding — no single defense layer has theoretical guarantees (hence defense-in-depth is required), but the same theory gap means defense-in-depth itself lacks formal assurance of adequacy, creating an irreducible reliance on empirical validation.
- IN engineering-maturity-systematically-outpaces-theoretical-understanding — The LLM field is systematically characterized by engineering maturity outrunning theoretical understanding — standardized pipelines, reproducible results, and practical compensations consistently succeed at every level of the stack despite fundamental theoretical insufficiency that would normally preclude confidence.