mathematical-completeness-fails-to-guarantee-practical-reliability
IN derived (depth 3)
Created 2026-06-21T10:10:05+00:00 · Reviewed 2026-06-21T14:41:08+00:00
RLHF and prompting both illustrate cases where formal or mathematical specification proves insufficient for practical reliability: RLHF has a fully specified mathematical pipeline yet naive implementations fail without dozens of engineering details (motivating simpler alternatives like DPO/IPO/KTO), while prompting exhibits irreducible sensitivity and architectural injection vulnerabilities rooted in the model's inability to formally parse prompt structure. These two examples suggest that in at least some core LLM techniques, mathematical completeness or formal specification does not guarantee practical reliability.
Justifications
SL — Two independent subsystems (training-time alignment, inference-time control) exhibit the same completeness-without-reliability pattern
Antecedents (all must be IN):
- IN rlhf-completeness-paradox-motivated-simpler-alternatives — RLHF has a fully specified mathematical pipeline (Bradley-Terry, KL penalty, PPO clipping), yet DPO, IPO, and KTO independently eliminated its separate reward model and RL loop by collapsing alignment into a single optimization step with shared β-controlled KL regularization — suggesting that the multi-component pipeline's complexity was a significant factor motivating simpler alternatives.
- IN prompt-vulnerability-is-dual-architectural-limitation — Both prompt sensitivity (40%+ accuracy shifts from formatting) and prompt injection (inability to distinguish instructions from input) appear to be intrinsic architectural properties of instruction-following LLMs that resist improvement through scaling alone. Both phenomena may share a common factor in the model's lack of formal prompt structure parsing, though the antecedents establish each independently rather than proving a unified root cause.
Dependents
These beliefs depend on this one:
- IN alignment-diversification-was-theoretically-inevitable — Alignment diversification beyond RLHF was theoretically inevitable rather than merely pragmatically convenient: RLHF's irreducible theoretical complexity (non-Markovian optimal policies, online/offline formulation divergence) combined with the broader pattern of mathematical completeness failing to guarantee practical reliability left no viable path to reliable alignment through a single paradigm.
- IN formal-understanding-insufficient-across-llm-stack — Formal theoretical understanding consistently proves insufficient across the entire LLM stack: RLHF's complete mathematical specification fails without dozens of engineering details, prompting's irreducible sensitivity resists formal analysis, the capacity bottleneck inverts between pretraining and alignment stages — and the inversion means that even a correct scaling theory for one stage actively misleads for the next.