prompt-sensitivity-is-fundamental-not-solvable-by-scale
IN derived (depth 1)
Created 2026-06-21T09:52:15+00:00 · Reviewed 2026-06-21T14:41:08+00:00
LLM performance sensitivity to prompt formatting (40%+ accuracy shifts from reordering, 76-point swings from format changes) persists across model sizes, few-shot counts, and instruction tuning — it is an intrinsic property, not a fixable deficiency.
Summary
How you arrange and format the text you feed into a language model can swing its accuracy by 40 to 76 points, and this fragility is baked into the architecture rather than being a temporary flaw that bigger models or more training data will smooth out. In practice, that means prompt structure is a permanent, first-order engineering variable: you have to treat it with the same rigor as choosing a model, because "just upgrade to a larger model" will not make your prompts suddenly robust.
Justifications
SL — Sensitivity persists at scale, ruling out the hypothesis that it's a small-model artifact
Antecedents (all must be IN):
- IN prompt-sensitivity-40-percent-accuracy-shift — LLM performance is highly sensitive to prompt design, with accuracy shifts of over 40 percentage points from minor changes such as reordering examples, and up to 76 accuracy points difference across formatting changes
- IN prompt-sensitivity-persists-at-scale — Prompt sensitivity persists even with larger models, more few-shot examples, or instruction tuning — it is not solved by scaling alone
- IN prompt-sensitivity-reorder-40pct-format-76pts — LLMs are highly sensitive to prompt formatting: reordering few-shot examples can shift accuracy by 40%+, and formatting changes can cause up to 76 accuracy points of variation
Dependents
These beliefs depend on this one:
- IN prompt-vulnerability-is-dual-architectural-limitation — Both prompt sensitivity (40%+ accuracy shifts from formatting) and prompt injection (inability to distinguish instructions from input) appear to be intrinsic architectural properties of instruction-following LLMs that resist improvement through scaling alone. Both phenomena may share a common factor in the model's lack of formal prompt structure parsing, though the antecedents establish each independently rather than proving a unified root cause.
- IN prompting-sophistication-compensates-for-irreducible-sensitivity — Prompt sensitivity is a persistent, intrinsic property not resolved by scaling, and the field developed increasingly structured prompting approaches (CoT → self-consistency → tree-of-thoughts) that add search structure to reasoning. These techniques manage prompt-dependent variability by structuring the reasoning process, though the antecedents do not establish that this was the explicit motivation for their development.