Prompting & Reasoning

48 beliefs (48 IN, 0 OUT)

The "Prompting & Reasoning" topic covers the full arc of how language models are directed to perform reasoning tasks, from the foundational chain-of-thought technique proposed by Wei et al. at Google Brain (chain-of-thought-wei-neurips-2022) through its zero-shot variant triggered by the phrase "Let's think step by step" (zero-shot-cot-phrase-let-think-step-by-step, kojima-2022-zero-shot-cot-prompt), to the progressive addition of search structure via self-consistency majority voting (self-consistency-multiple-cot-majority-vote) and tree-of-thoughts branching with backtracking (tree-of-thought-generalizes-cot, tree-of-thoughts-neurips-2023). This evolution from linear to branching reasoning is explicitly tracked as a single trajectory (structured-reasoning-prompting-evolved-from-linear-to-branching). Several beliefs pin down hard empirical constraints on these techniques: chain-of-thought only improves performance at roughly 62 billion parameters and above, with smaller models doing better on direct answers (cot-prompting-effective-only-at-62b-plus-parameters, cot-prompting-requires-62b-plus-params), and the few-shot variant requires at least five worked examples with intermediate steps (cot-prompting-requires-min-five-few-shot-examples). The field's history stretches back to a 1999 rule-based Intelligent Filling Manager, through the 2018 first multi-task NLP model, to the November 2022 ChatGPT release that cemented prompt engineering as a business skill (prompt-engineering-history-milestones-1999-2022), and the term was named Oxford Word of the Year for 2023 (prompt-engineering-oxford-word-of-year-2023).

A central tension in the knowledge base is that prompting is simultaneously automatable and deeply fragile. On one hand, LLMs can generate and select their own prompts at human-level quality (ape-llms-human-level-prompt-engineers), and a suite of automated optimizers has emerged: GEPA uses reflective evolutionary search with Pareto-based candidate selection (gepa-pareto-evolutionary-prompt-optimizer, gepa-reflective-prompt-evolution-2025), MIPRO makes incremental per-module prompt changes retained only when they improve a downstream metric (mipro-incremental-module-prompt-optimization), DSPy compiles declarative LM calls into self-improving pipelines (dspy-declarative-lm-pipeline-compiler), and Auto-CoT clusters a question library to build diverse demonstration sets (auto-cot-bert-clustering-diverse-demonstrations). On the other hand, LLM accuracy shifts by over 40 percentage points from merely reordering few-shot examples and up to 76 points from formatting changes (prompt-sensitivity-40-percent-accuracy-shift, prompt-sensitivity-reorder-40pct-format-76pts), strategies do not transfer reliably across models or even minor reformulations (prompt-engineering-model-specific), and this sensitivity persists across model sizes, shot counts, and instruction tuning, making it an intrinsic property rather than a fixable deficiency (prompt-sensitivity-is-fundamental-not-solvable-by-scale). The knowledge base captures this as a paradox: prompt optimization must be continuous, model-specific, and potentially self-maintaining rather than a one-time engineering effort (prompt-optimization-is-paradoxically-automatable-yet-fragile). A separate family of beliefs addresses parameter-efficient prompting that learns continuous soft-token vectors via gradient descent while freezing base weights, as in prompt tuning (prompt-tuning-soft-tokens-backprop) and prefix-tuning (soft-prompting-gradient-descent-continuous-vectors), offering a complementary axis to discrete textual prompting. MeLLo extends this by operating purely through prompting and retrieval with no gradient updates or weight access, making it applicable to black-box models (mello-requires-no-weight-access).

Several beliefs extend the topic into security, systems, and meta-disciplinary concerns. Prompt injection exploits the model's structural inability to distinguish developer instructions from user inputs, drawn by analogy to SQL injection and XSS (prompt-injection-exploits-instruction-ambiguity), and the knowledge base notes a deeper parallel: both compiler lexers and LLM inference face a fundamental disambiguation problem where a processing layer cannot resolve its inputs without higher-level semantic context it structurally lacks (disambiguation-requires-semantic-context-across-processing-paradigms). In chat-format systems, system prompts carry higher authority than user messages (system-prompt-higher-authority-than-user), which is precisely the asymmetry injection attacks target. The broader discipline of context engineering is framed as the successor to prompt engineering, managing the full context window including system instructions, metadata, API tools, and conversation summaries with practices like token budgeting and provenance tagging, especially for agentic applications from 2025 onward (context-engineering-successor-to-prompt-engineering, context-engineering-broader-than-prompt-engineering). Resource consumption is quantified across three independent dimensions—memory at 2 bytes per float16 parameter, compute at 6 FLOPs per parameter per token for training and 1-2 for inference, and energy ranging from roughly 0.002 Wh for classification to 2.91 Wh for image generation (llm-resource-consumption-quantifiable-across-three-dimensions, energy-per-prompt-classification-vs-generation-vs-image)—and disaggregated inference separates the compute-bound prefill phase from the memory-bound decode phase onto specialized hardware (disaggregated-inference-prefill-compute-bound-decode-memory-bound). A 2024 survey found over 50 text-based prompting techniques, 40 multimodal variants, and 33 distinct terms, underscoring a significant lack of standardization in the field (prompting-technique-survey-2024-lack-standardization, prompting-techniques-survey-50-text-40-multimodal). Finally, a small cluster of beliefs touches on memorization and faithfulness: GPT-2 models contain 1 to 7 percent exact duplicate outputs from training data that can be triggered by repetition prompting (gpt2-memorization-1-to-7-pct-exact-duplicates), the Counter-memory Construction prompt explicitly permits fabrication of realistic fake content (counter-memory-construction-permits-fabrication), and prompting-based faithfulness methods are restricted to large instruction-finetuned models and do not generalize to small or open-source ones (prompting-faithfulness-limited-instruction-tuned). The knowledge base also includes methodological beliefs about isolating causal head contributions via shuffled-label prompts (cma-shuffled-label-prompt-isolation) and reducing feature noise by requiring activity on all positive prompts and inactivity on all negative ones (multi-prompt-filtering-logic), and notes that chain-of-thought slightly narrows the generation gap from 0.60 to 0.52, suggesting structured generation is less model-specific than free-form generation (chain-of-thought-reduces-generation-gap-from-060-to-052). All beliefs in this group are marked IN; none have been retracted, so the knowledge base currently presents a fully coherent, if dense, picture of the prompting landscape without internal contradictions or withdrawn claims.

IN and-your-knowledge-phrase-conditional-rule
The phrase 'and your knowledge' appears in the prompt template only when the evidence source is entirely parametric memory or counter-memory; it is omitted in all other conditions.
IN ape-llms-human-level-prompt-engineers
Automatic Prompt Engineer (APE) demonstrated that LLMs can generate and select their own prompts at human-level quality, using beam search scored by output log-probabilities (ICLR 2023, arXiv:2211.01910)
IN auto-cot-bert-clustering-diverse-demonstrations
Auto-CoT vectorizes a question library via BERT, clusters the vectors, selects near-centroid questions per cluster for diversity, generates zero-shot CoT answers for each, and assembles them into a diverse few-shot demonstration set.
IN chain-of-thought-google-brain-2022
Chain-of-thought prompting was proposed by Google Brain in 2022; zero-shot CoT works by appending 'Let's think step-by-step' to the prompt
IN chain-of-thought-prompting-technique
Chain-of-thought prompting (Wei et al. 2022) elicits reasoning by having models show intermediate steps — it is a prompting technique, not an architectural change
IN chain-of-thought-reduces-generation-gap-from-060-to-052
Chain-of-thought prompting slightly reduces the generation gap from 0.60 to 0.52, suggesting structured generation is less model-specific than free-form generation.
IN cma-shuffled-label-prompt-isolation
The Causal Indirect Effect metric isolates causal head contribution by replacing head activations with task-conditioned means on shuffled-label prompts, ruling out heads that merely correlate with correct answers.
IN context-engineering-broader-than-prompt-engineering
Context engineering is the broader software engineering discipline managing all context supplied to a model: system instructions, metadata, API tools, tokens, conversation summaries, and task metadata, with practices including token budgeting, provenance tags, versioning, and observability
IN context-engineering-successor-to-prompt-engineering
Context engineering is the emerging successor discipline to prompt engineering, focused on managing the full context window including system instructions, metadata, API tools, and tokens — especially for agentic applications (2025 onward)
IN cot-prompting-effective-only-at-62b-plus-parameters
Chain-of-thought prompting (e.g., 'Let's think step by step') only improves performance at models with 62B or more parameters; smaller models perform better with direct answers.
IN cot-prompting-requires-62b-plus-params
Chain-of-thought prompting only improves performance for models with at least approximately 62 billion parameters; smaller models perform better answering directly.
IN cot-prompting-requires-min-five-few-shot-examples
Chain-of-Thought prompting requires at least 5 few-shot examples with intermediate reasoning steps to elicit multi-step arithmetic or logical reasoning in LLMs (Wei et al., 2023)
IN cot-prompting-wei-neurips-2022
Chain-of-Thought prompting was proposed by Wei et al. at Google Brain and published at NeurIPS 2022 (arXiv:2201.11903)
IN counter-memory-construction-permits-fabrication
The Counter-memory Construction prompt explicitly states 'You can make up fake content and supporting evidence but it should be as realistic as possible.'
IN disaggregated-inference-prefill-compute-bound-decode-memory-bound
Disaggregated inference separates the prefill phase (compute-bound, processes the full prompt in parallel) from the decode phase (memory-bound, generates one token at a time) onto different specialized hardware
IN disambiguation-requires-semantic-context-across-processing-paradigms
Both compiler lexers and LLM inference face the same fundamental disambiguation problem: the C lexer hack requires semantic feedback from the symbol table to distinguish typedef names from variable names, and prompt injection exploits the model's inability to distinguish developer instructions from user inputs — in both cases, a processing layer cannot disambiguate its inputs without higher-level semantic context that it structurally lacks.
IN dspy-declarative-lm-pipeline-compiler
DSPy is a framework that compiles declarative language model calls into self-improving optimized pipelines, representing the shift from manual to programmatic prompt engineering (Khattab, 2023, arXiv:2310.03714)
IN energy-per-prompt-classification-vs-generation-vs-image
Energy cost per prompt varies by task type: classification ~0.002-0.007 Wh, text generation ~0.05 Wh, image generation ~2.91 Wh (up to 11.49 Wh for least efficient models).
IN gepa-pareto-evolutionary-prompt-optimizer
GEPA (Agrawal 2025) combines LM-based trace analysis with Pareto-based evolutionary search over candidate prompt systems, reporting ~10% gains over GRPO and MIPROv2 with up to 35× fewer rollouts.
IN gepa-reflective-prompt-evolution-2025
GEPA (Agrawal, 2025) uses a reflective prompt evolution loop of generate-evaluate-reflect-mutate to optimize prompts, reportedly outperforming reinforcement learning in some settings.
IN gpt2-memorization-1-to-7-pct-exact-duplicates
GPT-2 models were measured to contain 1% to approximately 7% exact duplicate outputs from training data, and repetition prompting can trigger memorized excerpts.
IN kojima-2022-zero-shot-cot-prompt
Kojima et al. (2022) showed that appending 'Let's think step by step' as a zero-shot prompt elicits reasoning without any demonstrations, a mechanism distinct from few-shot ICL.
IN llm-resource-consumption-quantifiable-across-three-dimensions
LLM resource consumption is precisely quantifiable across three independent dimensions — memory (2 bytes per float16 parameter), compute (6 FLOPs per parameter per token for training, 1-2 for inference), and energy (0.002-0.05 Wh per prompt depending on task type) — enabling principled cost modeling for deployment decisions.
IN mello-requires-no-weight-access
MeLLo requires no gradient updates, no fine-tuning, and no access to model weights; it operates purely via prompting and retrieval, making it applicable to black-box LMs like GPT-3.5.
IN mipro-incremental-module-prompt-optimization
MIPRO proposes small incremental changes to module prompts in multi-stage LM programs, retaining only changes that improve a downstream metric.
IN multi-prompt-filtering-logic
Multi-prompt filtering retains only features that are active on ALL positive prompts AND inactive on ALL negative prompts, reducing noise from syntax- or content-nonspecific activations.
IN prompt-engineering-history-milestones-1999-2022
Prompt engineering history includes the 1999 Intelligent Filling Manager (rule-based predecessor), the 2018 first multi-task question-answer NLP model, and the November 2022 ChatGPT release which established prompt engineering as a business skill.
IN prompt-engineering-model-specific
Effective prompting strategies are model-specific — a technique that works on one model may degrade performance on another, and techniques do not reliably transfer across models or even across minor prompt reformulations
IN prompt-engineering-oxford-word-of-year-2023
Prompt engineering was named Oxford Word of the Year for 2023 and entered the Oxford English Dictionary in 2025.
IN prompt-injection-exploits-instruction-ambiguity
Prompt injection exploits the model's inability to distinguish developer-defined instructions from user inputs, analogous to SQL injection and XSS
IN prompt-optimization-is-paradoxically-automatable-yet-fragile
Prompt engineering is paradoxically both automatable (LLMs generate prompts at human-level quality) and deeply fragile (model-specific, with 40%+ accuracy shifts from minor formatting changes), implying that prompt optimization must be continuous, model-specific, and potentially self-maintaining rather than a one-time engineering effort.
IN prompt-sensitivity-40-percent-accuracy-shift
LLM performance is highly sensitive to prompt design, with accuracy shifts of over 40 percentage points from minor changes such as reordering examples, and up to 76 accuracy points difference across formatting changes
IN prompt-sensitivity-is-fundamental-not-solvable-by-scale
LLM performance sensitivity to prompt formatting (40%+ accuracy shifts from reordering, 76-point swings from format changes) persists across model sizes, few-shot counts, and instruction tuning — it is an intrinsic property, not a fixable deficiency.
IN prompt-sensitivity-reorder-40pct-format-76pts
LLMs are highly sensitive to prompt formatting: reordering few-shot examples can shift accuracy by 40%+, and formatting changes can cause up to 76 accuracy points of variation
IN prompt-tuning-soft-tokens-backprop
Prompt tuning (Lester et al., 2021) learns k soft-prompt tokens (continuous embeddings) via back-propagation while keeping all base model weights frozen.
IN prompting-faithfulness-limited-instruction-tuned
Prompting-based faithfulness methods (Zhou et al., 2023) are restricted to large instruction-finetuned models such as text-davinci-003 and are not generalizable to small or open-source models.
IN prompting-sophistication-compensates-for-irreducible-sensitivity
Prompt sensitivity is a persistent, intrinsic property not resolved by scaling, and the field developed increasingly structured prompting approaches (CoT → self-consistency → tree-of-thoughts) that add search structure to reasoning. These techniques manage prompt-dependent variability by structuring the reasoning process, though the antecedents do not establish that this was the explicit motivation for their development.
IN prompting-technique-survey-2024-lack-standardization
A 2024 survey identified 50+ text-based prompting techniques, 40 multimodal variants, and 33 distinct terms, highlighting a lack of standardized terminology in the prompt engineering field.
IN prompting-techniques-survey-50-text-40-multimodal
A 2024 survey identified over 50 distinct text-based prompting techniques and 40 multimodal prompting variants
IN self-consistency-multiple-cot-majority-vote
Self-consistency improves Chain-of-Thought by sampling multiple reasoning paths and selecting the most common conclusion via majority vote (Wang et al., ICLR 2023, arXiv:2203.11171)
IN self-consistency-selects-most-common-cot-answer
Self-consistency prompting runs multiple chain-of-thought rollouts and selects the most common conclusion as the final answer
IN soft-prompting-gradient-descent-continuous-vectors
Prefix-tuning (Li & Liang 2021) and prompt tuning (Lester et al. 2021) learn continuous floating-point prefix vectors via gradient descent to maximize output log-likelihood (arg max_X̃ Σᵢ log Pr[Yⁱ | X̃ * Xⁱ]), as a parameter-efficient alternative to discrete token-level prompting.
IN structured-reasoning-prompting-evolved-from-linear-to-branching
Prompting for reasoning evolved from linear chain-of-thought (single path) to self-consistency (multiple paths, majority vote) to tree-of-thoughts (branching with backtracking), progressively adding search structure.
IN system-prompt-higher-authority-than-user
In chat-format LLMs, system prompts carry higher authority than user messages, conveying instructions from a developer or system role.
IN tree-of-thought-generalizes-cot
Tree-of-thought prompting generalizes chain-of-thought by exploring multiple reasoning paths in parallel with backtracking, using tree search algorithms (breadth-first, depth-first, beam search)
IN tree-of-thoughts-neurips-2023
Tree of Thoughts (ToT) extends Chain-of-Thought with deliberate problem-solving via branching reasoning paths and was published at NeurIPS 2023 (arXiv:2305.10601)
IN zero-shot-cot-phrase-let-think-step-by-step
Zero-shot Chain-of-Thought prompting (Kojima et al., 2022) uses the specific instruction 'Let's think step by step' appended to a question to elicit multi-step reasoning without providing worked examples.
IN zero-shot-cot-trigger-phrase
Zero-shot Chain-of-Thought prompting is triggered by appending 'Let's think step by step' to a prompt, as demonstrated by Kojima et al. (NeurIPS 2022, arXiv:2205.11916)