Context Faithfulness

70 beliefs (70 IN, 0 OUT)

The Context Faithfulness topic addresses a central problem in modern LLM usage: models frequently favor their parametric (memorized) knowledge over information explicitly provided in the input context, producing outputs that are unfaithful to the source even when accurate. This matters because it undermines the reliability of retrieval-augmented generation, instruction-following, and any pipeline where the context is supposed to be the authoritative source. The beliefs establish that the phenomenon is not an artifact of poorly constructed counter-evidence (prior work using word-level entity substitution produced incoherent text that models could trivially reject, per chameleon-prior-work-incoherent-counter), but a genuine confirmation bias that grows monotonically with the proportion of parametric memory among all evidence (chameleon-quantity-mr-monotonic) and intensifies as the absolute count of supporting evidence increases even at fixed ratios (chameleon-quantity-amplifies-bias). The Xu et al. 2024 survey formalizes this within a three-type taxonomy—context-memory, inter-context, and intra-memory conflict (three-conflict-types-cm-ic-im, xu2024-three-conflict-types-taxonomy)—and identifies temporal misalignment and misinformation pollution as the two primary causes (cm-two-primary-causes). A key conceptual distinction drawn throughout is that faithfulness (grounding in provided context) is separate from accuracy (producing the correct answer), since a model can be accurate via parametric recall yet entirely unfaithful (zhou-faithfulness-distinct-from-accuracy).

The beliefs cluster around three complementary intervention strategies, all of which are training-free and operate at inference or prompt time. First, Context-Aware Decoding (CAD, NAACL 2024, cad-authors-and-affiliations) modifies the sampling distribution by subtracting the context-free logit from the context-conditioned logit, scaled by a contrastive strength parameter α, in a structure explicitly analogous to classifier-free guidance in diffusion models (cad-core-decoding-formula, cad-classifier-free-guidance-analogy). CAD is a contrastive ensemble of the same model's two logit vectors rather than a blend of different models (cad-contrastive-ensemble-same-model), which distinguishes it from DExperts and contrastive decoding (cad-distinguishes-from-dexperts). Its theoretical grounding uses Pointwise Mutual Information and a product-of-experts distribution that must be softmax-renormalized to form a valid probability (cad-pmi-theoretical-grounding, cad-softmax-renormalization-required). Empirically, CAD yields its largest gains in knowledge-conflict settings where context contradicts training-time priors (cad-knowledge-conflict-greatest-gain), with improvements scaling with model size because larger models lean more heavily on parametric memory (cad-gain-scales-with-model-size, cad-model-size-benefit-knowledge-conflict); GPT-Neo 20B achieves +128% on NQ-Swap (cad-gpt-neo-20b-nqswap-gain) and LLaMA-30B gains +21% ROUGE-L on CNN-DM (cad-llama30b-cnn-dm-results). The cost is two forward passes per token (cad-two-forward-passes-per-token), and the method becomes a no-op if the context is conditionally independent of generation given the prompt (cad-conditional-independence-noop). Second, the Zhou et al. EMNLP 2023 prompting framework (zhou-2023-emnlp-usc-microsoft-research) combines opinion-based and instruction-based prompts with counterfactual demonstrations, reducing the Memorization Ratio on Natural Questions from 35.2% to 3.0% for GPT-3.5 (zhou-memorization-ratio-reduced-35.2-to-3.0, zhou-2023-memorization-ratio-formula) and showing that counterfactual demonstrations can outperform factual ones as few-shot exemplars (zhou-2023-counterfactual-demos-outperform-factual). This approach is fully black-box compatible (zhou-black-box-api-compatible, zhou-2023-no-fine-tuning-no-whitebox), though smaller models (≤6.7B) tend to overabstain under opinion prompts (zhou-small-models-overabstain-opinion-prompts, zhou-2023-smaller-llm-opinion-degrades-abstention). Third, the Xie et al. ICLR 2024 "chameleon/sloth" study (chameleon-iclr-2024) provides a diagnostic framework with five steps (chameleon-five-step-framework) and finds that model behavior is dual: LLMs are receptive to coherent single-source counter-memory but strongly confirmation-biased when both supporting and conflicting evidence coexist (chameleon-dual-behavior-chameleon-sloth), with GPT-4 and Vicuna-33B being the most resistant to counter-memory (chameleon-gpt4-vicuna-resist-counter) and evidence-order sensitivity varying by model (chameleon-order-sensitivity-model-dependent).

Several cross-cutting distinctions are important. CAD, the Zhou prompting methods, and the chameleon diagnostic are independent contributions that address the same underlying failure mode from different angles—decoding distribution, prompt engineering, and behavioral measurement respectively—and are not presented as competing but as complementary. The Xie et al. 2021 beliefs (xie-2021-delimiter-deterministic-emission, xie-2021-mixture-of-hmms-setting, xie-2021-tv-distance-bound-assumption-3, xie-2021-in-context-predictor-argmax, xie-2021-prompt-structure-delimiter, xie-2021-gpt3-lambada-triviaqa-improvement) concern a theoretical in-context learning analysis via mixture-of-HMMs and appear to serve as a background premise for understanding how pretraining distributions condition on prompts, rather than a direct intervention for context faithfulness. RAG is noted not to eliminate context-memory conflict because parametric weights remain unchanged; it merely adds a new potential contradiction source (rag-does-not-eliminate-cm-conflict). The evidence that models genuinely read context (not just re-rank retrieval) is supported by accuracy scaling with document-relevance count independently of BM25 recall (context-faithfulness-evidence). All beliefs in this topic are currently IN; none have been retracted, so the full constellation of claims stands as a coherent and mutually consistent set of knowledge about the problem and its mitigations.

IN cad-alpha-task-specific-values
The recommended α values are 0.5 for summarization tasks (CNN-DM, XSUM) and 1.0 for knowledge-conflict tasks (MemoTrap, NQ-Swap).
IN cad-alpha-zero-standard-decoding
Setting α=0 in the CAD formula reduces the adjusted distribution exactly to standard autoregressive decoding (p_θ(y_t|c,x,y_
IN cad-authors-and-affiliations
The CAD paper (NAACL 2024) is authored by Shi, Han, Lewis, Tsvetkov, Zettlemoyer, and Yih from University of Washington and Meta AI.
IN cad-benchmarks-used
CAD is evaluated on summarization benchmarks (XSUM, CNN-DM) and knowledge-conflict benchmarks (Memotrap, NQSWAP) across OPT models of varying sizes.
IN cad-classifier-free-guidance-analogy
CAD's contrastive structure (1+α)·logit(y|c,x) − α·logit(y|x) is explicitly framed as the text-generation analogue of classifier-free guidance in diffusion models (1+α)·ε_θ(x,c) − α·ε_θ(x).
IN cad-conditional-independence-noop
If context c is conditionally independent of the generation given (x, y_
IN cad-contrastive-ensemble-same-model
In its standard form, CAD is a contrastive ensemble of two logit vectors from the same model (with-context and without-context), not a blend of two different models, though multi-model generalization is possible.
IN cad-core-decoding-formula
The CAD decoding formula is y_t ~ softmax[(1+α)·logit_θ(y_t|c,x,y_
IN cad-default-alpha-0-5
The empirically robust default value for CAD's hyperparameter α is 0.5, with larger values providing additional benefit specifically on knowledge-conflict tasks (Memotrap, NQSWAP).
IN cad-distinguishes-from-dexperts
Unlike DExperts (which demotes a separate anti-expert model) and contrastive decoding (Li et al., 2022) (which demotes a separate small amateur model), CAD demotes the same model's own context-free prior-knowledge distribution.
IN cad-gain-scales-with-model-size
On knowledge-conflict tasks (MemoTrap, NQ-Swap), the relative performance gain from CAD increases with model size, indicating larger models rely more heavily on prior knowledge.
IN cad-gpt-neo-20b-nqswap-gain
GPT-Neo 20B achieves +128% improvement on NQ-Swap and +54.4% on MemoTrap with CAD over standard decoding.
IN cad-knowledge-conflict-greatest-gain
CAD shows its largest performance gains in the knowledge-conflict setting where the input context contradicts the model's training-time prior knowledge.
IN cad-llama30b-cnn-dm-results
LLaMA-30B on CNN-DM achieves +21% ROUGE-L, +14.3% FactKB, and +7.8% BERT-Precision with CAD versus standard decoding.
IN cad-memotrap-construction
MemoTrap presents well-known proverbs with an instruction to end them with a different word (e.g., 'Better late than ___' → 'early' instead of the memorized 'never'), testing instruction-following over memorized completions.
IN cad-model-size-benefit-knowledge-conflict
On knowledge-conflict benchmarks (Memotrap, NQSWAP), CAD's improvement increases with model size because larger models have a stronger tendency to rely on parametric prior knowledge.
IN cad-nqswap-construction
NQ-Swap is constructed by taking NQ questions, identifying the gold named-entity answer, finding its supporting document, and replacing the gold entity with a random entity; a faithful model must answer with the replaced entity.
IN cad-pmi-theoretical-grounding
CAD's theoretical justification uses Pointwise Mutual Information between the context and the generated token, conditioned on the query and previously generated tokens, expressed as a product-of-experts distribution.
IN cad-softmax-renormalization-required
The raw product-of-experts expression p_θ(y_t|c,x,y_
IN cad-training-free-inference-time
Context-Aware Decoding (CAD) requires no fine-tuning or parameter updates; it modifies only the decoding/sampling distribution at inference time on pretrained models.
IN cad-two-forward-passes-per-token
CAD requires two forward passes per generated token: one with context (c, x, y_
IN chameleon-closed-book-qa-elicitation
Parametric memory elicitation uses closed-book QA where the LLM is prompted to answer a question and explain its reasoning, with the explanation constituting the parametric memory.
IN chameleon-dual-behavior-chameleon-sloth
LLMs exhibit dual behavior: high receptiveness to coherent single-source counter-memory (chameleon-like) but strong confirmation bias preferring parametric memory when both supporting and conflicting evidence are present simultaneously (sloth-like).
IN chameleon-five-step-framework
The LLM knowledge conflict framework consists of 5 steps: parametric memory elicitation, counter-memory construction, entailment checking (DeBERTa-V2), answer consistency verification, and evidence preference evaluation in multiple-choice format.
IN chameleon-gpt4-popularity-mr
GPT-4 shows the strongest confirmation bias for high-popularity questions, reaching up to ~90% memorization ratio at 10^4–10^5 Wikipedia page views.
IN chameleon-gpt4-vicuna-resist-counter
GPT-4 and Vicuna-33B resist counter-memory most strongly, still predominantly following their parametric memory even at a 1:3 evidence ratio favoring the counter-answer.
IN chameleon-iclr-2024
The LLM knowledge conflict paper by Xie et al. was published at ICLR 2024.
IN chameleon-models-evaluated
The LLM knowledge conflict study evaluates 3 closed-source models (ChatGPT, GPT-4, PaLM2) and 5 open-source models (Qwen-7B, Llama2-7B/70B, Vicuna-7B/33B) in zero-shot settings.
IN chameleon-nli-model-deberta-v2
DeBERTa-V2 (microsoft/deberta-v2-xxlarge-mnli) is used for entailment checking of counter-memory, validated at 99% accuracy on 200 manually checked samples.
IN chameleon-order-sensitivity-model-dependent
Evidence order sensitivity is model-dependent: ChatGPT favors first-presented evidence, while PaLM2 and Llama2-7B favor later-presented evidence with fluctuations exceeding 30%.
IN chameleon-prior-work-incoherent-counter
Prior work (Longpre et al., 2021) concluded LLMs are 'stubborn' using word-level entity substitution for counter-memory, which produced incoherent text that LLMs could trivially detect as inconsistent.
IN chameleon-quantity-amplifies-bias
Confirmation bias in LLMs grows as the absolute count of parametric-memory evidence increases, even when the supporting-to-contradicting ratio stays the same (e.g., 1:2 vs. 2:4).
IN chameleon-quantity-mr-monotonic
As the proportion of parametric memory among all evidence increases from 0% to 100%, memorization ratio rises monotonically (e.g., ChatGPT goes from 3.7% MR at 0% PM to 99.8% at 100% PM on POPQA).
IN cm-two-primary-causes
The two primary causes of context-memory conflict are temporal misalignment (training data outdated, context is correct) and misinformation pollution (context contains false info, memory is correct).
IN context-faithfulness-evidence
The model genuinely reads and uses provided context (not merely re-ranking retrieval) because accuracy scales with document-relevance count even when BM25 recall shows only mild dependence on the same variable.
IN llm-confirmation-bias-favors-memory
Empirically, LLMs exhibit confirmation bias by favoring information consistent with their internal parametric memory over strong external contextual evidence (Chen et al., 2022; Xie et al., 2023).
IN pcnn-sentence-bert-pairwise-contradiction
PCNN (Hsu et al., 2021) uses fine-tuned Sentence-BERT embeddings to compute pairwise contradiction probabilities for inter-context conflict detection.
IN rag-does-not-eliminate-cm-conflict
RAG does not eliminate context-memory conflict because the model's parametric weights remain unchanged; it merely adds a new potential source of contradiction.
IN three-conflict-types-cm-ic-im
The Xu et al. 2024 survey defines three knowledge conflict types: Context-Memory (CM), Inter-Context (IC), and Intra-Memory (IM), with CM being the most extensively studied.
IN xie-2021-delimiter-deterministic-emission
Assumption 1 in Xie et al. (2021) states that a subset D of hidden states deterministically emits the delimiter token: p(o_delim | h_delim, θ) = 1 for h_delim ∈ D, and 0 otherwise, so the delimiter reveals state membership in D but not which element.
IN xie-2021-gpt3-lambada-triviaqa-improvement
GPT-3 (Brown et al., 2020) improved LAMBADA by 18% and TriviaQA by 3% over prior state-of-the-art.
IN xie-2021-in-context-predictor-argmax
The in-context predictor is defined as fₙ(x_test) = argmax_y p(y | Sₙ, x_test), i.e., the argmax over the pretraining distribution conditioned on the full prompt, corresponding to greedy decoding in practice.
IN xie-2021-mixture-of-hmms-setting
The pretraining distribution in Xie et al. (2021) is modeled as a mixture of HMMs: p(o₁,...,o_T) = ∫ p(o₁,...,o_T|θ) p(θ) dθ, where θ parameterizes the HMM transition matrix.
IN xie-2021-prompt-structure-delimiter
The prompt format in Xie et al. (2021) is [x₁, y₁, o_delim, x₂, y₂, o_delim, …, xₙ, yₙ, o_delim, x_test] ∼ p_prompt, where each example is independently generated from a fresh start hidden state.
IN xie-2021-tv-distance-bound-assumption-3
Assumption 3 in Xie et al. (2021) bounds distribution shift via total variation: max TV(p_prompt(h) ‖ p(h|h_delim, θ*)) < Δ/4, where Δ is the margin between the most and second-most likely labels.
IN xu2024-three-conflict-types-taxonomy
The Xu et al. EMNLP 2024 survey classifies knowledge conflicts in LLMs into three types: context-memory (prompt vs. parametric), inter-context (between provided passages), and intra-memory (self-contradiction within parametric knowledge).
IN zhou-2023-ape-instructions-per-dataset
The top-1 APE-generated instructions used were: Natural Questions: 'Read the given information and answer the corresponding question.', Re-TACRED: 'Identify the relationship between two entities from a list of options.', RealTime QA: 'Answer a question based on the provided input-output pairs.'
IN zhou-2023-ape-text-davinci-003-instruction-gen
Because smaller LLMs failed to produce meaningful instructions, text-davinci-003 was used to generate instructions via APE across all model scales tested, with top-3 instructions generated per dataset and the top-1 selected for experiments.
IN zhou-2023-brier-score-reduction-gpt35
OPIN+INSTR reduced the Brier score by 24.2% in zero-shot and 7.8% in few-shot settings for GPT-3.5 (text-davinci-003) compared to the base prompt.
IN zhou-2023-counterfactual-demos-outperform-factual
Counterfactual demonstrations (where context facts are replaced with false ones) outperformed original factual demonstrations as few-shot exemplars, and sometimes even outperformed zero-shot inference on Wikipedia-based MRC data.
IN zhou-2023-emnlp-usc-microsoft-research
The context-faithful prompting paper by Zhou, Zhang, Poon, and Chen (USC + Microsoft Research) was published at EMNLP 2023 (arXiv:2303.11315).
IN zhou-2023-evaluation-two-tasks-three-datasets
The evaluation spans two tasks—machine reading comprehension (knowledge conflict) and relation extraction (prediction with abstention)—on three datasets, using two model scales (175B InstructGPT and 7B LLaMA-2-chat) in both zero-shot and few-shot regimes.
IN zhou-2023-funding-nsf-darpa
The paper's work was funded by NSF Grant IIS-2105329 and DARPA MCS (Contract N660011924033).
IN zhou-2023-gpt35-mr-reduction-zero-shot
For GPT-3.5 (text-davinci-003), the zero-shot to OPIN+INSTR+Counterfactual transition yielded a 32.2% reduction in MR for MRC and a 10.9% reduction for RE.
IN zhou-2023-joint-vs-pertoken-probability
For abstention probability calculation, joint probability works best for GPT-3.5 while per-token probability (joint normalized by length) works best for LLaMA-2.
IN zhou-2023-llama2-mr-reduction-mrc-re
For LLaMA-2-7B-chat, the OPIN+INSTR+Counterfactual method achieved a 39.4% reduction in MR for MRC and 57.3% reduction for RE compared to zero-shot baseline.
IN zhou-2023-max-16-demonstrations
The maximum number of few-shot demonstration instances used in experiments was 16, limited by the context window; abstention tasks used 3 demonstrations per instance.
IN zhou-2023-memorization-ratio-formula
The Memorization Ratio (MR) is defined as MR = po / (po + ps), where po is the frequency of predicting the original (memorized) answer and ps is the frequency of predicting the substituted (counterfactual) answer; a fully faithful model has MR = 0.
IN zhou-2023-no-fine-tuning-no-whitebox
The proposed context-faithful prompting methods require no fine-tuning, no white-box access to model internals, and no confidence re-calibration, distinguishing them from prior work such as Neeman et al. and Wang et al. 2022/2023.
IN zhou-2023-opinion-instruction-combination-best
The combination of opinion-based prompting and instruction-based prompting (OPIN+INSTR) produced the best performance, ranking first on 23 out of 24 metrics for GPT-3.5 in knowledge conflict and 19/24 for LLaMA-2.
IN zhou-2023-realtime-qa-50-unanswerable
In the RealTime QA abstention task, 50 of 113 test instances were unanswerable (retrieved documents did not contain the answer) and were relabeled to the 'I don't know' option.
IN zhou-2023-smaller-llm-opinion-degrades-abstention
For smaller LLMs (LLaMA-2-7B-chat), opinion-based prompting can degrade abstention performance (higher Brier score) because it converts uncertain-but-correct predictions to 'I don't know'.
IN zhou-black-box-api-compatible
The context-faithful prompting methods work with black-box API LLMs and do not require white-box model access, unlike confidence calibration approaches.
IN zhou-code-repository-github
Code and data for Zhou et al. (2023) context-faithful prompting are available at https://github.com/wzhouad/context-faithful-llm.
IN zhou-eval-datasets-nq-rtqa-retacred
Experiments are conducted on machine reading comprehension datasets (Natural Questions, RealTime QA, SQuAD 2.0, CoQA, QuAC) and relation extraction dataset Re-TACRED.
IN zhou-faithfulness-distinct-from-accuracy
Faithfulness is defined as the validity of the extraction process (grounding in provided context), distinct from answer accuracy, since a model can be accurate via parametric recall yet unfaithful.
IN zhou-memorization-ratio-reduced-35.2-to-3.0
Applying the proposed prompting strategies reduced the memorization ratio of text-davinci-003 from 35.2% to 3.0% on the Natural Questions dataset.
IN zhou-no-degradation-on-factual-contexts
On non-conflicting (original/factual) contexts, opinion-based and combined prompts yield exact-match scores of 57.2%–80.1%, showing no performance degradation compared to base prompting.
IN zhou-small-models-overabstain-opinion-prompts
Smaller LLMs (≤6.7B parameters) show worse selective prediction with opinion-based prompts because they lack sufficient reading-comprehension ability and misclassify answerable questions as 'I don't know.'
IN zhou-training-free-prompting-strategies
Both opinion-based prompting and counterfactual demonstrations are purely prompt-level interventions requiring no additional model training or weight updates.