Bayesian Inference

39 beliefs (39 IN, 0 OUT)

This topic collects the core theoretical, historical, and applied claims surrounding Bayesian inference as a framework for statistical reasoning under uncertainty. At its center is Bayes' theorem, P(H|E) = P(E|H)·P(H)/P(E), which the group records in both its standard form (bayes-theorem-posterior-formula) and its proportional version where the posterior is the likelihood times the prior normalized by the marginal likelihood (bayes-rule-posterior-proportional-likelihood-prior). The group traces the historical arc from Thomas Bayes's 1763 demonstration that probabilistic limits could be placed on an unknown event (bayes-1701-1761-probabilistic-limits), through Laplace's formalization of the theorem as Principle VI and its application to celestial mechanics and jurisprudence (laplace-1749-1827-formalized-bayes), to the early terminology of "inverse probability" and Laplace's principle of insufficient reason for uniform priors (inverse-probability-early-terminology). After the 1920s, frequentist methods largely displaced this approach in mainstream statistics (post-1920s-frequentist-dominance), a shift the group contrasts by noting the foundational difference in how parameters are treated: fixed unknowns for frequentists versus random variables with prior distributions for Bayesians (frequentist-fixed-bayesian-random-parameters). The topic matters because it provides the mathematical backbone for updating beliefs in light of evidence, and the group includes both the canonical cookie-bowl worked example yielding P(H₁|E)=0.6 (cookie-problem-posterior-0-6) and a philosophical framing from Jaynes (1998) that positions Bayesian inference as an extension of deductive logic to scientific reasoning (jaynes-1998-logic-of-science).

The theoretical substructure distinguishes several closely related but conceptually separate objects. The likelihood P(E|H) is a function of the data with the hypothesis fixed, while the posterior P(H|E) is a function of the hypothesis with the data fixed (bayes-likelihood-vs-posterior-function). Hyperparameters α that shape the prior are explicitly distinguished from the model parameters θ over which the posterior is estimated (bayes-hyperparameters-distinct-from-parameters). Priors themselves form a hierarchy: conjugate priors keep the posterior in the same parametric family for closed-form updates (conjugate-prior-same-parametric-family); improper priors can sometimes yield proper posteriors but not always, and validity requires the posterior to be proper (improper-prior-can-yield-proper-posterior); and Cromwell's rule warns that assigning zero or one to a prior makes the hypothesis immune to any finite evidence (cromwell-rule-zero-prior-immune). For comparing competing models, the Bayes factor is the ratio of marginal likelihoods, integrating over the full parameter space, and is therefore distinct from a simple likelihood ratio (bayes-factor-ratio-marginal-likelihoods, bayes-factor-evidence-ratio-definition). The marginal likelihood P(E) is identical across hypotheses and cancels in posterior-odds comparisons (marginal-likelihood-constant-across-hypotheses), but if it equals zero, Bayes' rule breaks down entirely (marginal-likelihood-zero-bayes-failure). On the estimation side, the posterior mean and MAP (posterior mode) coincide only for symmetric posteriors (map-posterior-mean-symmetric-only), and Wald's Admissibility Theorem guarantees that every unique Bayesian procedure is admissible and that every admissible procedure is Bayesian or a limit of Bayesian procedures (wald-admissibility-theorem), a result the group narrows to the squared-error-loss setting where all proper-prior Bayes estimators are admissible (bayes-estimator-admissible-squared-error). Sequential updating formalizes the iterative structure: the posterior from one evidence round becomes the next prior, and for i.i.d. observations the likelihoods multiply (bayes-sequential-updating-rule). The group also records a philosophical caveat that Bayesian updating is not the sole rational rule; Jeffrey's rule and other probability-kinematics schemes likewise avoid Dutch-book incoherence, as Hacking and van Fraassen noted (bayes-probability-kinematics-not-unique). Modern practice splits into objective Bayesianism (analysis determined by model, data, and a prior-assignment rule) and subjective Bayesianism (priors encode analyst beliefs) (objective-vs-subjective-bayesian-currents).

Computationally, the group credits the 1980s popularization of Markov chain Monte Carlo methods—Metropolis–Hastings, Gibbs sampling—with removing the barrier of intractable high-dimensional posteriors and catalyzing the field's modern growth (mcmc-1980s-computational-revolution, mcmc-standard-bayesian-posterior-approximation). Gelman et al.'s 2013 Bayesian Data Analysis (3rd edition) is identified as the standard applied reference for MCMC and hierarchical modeling (gelman-2013-bda-3rd-edition). Applications span Bayesian regression, where GLM coefficients become random variables with priors rather than fixed unknowns (bayesian-regression-priors-on-parameters); the Planck 2018 cosmological parameter constraints derived via Bayesian MCMC (planck-2018-bayesian-mcmc-parameters); the UK Court of Appeal's 1996 admission of Bayes' theorem as expert evidence in R v Adams, with a caution against unnecessary theoretical complexity (r-v-adams-1996-bayes-admitted); Klayman and Ha's 1987 result that the positive test strategy is Bayesian-optimal when the true rule is narrow (klayman-ha-positive-test-strategy-bayesian-optimal); and Lord et al.'s 1979 demonstration that participants rated the fictional capital-punishment study matching their prior view as better conducted regardless of data (lord-1979-fictional-studies-swapped-conclusions). At the intersection with modern machine learning, Xie et al. (2021) formalize in-context learning as implicit Bayesian inference over a latent concept, with the posterior predictive as the core object (xie-2021-icl-as-bayesian-inference, xie-2021-icl-implicit-bayesian-inference). A few beliefs in this group—GPT-5's routing architecture (gpt5-router-selects-between-fast-and-reasoning-model), the provenance of SwiGLU, RoPE, and RMSNorm in Llama (swiglu-rope-rmsnorm-not-meta-inventions), the positional-prior shuffling experiment in Liu et al. 2023 (llm-lost-in-middle-shuffle-prior-removal), and the Xu et al. 2024 knowledge-conflict survey and its lack of new experiments (xu-2024-survey-emnlp-first-dedicated, xu2024-no-new-computational-experiments)—appear to have been grouped here through loose keyword overlap with "inference" or "prior" rather than through substantive connection to Bayesian statistical methodology. All beliefs in this topic are marked IN; none have been retracted, so no implications from OUT-status arise.

IN bayes-1701-1761-probabilistic-limits
Thomas Bayes (1701–1761) proved that probabilistic limits could be placed on an unknown event.
IN bayes-estimator-admissible-squared-error
Under squared-error loss, every Bayes estimator with a proper prior is admissible, though exceptions exist for incomplete classes and 0–1 loss contexts.
IN bayes-factor-evidence-ratio-definition
The Bayes factor is defined as P(E|H₁)/P(E|H₂) and quantifies relative evidence for one hypothesis over another.
IN bayes-factor-ratio-marginal-likelihoods
The Bayes factor is the ratio of marginal likelihoods (evidence) of two competing models and is distinct from a simple likelihood ratio because it integrates over the full parameter space.
IN bayes-hyperparameters-distinct-from-parameters
Hyperparameters (α) parameterize the prior distribution itself, distinguishing them from model parameters (θ) that the posterior is estimated over.
IN bayes-likelihood-vs-posterior-function
The likelihood P(E|H) is a function of the data E with the hypothesis H fixed, while the posterior P(H|E) is a function of the hypothesis H with the data E fixed.
IN bayes-probability-kinematics-not-unique
Bayesian updating is not the only rational updating rule; probability kinematics (e.g., Jeffrey's rule) also avoids Dutch-book incoherence, as noted by Hacking and van Fraassen.
IN bayes-rule-posterior-proportional-likelihood-prior
The posterior distribution p(θ|X,α) is proportional to the product of the likelihood p(X|θ,α) and the prior p(θ|α), with the marginal likelihood p(X|α) as the normalizing constant.
IN bayes-sequential-updating-rule
In Bayesian sequential updating, the posterior from one round of evidence becomes the prior for the next round, and for i.i.d. observations the likelihoods multiply.
IN bayes-theorem-posterior-formula
Bayes' theorem states P(H|E) = P(E|H)·P(H) / P(E), where the posterior is proportional to likelihood times prior.
IN bayesian-regression-priors-on-parameters
Bayesian regression extends the GLM framework by treating coefficients as random variables with prior distributions rather than fixed unknowns.
IN conjugate-prior-same-parametric-family
A conjugate prior is a prior family chosen so that the resulting posterior belongs to the same parametric family, enabling closed-form Bayesian updates.
IN cookie-problem-posterior-0-6
In the canonical cookie-bowl problem, with P(H₁)=0.5, P(E|H₁)=0.75, and P(E|H₂)=0.5, the posterior is P(H₁|E)=0.6.
IN cromwell-rule-zero-prior-immune
Cromwell's rule states that if P(M)=0 then P(M|E)=0, and if P(M)=1 and P(E)>0 then P(M|E)=1, meaning hard priors are immune to any evidence.
IN frequentist-fixed-bayesian-random-parameters
Frequentists treat statistical parameters as fixed unknowns, while Bayesians treat them as random variables with prior distributions.
IN gelman-2013-bda-3rd-edition
Gelman, Carlin, Stern, Dunson, Vehtari & Rubin (2013) 'Bayesian Data Analysis' is the 3rd edition and serves as the standard reference for MCMC and hierarchical modeling in applied Bayesian data analysis.
IN gpt5-router-selects-between-fast-and-reasoning-model
GPT-5 (August 2025) introduced a router that selects between a faster model and a slower reasoning model per task, representing a distinct architectural departure from prior GPT generations.
IN improper-prior-can-yield-proper-posterior
An improper prior can sometimes yield a proper posterior, but not always; the posterior must still be proper for Bayesian inference to be valid.
IN inverse-probability-early-terminology
Early Bayesian inference was historically called 'inverse probability,' referring to inferring parameters (causes) from observations (effects), and relied on uniform priors via Laplace's principle of insufficient reason.
IN jaynes-1998-logic-of-science
E.T. Jaynes' 'Probability Theory: The Logic of Science' (1998) frames Bayesian inference as an extension of deductive logic to scientific reasoning under uncertainty.
IN klayman-ha-positive-test-strategy-bayesian-optimal
Klayman and Ha (1987) demonstrated that the positive test strategy is optimal under Bayesian/information-theoretic criteria when the true rule is narrow and low-probability, becoming suboptimal only when the true rule is broad.
IN laplace-1749-1827-formalized-bayes
Pierre-Simon Laplace (1749–1827) formalized what is now called Bayes' theorem as Principle VI and applied it to celestial mechanics, medical statistics, reliability, and jurisprudence.
IN llm-lost-in-middle-shuffle-prior-removal
In Liu et al. 2023 Appendix C, adding the instruction 'The search results are ordered randomly' and shuffling the k−1 distractor documents removes the decreasing-relevance positional prior but the U-shaped curve remains.
IN lord-1979-fictional-studies-swapped-conclusions
In Lord et al. (1979) Stanford capital-punishment study, participants read two fictional studies with swapped conclusions and rated the study matching their prior view as better-conducted regardless of actual data.
IN map-posterior-mean-symmetric-only
The MAP estimator (mode of the posterior) and the posterior mean coincide only for symmetric posterior distributions.
IN marginal-likelihood-constant-across-hypotheses
The marginal likelihood P(E) is identical across competing hypotheses and therefore cancels when comparing posterior odds.
IN marginal-likelihood-zero-bayes-failure
If the marginal likelihood p(X|α) equals zero, Bayes' rule cannot be applied because the posterior is undefined.
IN mcmc-1980s-computational-revolution
Markov chain Monte Carlo (MCMC) methods were popularized in the 1980s, removing major computational barriers and catalyzing dramatic growth in Bayesian research and applications.
IN mcmc-standard-bayesian-posterior-approximation
MCMC methods (Metropolis–Hastings, Gibbs sampling) are the standard computational tools for approximating Bayesian posteriors in high-dimensional settings where analytical solutions are intractable.
IN objective-vs-subjective-bayesian-currents
Modern Bayesian practice splits into two currents: objective (non-informative) Bayesianism, where analysis depends only on model, data, and prior-assignment rule, and subjective (informative) Bayesianism, where priors encode the analyst's beliefs.
IN planck-2018-bayesian-mcmc-parameters
The Planck 2018 cosmological parameter constraints (H₀, Ω_b, Ω_c, σ₈, n_s, A_s) were derived via Bayesian MCMC inference.
IN post-1920s-frequentist-dominance
After the 1920s, the method of inverse probability (early Bayesian inference) was largely supplanted by frequentist methods in mainstream statistical teaching and practice.
IN r-v-adams-1996-bayes-admitted
In R v Adams (UK, 1996), Bayes' theorem was admitted as expert evidence, and the Court of Appeal upheld the conviction while cautioning against 'inappropriate and unnecessary realms of theory.'
IN swiglu-rope-rmsnorm-not-meta-inventions
SwiGLU (Shazeer 2020), RoPE (Su et al. 2021), and RMSNorm (Zhang & Sennrich 2019) used in Llama are adopted from prior research, not Meta inventions
IN wald-admissibility-theorem
Wald's Admissibility Theorem states that every unique Bayesian procedure is admissible, and every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures.
IN xie-2021-icl-as-bayesian-inference
Xie et al. (2021) formalize in-context learning as the LM performing implicit Bayesian inference over a latent concept θ, where the posterior predictive p(output|prompt) = ∫ p(output|concept, prompt) p(concept|prompt) d(concept) is the core mathematical object.
IN xie-2021-icl-implicit-bayesian-inference
Xie et al. (2021) explain ICL as the model implicitly mapping a set of demonstrations to a latent concept (task) learned during pretraining, framed as implicit Bayesian inference.
IN xu-2024-survey-emnlp-first-dedicated
Xu et al. (EMNLP 2024) was the first dedicated systematic survey of knowledge conflicts in LLMs as of mid-2024; prior surveys addressed it only as a subtopic.
IN xu2024-no-new-computational-experiments
The Xu et al. 2024 survey did not run new computational experiments for its quantitative comparison (§A.1); results were organized as reported in prior literature.