icl-bayesian-ridge-lambda-sigma2-tau2
IN premise — summaries/2026/08/24/akyurek-2023-icl-algorithm-s0-abstract-chunk-1.md
Created 2026-08-24T17:10:49+00:00
With Gaussian prior w~N(0,τ²) and Gaussian label noise ε~N(0,σ²), the minimum-Bayes-risk estimator takes the ridge form with regularization parameter λ = σ²/τ², and ICL behaviorally matches this for all tested noise/prior ratios.
Summary
This establishes that ridge regression is the statistically optimal way to make predictions when you know the noise level and the prior spread of the weights, with the regularization strength set precisely by the ratio of noise variance to prior variance. The key implication is that ICL reproduces this optimal behavior across every tested noise-to-prior ratio, meaning it is not doing something ad hoc but is effectively running the best possible statistical inference given the data it observes.