regularization-trades-bias-for-variance

IN premise — entries/2026/06/21/wiki-BiasE28093variance_tradeoff-chunk-1.md

Created 2026-06-21T09:55:49+00:00

Regularization (e.g., L1/L2, shrinkage, dropout) trades increased bias for decreased variance to improve generalization.

Dependents

These beliefs depend on this one:

• IN dropout-ensemble-interpretation — Dropout is implicitly an ensemble method within a single network — randomly omitting hidden units during training samples from an exponential family of sub-networks, connecting the probabilistic interpretation of neural networks to the variance-reducing properties of ensembles.
• IN overfitting-multi-layered-defense — Defense against overfitting can operate at multiple levels: detection (monitoring train-vs-validation error divergence), prevention (feature selection to reduce dimensionality), and regularization (L1/L2 penalize complexity, dropout prevents co-adaptation) — these mechanisms trade bias for variance in different ways, suggesting that combining approaches may provide more robust protection than relying on any single one.
• IN regularization-as-bias-variance-engineering — Regularization is fundamentally bias-variance engineering — it deliberately increases bias to reduce variance, with L2 producing diffuse weight shrinkage and L1 producing sparsity, both interpretable as Bayesian priors on the weight distribution.