Generative Models

52 beliefs (52 IN, 0 OUT)

• IN 1nn-bias-vanishes-infinite-data
  The 1-NN classifier's bias vanishes as the training set size approaches infinity.
• IN amazon-2018-recruiting-bias-example
  Amazon's 2018 recruiting tool penalized women due to male-skewed training data, demonstrating that biased training data produces biased outputs.
• IN bayesian-interpretation-regularization-prior
  The regularization penalty in structural risk minimization corresponds to the negative log prior, making the regularized objective equivalent to posterior probability
• IN benign-overfitting-overparameterization
  Benign overfitting — where a model perfectly fits noisy training data yet still generalizes well — requires overparameterization where the number of unimportant directions in parameter space significantly exceeds the sample size.
• IN bias-variance-decomposition-three-terms
  Expected generalization error under MSE decomposes into exactly three additive terms: squared bias, variance, and irreducible error (noise).
• IN bias-variance-expectation-over-training-sets
  In the bias-variance decomposition, expectations are taken over random draws of the training set D from P(x, y), not over a single fixed dataset.
• IN bias-variance-specific-to-supervised-learning
  The bias-variance tradeoff is specific to supervised learning problems.
• IN bias-variance-tradeoff-prediction-error
  In supervised learning, prediction error decomposes approximately as bias + variance: flexible models have low bias but high variance, rigid models have high bias but low variance
• IN bias-variance-tradeoff-three-components
  The bias-variance tradeoff decomposes total prediction error into three components: bias error, variance error, and irreducible error.
• IN bias-variance-unifies-all-generalization-techniques
  The bias-variance tradeoff serves as a central organizing principle connecting several major ML generalization techniques — regularization directly engineers the tradeoff by trading bias for variance reduction, ensemble methods decompose and target its components independently through bagging and boosting, and overfitting defense operates across multiple layers including detection, prevention, and regularization — all addressing aspects of the same fundamental error decomposition.
• IN classical-generalization-theory-overturned
  Classical generalization theory — the U-shaped bias-variance tradeoff — has been overturned by two empirical phenomena: double descent shows test error decreasing again far past the interpolation threshold, and benign overfitting shows perfect training fit coexisting with good generalization in overparameterized regimes.
• IN classification-bv-decomposition-differs-from-regression
  Under 0-1 loss in classification, the bias-variance decomposition differs from regression because the variance term depends on the true target label.
• IN complex-models-not-necessarily-high-variance
  It is a fallacy to assume that complex models must have high variance; high variance models are complex, but the reverse does not hold.
• IN double-descent-listed-as-core-ai-concept
  Double descent is classified as a core AI concept alongside the classical bias-variance tradeoff in the ML taxonomy.
• IN dropout-ensemble-interpretation
  Dropout is implicitly an ensemble method within a single network — randomly omitting hidden units during training samples from an exponential family of sub-networks, connecting the probabilistic interpretation of neural networks to the variance-reducing properties of ensembles.
• IN dropout-regularization-mechanism
  Dropout regularization works by randomly omitting hidden units during training to prevent co-adaptation on rare dependencies in the training data
• IN ensemble-principle-spans-interpretability-capability-divide
  The ensemble principle appears to span both the classical-deep ML divide and the interpretability-capability tension — it operates as explicit, interpretable variance reduction in random forests and as implicit regularization (dropout) in neural networks. This makes it a notable example of an ML mechanism that functions across both interpretable-classical and opaque-deep contexts, though whether it is unique in this regard is not established by the available evidence.
• IN erm-neg-log-likelihood-equals-mle
  Empirical risk minimization with negative log-likelihood loss is equivalent to maximum likelihood estimation
• IN feature-selection-prevents-overfitting
  Feature selection prevents overfitting by reducing the number of features, preventing models from becoming too specific to training data.
• IN freedmans-paradox-spurious-significance
  Freedman's paradox states that when many explanatory variables are unrelated to the target, some will be falsely found statistically significant, leading to overfitting.
• IN generalization-framework-unified-yet-under-revision
  The bias-variance framework simultaneously unifies all classical generalization techniques (regularization, ensembles, dropout) AND is being fundamentally revised by double descent and benign overfitting — the organizing principle works as engineering guidance but its theoretical foundations are shifting beneath it.
• IN generalization-theory-in-fundamental-revision
  ML generalization theory is undergoing fundamental revision on two independent fronts — the No Free Lunch theorem established that no universal best model exists (killing the quest for a single optimal algorithm), while double descent and benign overfitting overturned the classical U-shaped bias-variance curve (killing the traditional model selection heuristic) — leaving the field without a reliable theoretical guide to practice.
• IN inductive-bias-governs-generalization
  Inductive bias is the set of assumptions a learner uses to predict outputs for unseen inputs; every supervised algorithm has one (e.g., linearity in linear regression, axis-aligned splits in decision trees)
• IN irreducible-error-lower-bound-on-mse
  The irreducible error σ² is a hard lower bound on achievable MSE, independent of model choice, because all three decomposition terms are non-negative.
• IN l2-diffuse-l1-sparse-regularization
  L2 regularization (weight decay) penalizes squared weight magnitude and encourages diffuse weight vectors; L1 regularization produces sparse weights; elastic net combines both L1 and L2
• IN l2-ridge-l1-lasso-regularization
  L2 regularization (sum of squared weights) is ridge regression; L1 regularization (sum of absolute weights) is lasso and induces sparsity; L0 counts non-zero weights
• IN l2-weight-decay-l1-sparsity-distinction
  L2 regularization is equivalent to weight decay; L1 regularization promotes sparsity — these are distinct regularization strategies
• IN larger-datasets-favor-complex-models
  Larger training datasets favor lower-bias (more complex) models because variance decreases with more data, shifting the optimal complexity point.
• IN learning-rate-speed-accuracy-tradeoff
  High learning rates train faster with lower accuracy; low learning rates train slower with potentially greater accuracy.
• IN mcmc-only-asymptotically-unbiased
  MCMC methods are only asymptotically unbiased; controlled bias is sometimes accepted to dramatically reduce variance and overall estimation error.
• IN ml-error-decomposition-universal-principle
  Error decomposition into irreducible and reducible components appears as a recurring structural pattern across ML paradigms — supervised learning decomposes prediction error into bias + variance + irreducible noise, while reinforcement learning independently decomposes suboptimality into asymptotic bias + overfitting. This parallel suggests that separating what-can-be-learned from what-cannot may be a common structural feature of learning problems, though evidence from only two paradigms is insufficient to establish it as a universal invariant of learning itself.
• IN ml-holdout-split-convention
  The holdout evaluation method conventionally splits data into 2/3 training and 1/3 test sets
• IN ml-ridge-regression-regularization
  Ridge regression adds regularization to linear regression to mitigate overfitting by penalizing model complexity
• IN more-training-data-reduces-variance-not-bias
  Increasing training data reduces variance but does not affect bias.
• IN mse-cross-term-vanishes-independence-zero-mean
  In the MSE bias-variance derivation, the cross-term between (f - f̂) and ε vanishes because noise ε is independent of inputs and has zero mean — both assumptions are required.
• IN no-universal-optimal-model
  No universally optimal ML model exists: the No Free Lunch theorem establishes this impossibility in principle, and the bias-variance decomposition reveals the mechanism — any fixed model trades bias against variance depending on the problem.
• IN overfitting-definition-noise-as-signal
  Overfitting occurs when a model extracts noise (residual variation) from training data as if it were genuine structure, resulting in excellent training performance but poor generalization to unseen data.
• IN overfitting-diagnostic-signature
  The diagnostic signature of overfitting is training error decreasing while validation error simultaneously increases.
• IN overfitting-multi-layered-defense
  Defense against overfitting can operate at multiple levels: detection (monitoring train-vs-validation error divergence), prevention (feature selection to reduce dimensionality), and regularization (L1/L2 penalize complexity, dropout prevents co-adaptation) — these mechanisms trade bias for variance in different ways, suggesting that combining approaches may provide more robust protection than relying on any single one.
• IN overfitting-stochastic-and-deterministic-noise
  Overfitting can be caused by stochastic noise (measurement error) or deterministic noise (target function complexity exceeding model capacity)
• IN parameters-geq-observations-memorization
  When a model's parameter count equals or exceeds the number of observations, it can perfectly memorize training data but will generalize poorly.
• IN permutation-importance-correlated-features-failure
  Permutation importance undervalues features in large correlated groups because permuting one correlated feature has little effect when others carry redundant information; the fix is to permute correlated groups together.
• IN regularization-as-bias-variance-engineering
  Regularization is fundamentally bias-variance engineering — it deliberately increases bias to reduce variance, with L2 producing diffuse weight shrinkage and L1 producing sparsity, both interpretable as Bayesian priors on the weight distribution.
• IN regularization-biased-lower-mse-than-ols
  Regularized estimators (LASSO, ridge regression) are biased but can achieve lower MSE than unbiased OLS due to variance reduction.
• IN regularization-dual-purpose-over-under-fitting
  Regularization is a dual-purpose tool: it primarily prevents overfitting by penalizing model complexity, but tuning it too aggressively causes underfitting.
• IN regularization-trades-bias-for-variance
  Regularization (e.g., L1/L2, shrinkage, dropout) trades increased bias for decreased variance to improve generalization.
• IN rl-suboptimality-bias-overfitting-decomposition
  In reinforcement learning, suboptimality decomposes into asymptotic bias (algorithm-dependent) and overfitting (data-dependent), analogous to the classical bias-variance tradeoff.
• IN single-training-example-infinite-weight-solutions
  A single training example may yield infinitely many optimal weight configurations; additional training data or regularization constraints are needed to converge to a unique solution.
• IN stochastic-training-avoids-local-minima
  Stochastic training (adjusting weights after each individual sample) introduces noise that helps avoid getting stuck in local minima.
• IN structural-risk-minimization-regularization
  Structural risk minimization adds a regularization penalty C(g) to empirical risk: J(g) = R_emp(g) + λC(g), where λ=0 gives pure ERM and large λ gives high bias/low variance
• IN total-prediction-error-decomposition
  Total prediction error decomposes as: Bias² + Variance + Irreducible noise (Bayes error).
• IN underfitting-high-bias-low-variance
  Underfitting produces high bias and low variance, while overfitting produces low bias and high variance.