ML Crisis & Reliability

140 beliefs (139 IN, 0 OUT)

• IN bigan-three-player-encoder
  BiGAN (Bidirectional GAN) is a 3-player, 2-team game (generator + encoder vs. discriminator) where the discriminator operates on (image, latent) pairs, enabling learned inference
• IN biggan-scaling-improves-sample-quality
  BigGAN (Brock et al., 2018) demonstrated that scaling up GANs with large batch sizes and more parameters dramatically improves sample quality
• IN centered-kerf-converges-faster-than-uniform
  Centered KeRF converges at rate O(n^{-1/(3 + d log 2)} · (log n)^2) while Uniform KeRF converges at O(n^{-2/(6 + 3d log 2)} · (log n)^2), making Centered KeRF strictly faster, under Lipschitz target function, uniform X on [0,1]^d, and Gaussian noise.
• IN classical-ml-vs-deep-learning-complementary-strengths
  SVMs and random forests illustrate complementary strengths within classical ML — SVMs achieve mathematical elegance through convex optimization, dual formulation, and the kernel trick, while random forests achieve robust generalization through variance reduction via decorrelated trees, overfitting immunity, and built-in OOB error estimation. Deep learning's rise was driven in significant part by compute scaling (a 300,000x increase from AlexNet to AlphaZero), though the relative importance of compute versus algorithmic innovation remains unestablished.
• IN cnn-same-padding-formula
  Same padding (preserving spatial dimensions when stride=1) requires P = (K - 1) / 2 zero-padding
• IN compression-prediction-equivalence
  Optimal prediction and optimal data compression are formally equivalent: an optimal predictor of sequence probabilities can be used for optimal compression via arithmetic coding, and vice versa
• IN cyclegan-four-player-unpaired-translation
  CycleGAN is a 4-player, 2-team game that performs unpaired domain translation using two generators, two discriminators, and a cycle-consistency loss enforcing round-trip reconstruction G_Y(G_X(x)) ≈ x
• IN cyclegan-unpaired-pix2pix-paired
  CycleGAN performs unpaired image-to-image translation while pix2pix performs paired image-to-image translation; both originated from the Berkeley group (Zhu, Isola, Efros et al.)
• IN dcgan-bridge-fc-to-conv-gans
  DCGAN (Radford, Metz, Chintala 2016) established convolutional architecture best practices for GANs, bridging fully-connected to convolutional GAN architectures
• IN dcgan-fully-convolutional
  DCGAN uses only convolution-deconvolution layers (fully convolutional) for both generator and discriminator with no fully connected layers
• IN diffusion-models-eclipsed-gans-2022
  Diffusion models (DALL-E 2, Stable Diffusion) eclipsed GANs for generative modeling from approximately 2022 onward
• IN diffusion-models-supplanted-gans-text-to-image
  Latent diffusion models (Stable Diffusion, DALL-E) have largely supplanted GANs as the dominant approach for text-to-image generation tasks
• IN dominant-paradigms-empirically-fragile-and-transient
  The most successful ML paradigms are simultaneously dominant and fragile — pretrain-then-finetune is standard practice yet empirically hurtful in some transfer settings, GANs dominated generative modeling for years yet were displaced by diffusion — suggesting that current best practices are locally optimal recipes liable to succession rather than fundamental principles.
• ? exam-fail-q33
  
• IN fid-introduced-in-ttur-paper
  Fréchet Inception Distance (FID) was introduced in the Two Time-Scale Update Rule (TTUR) paper by Heusel et al. (2017), not as a standalone metrics paper
• IN gan-architectural-asymmetry-mirrors-game-theory
  GAN architectural asymmetry (deconvolutional generator vs convolutional discriminator in image tasks) parallels its game-theoretic asymmetry — the discriminator has a closed-form optimal solution while the generator does not, and these two asymmetries coexist in the minimax game structure, though the antecedents do not establish that the architectural choices were designed to instantiate the mathematical asymmetry.
• IN gan-asymptotically-consistent
  GANs are asymptotically consistent estimators of the data distribution due to the universal approximation theorem for neural networks
• IN gan-complete-theoretical-characterization
  GAN theory provides a complete analytical characterization of optimal behavior: the optimal discriminator has a closed-form solution (Radon-Nikodym derivative), the objective implicitly minimizes Jensen-Shannon divergence, and at the unique equilibrium the generator exactly recovers the data distribution.
• IN gan-critical-deployments-compound-accountability-crisis
  GAN deployment in critical domains — accelerating CERN particle physics simulations and generating synthetic medical images to circumvent privacy barriers — compounds ML's accountability crisis by deploying implicit generative models (no explicit likelihood, no interpretable internals) in precisely the domains where accountability matters most.
• IN gan-cross-domain-practical-impact
  GANs have demonstrated practical impact across radically different domains — accelerating particle physics simulations at CERN, generating privacy-preserving synthetic medical images, and producing auction-worthy art ($432,500 for Edmond de Belamy) — making them among the most broadly applied generative models.
• IN gan-edmond-de-belamy-432500
  The GAN-generated painting 'Edmond de Belamy', trained on 15,000 WikiArt portraits, sold for US$432,500 at auction in 2018
• IN gan-equilibria-coincide-original-only
  Sequential (minimax/maximin) and Nash equilibria all coincide for the original GAN game, but this equivalence is not guaranteed for general GAN variants
• IN gan-equilibrium-generator-matches-data
  At the unique GAN equilibrium, the generator matches the data distribution exactly (μ̂_G = μ_ref), the discriminator outputs 1/2 everywhere, and the objective value is −2 ln 2
• IN gan-exemplifies-pragmatism-crisis-at-model-level
  GANs recapitulate at the individual model level the field-wide pattern where pragmatic shortcuts drive both capability and crisis — their implicit generative approach (pragmatically avoiding intractable likelihood computation) simultaneously enabled unique capabilities (single-pass generation, cross-domain applications) and created fundamental training instability, making GANs the clearest single-model exemplar of the pragmatism-crisis dynamic.
• IN gan-game-theoretic-foundations-fragile-beyond-original
  GAN game-theoretic foundations are fragile beyond the original formulation — equilibrium equivalence (minimax, maximin, Nash) holds only for the original game and not its variants, Nash equilibria are not guaranteed to exist in general (Farnia & Ozdaglar 2020), and the two dominant failure modes (mode collapse and vanishing gradients) represent opposed destabilizing forces that the equilibrium theory does not resolve.
• IN gan-generator-deconv-discriminator-conv
  For image tasks, the GAN generator is typically a deconvolutional neural network and the discriminator is typically a convolutional neural network
• IN gan-implicit-generative-model
  GANs are implicit generative models that do not explicitly model the likelihood function and cannot find the latent variable for a given sample, unlike flow-based models or VAEs
• IN gan-implicit-nature-explains-training-difficulty
  GANs' implicit generative nature (no explicit likelihood function) is plausibly connected to their need for multiple complementary training interventions — without a tractable objective to optimize directly, training stability relies on several distinct design choices (non-saturating loss, TTUR, deterministic discriminators) each addressing a different failure mode, which may partly substitute for the more direct optimization signal that explicit-likelihood models enjoy.
• IN gan-introduced-goodfellow-2014
  Generative Adversarial Networks were introduced by Ian Goodfellow et al. in 2014
• IN gan-minimizes-jensen-shannon-divergence
  The original GAN implicitly minimizes the Jensen-Shannon divergence (not KL divergence) between the generated and real data distributions
• IN gan-mode-collapse-vs-vanishing-gradient
  Mode collapse (generator produces limited modes) and vanishing gradient (generator cannot learn) are opposite GAN failure modes caused by the discriminator being too weak vs. too strong, respectively
• IN gan-nash-impossibility-mathematically-necessitates-model-level-crisis
  Farnia & Ozdaglar's proof that GANs lack guaranteed Nash equilibria provides formal grounding for the GAN-level pragmatism-crisis dynamic — the training instability that exemplifies the field-wide crisis pattern at the model level is not merely an empirical tendency but has a provable theoretical basis in adversarial game structure, strengthening the case that the pragmatism-crisis pattern at the model level is structurally rooted rather than incidental.
• IN gan-non-saturating-loss-original-paper
  The non-saturating generator loss (L_G = E[ln D(x)]) was recommended in the original GAN paper for faster convergence, addressing vanishing gradients when the generator is poor
• IN gan-optimal-discriminator-formula
  The optimal discriminator for a fixed generator is D*(x) = μ_ref(x) / (μ_ref(x) + μ_G(x)), which is the Radon-Nikodym derivative dμ_ref/d(μ_ref + μ_G)
• IN gan-original-paper-goodfellow-2014-nips
  The original GAN paper by Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, and Bengio was published at NIPS 2014
• IN gan-particle-physics-simulation-cern
  GANs accelerate particle physics simulations at CERN by approximating expensive computational bottlenecks for high-energy jet formation, calorimeter showers, and turbulent flow reconstruction
• IN gan-single-pass-generation
  GANs generate a complete sample in one forward pass, unlike autoregressive models (WaveNet, PixelRNN) which generate sequentially token-by-token
• IN gan-synthetic-medical-imaging-privacy
  GANs generate synthetic medical images (MRI, PET) to overcome patient privacy barriers that limit access to real medical imaging data
• IN gan-training-stability-requires-multiple-interventions
  GAN training stability benefits from several complementary design choices — non-saturating loss to address vanishing gradients when the generator is poor, two-timescale update rule for provable convergence to a stationary local Nash equilibrium (though not mode collapse prevention), and deterministic discriminators for optimality of the discriminator — each addressing a distinct failure mode, but none individually sufficient for overall stability.
• IN gan-ttur-convergence
  The Two Time-Scale Update Rule (TTUR) sets the generator's learning rate lower than the discriminator's and provably converges to a stationary local Nash equilibrium, but does not prevent mode collapse
• IN gan-two-network-minimax-game
  A GAN consists of exactly two neural networks (generator and discriminator) trained as a minimax zero-sum game where the generator minimizes and the discriminator maximizes the same objective function
• IN gan-vs-vae-training-paradigm
  GANs use adversarial training (minimax game) while VAEs use variational inference with an encoder-decoder structure; both are generative models but with fundamentally different training paradigms
• IN gans-2014-eclipsed-by-diffusion
  GANs were introduced by Goodfellow in 2014 (concept traces to Schmidhuber's 'artificial curiosity' 1991), reached peak quality with StyleGAN (2018), and were later eclipsed by diffusion models for image generation
• IN gans-adversarial-generator-discriminator
  Generative Adversarial Networks (GANs) pit two networks against each other: one generates content and the other discriminates real from fake, driving each other to improve.
• IN gans-no-guaranteed-nash-equilibrium
  Farnia & Ozdaglar (ICML 2020) proved that GANs do not always have Nash equilibria, establishing a theoretical limitation of adversarial training
• IN generative-model-families-autoregressive-diffusion-gan-vae
  Four main generative model families: autoregressive (GPT-style), diffusion (Stable Diffusion), adversarial (GAN), and variational (VAE).
• IN generative-modeling-paradigm-succession
  Generative modeling has seen a significant shift: GANs, introduced in 2014, were largely supplanted by diffusion models (such as DALL-E 2 and Stable Diffusion) from approximately 2022 onward, particularly for text-to-image generation, suggesting that even widely adopted adversarial training frameworks can be overtaken by alternative approaches.
• IN hyperparameters-set-before-training
  Hyperparameters (learning rate, batch size, number of layers/nodes) are set before training begins and tuned between training runs, not during them.
• IN infogan-three-player-game
  InfoGAN is a 3-player, 2-team game (generator + Q network vs. discriminator) that maximizes a variational lower bound on mutual information I(c, G(z,c)) to discover latent labels without supervision
• IN kerf-connection-function-is-kernel
  The KeRF connection function K_{M,n}(x, z) equals the proportion of trees in which x and z fall in the same cell, and this serves as the kernel for a Nadaraya-Watson-style kernel regression estimator.
• IN kernel-methods-bridge-bayesian-and-frequentist-ml
  Kernel methods occupy a unique position bridging Bayesian and frequentist approaches to ML — SVMs provide a frequentist max-margin framework using kernels for nonlinear classification, while Gaussian processes use the same kernel functions for Bayesian probabilistic modeling and hyperparameter optimization, with Bayesian SVMs explicitly connecting the two paradigms by reinterpreting SVM optimization as Bayesian inference.
• IN kv-caching-autoregressive-inference
  KV caching reuses previously computed key and value vectors during autoregressive inference since they don't change; prefilling is the initial forward pass that populates the cache.
• IN liblinear-large-scale-linear-svm
  LIBLINEAR (Fan et al. 2008) is optimized for large-scale linear classification and is the backend for scikit-learn's LinearSVC.
• IN libsvm-most-widely-used-svm-library
  LIBSVM is the most widely-used SVM library, supporting classification, regression, and distribution estimation.
• IN logistic-sigmoid-derivative-formula
  The logistic sigmoid function φ(z) = 1/(1 + e^(-z)) has the convenient derivative φ'(z) = φ(z)(1 - φ(z)).
• IN ml-auc-roc-interpretation
  The ROC curve plots True Positive Rate vs False Positive Rate across thresholds; higher AUC (Area Under the ROC Curve) indicates a better model, providing threshold-independent assessment
• IN ml-gans-goodfellow-2014
  Generative Adversarial Networks (GANs) were introduced by Goodfellow et al. in 2014 at NeurIPS
• IN ml-gaussian-processes-bayesian-optimization
  Gaussian processes use a covariance function (kernel) to model point relationships and are commonly used as surrogate models in Bayesian optimization for hyperparameter tuning
• IN ml-paradigm-impermanence-doubly-determined
  No current ML paradigm can persist: theoretical completeness is demonstrated insufficient for survival (GANs' closed-form analysis didn't prevent displacement by diffusion models), and empirical dominance is independently fragile (pretrain-finetune is standard yet empirically hurtful in some settings) — paradigm impermanence is overdetermined by both theoretical and empirical evidence.
• IN ml-platt-scaling-svm-probabilities
  Platt scaling enables probabilistic classification with SVMs, which are otherwise non-probabilistic classifiers
• IN ml-sensitivity-tpr-specificity-tnr
  Sensitivity (True Positive Rate) is the proportion of actual positives correctly identified; Specificity (True Negative Rate) is the proportion of actual negatives correctly identified
• IN ml-sparse-dictionary-learning-np-hard
  Sparse dictionary learning (representing examples as sparse linear combinations of basis functions) is strongly NP-hard; k-SVD is the standard heuristic solver
• IN one-in-ten-rule-overfitting
  The one-in-ten rule is a guideline recommending approximately 10 observations per independent variable in logistic or Cox regression to avoid overfitting.
• IN paradigm-survival-determined-by-scalability-not-theory
  Mathematical completeness and theoretical elegance are neither necessary nor sufficient for paradigm survival in ML — GANs had the most complete analytical characterization yet were eclipsed by diffusion models, SVMs had convex guarantees yet were outscaled by neural networks, while theoretically less grounded approaches that scaled with hardware thrived.
• IN parameter-count-poor-complexity-measure
  Number of parameters is a poor measure of model complexity — a model f(x) = a·sin(bx) has only two parameters but can interpolate arbitrarily many points via high-frequency oscillation.
• IN reparameterization-independently-discovered-extends-convergent-discovery-pattern
  The independent discovery of the reparameterization trick for both GANs and VAEs (2013-2014) extends the convergent discovery pattern already established for gradient computation, weight sharing, and gradient flow — a fourth foundational mechanism independently found across disconnected research groups, further validating that ML's core techniques are mathematical necessities discoverable from multiple starting points.
• IN reparameterization-trick-independent-discovery
  The reparameterization trick was independently discovered for GANs (Goodfellow) and VAEs (Kingma & Welling, Rezende et al.) in 2013-2014
• IN rf-kerf-approximation-bound-cell-occupancy
  The approximation bound between standard RF and KeRF predictions is |m_{M,n}(x) - m̃_{M,n}(x)| ≤ ((b_n - a_n)/a_n) · m̃_{M,n}(x), where a_n and b_n bound cell occupancy counts, so tighter bounds occur when all cells have similar counts.
• IN rmsnorm-llama-simplifies-layernorm
  RMSNorm (used by Llama) simplifies LayerNorm by removing the mean-centering step.
• IN rumelhart-weight-subscript-convention
  In the Rumelhart, Hinton & Williams (1986) convention, w_ij denotes the weight from unit i to unit j.
• IN schmidhuber-adversarial-networks-1991
  Schmidhuber's 1991 'artificial curiosity' system using two neural networks in a zero-sum game is a historical precursor to GANs
• IN scientific-applications-validate-capability-without-reliability
  ML's scientific applications (AlphaFold for protein structure prediction, GraphCast for weather forecasting, GANs for particle physics simulation at CERN) demonstrate that ML can achieve results matching or exceeding traditional computational methods in specific scientific domains, suggesting broad capability across diverse physical problem types.
• IN scornet-formalized-kerf-2015
  Scornet (2015) formalized the KeRF definitions and proved consistency results, building on earlier observations by Breiman and the RF-kernel connection shown by Lin & Jeon (2006).
• IN softmax-converts-outputs-to-posterior-probabilities
  The softmax function (y_i = e^(x_i) / sum_j(e^(x_j))) converts raw output scores into a probability distribution over classes, interpretable as posterior probabilities for classification.
• IN softmax-vs-sigmoid-vs-euclidean-loss
  Softmax loss is used for mutually exclusive single-class classification, sigmoid cross-entropy for independent multi-label classification, and Euclidean loss for regression to real-valued labels
• IN stochastic-pooling-hyperparameter-free
  Stochastic pooling replaces deterministic pooling with random selection according to a multinomial distribution over activations; it is hyperparameter-free and equivalent to max pooling over locally deformed copies
• IN structural-risk-minimization-connects-svm-to-bias-variance
  Structural risk minimization formally connects SVMs to the bias-variance tradeoff — SRM's regularization penalty λC(g) is precisely the mechanism that engineers the bias-variance balance, and SVMs instantiate this as ERM with hinge loss and L2 regularization, making SVMs the paradigmatic case where learning theory (PAC/SRM) and statistical decomposition (bias-variance) meet in a single coherent framework.
• IN stylegan-progression-three-versions
  StyleGAN (2019) → StyleGAN2 (2020, 'Analyzing and Improving Image Quality') → StyleGAN3 (2021, 'Alias-Free') were all by Karras et al. at NVIDIA
• IN svm-bayesian-auto-hyperparameter-uncertainty
  Bayesian SVM (Polson & Scott, 2011) interprets the SVM as a graphical model, enabling automatic hyperparameter tuning and predictive uncertainty quantification, unlike standard SVMs that require cross-validation.
• IN svm-bias-recovery-from-support-vector
  The SVM bias term b is recovered from any support vector on the margin boundary using b = w^T x_i - y_i, exploiting that y_i^(-1) = y_i for labels in {-1, +1}.
• IN svm-c-parameter-margin-error-tradeoff
  The regularization parameter C controls the trade-off between maximizing the margin and minimizing classification errors: large C approximates hard-margin behavior (narrow margin, fewer errors); small C allows more violations (wider margin, more errors).
• IN svm-codified-practical-methodology
  SVMs have an unusually prescriptive practical methodology for ML: standardize features first, default to RBF kernel, then grid-search C and gamma with cross-validation.
• IN svm-common-kernels-linear-poly-rbf-sigmoid
  Common SVM kernel functions are: Linear k(x,y)=x·y, Polynomial k(x,y)=(x·y+c)^d, Gaussian RBF k(x,y)=exp(-gamma||x-y||²), and Sigmoid k(x,y)=tanh(alpha*x·y+c).
• IN svm-completeness-self-limiting-at-scale
  SVMs illustrate a tension within mathematically complete ML frameworks — the same three-decade development that produced convex optimization with global guarantees, kernel theory for nonlinear classification, and sparse support-vector representations also produced a methodology that requires architectural decomposition for multiclass problems (OvA/OvO/Crammer-Singer) and scale-dependent solver selection (SMO vs Pegasos vs LIBLINEAR), creating a combinatorial burden that grows with problem complexity.
• IN svm-complexity-compounds-with-scale
  SVM complexity compounds as problems scale — multiclass classification requires architectural decomposition (OvA/OvO/Crammer-Singer) on top of already scale-dependent solver selection (SMO vs Pegasos vs LIBLINEAR), creating a combinatorial methodology burden that contrasts with neural network approaches which handle multiclass classification more naturally.
• IN svm-crammer-singer-single-optimization
  The Crammer-Singer method casts multiclass SVM as a single unified optimization problem rather than decomposing into multiple binary sub-problems.
• IN svm-decision-boundary-depends-only-on-support-vectors
  The SVM decision boundary depends only on the support vectors (points closest to the boundary), not on all training points, making SVMs memory-efficient at prediction time.
• IN svm-dual-enables-kernel-trick
  The dual formulation of the SVM involves only dot products between data points (x_i^T x_j), which is what enables the kernel trick by replacing these with k(x_i, x_j).
• IN svm-equivalent-erm-hinge-loss-tikhonov
  The soft-margin SVM is equivalent to empirical risk minimization (ERM) with hinge loss and Tikhonov (L2) regularization.
• IN svm-feature-scaling-critical
  Feature scaling (standardization) is critical for SVM performance and should always be applied before training.
• IN svm-framework-extends-beyond-binary-classification
  The SVM framework extends well beyond its original binary classification setting — SVR adapts the max-margin principle to regression via epsilon-insensitive loss, transductive SVMs bridge to semi-supervised learning by jointly optimizing over labeled and unlabeled data, and Bayesian SVMs reinterpret the framework probabilistically for automatic hyperparameter tuning with uncertainty quantification.
• IN svm-gan-methodology-maturity-contrast
  SVMs and GANs illustrate contrasting degrees of methodology codification in ML — SVMs have an unusually prescriptive practical recipe (standardize, default to RBF, grid-search C and gamma), while GAN training stability requires multiple complementary but individually insufficient interventions (non-saturating loss, two-timescale updates, deterministic discriminators) addressing distinct failure modes, suggesting that well-understood convex optimization enables more codified practice than implicit generative modeling with competing failure modes.
• IN svm-grounded-in-learning-theory-but-theory-itself-overturned
  SVMs are uniquely grounded in statistical learning theory (PAC framework, ERM with hinge loss recovering Bayes-optimal classifier, SRM regularization) — yet the classical generalization theory that grounds them has itself been overturned by double descent and benign overfitting, creating a paradox where SVMs' strongest theoretical virtue (principled learning-theoretic foundation) rests on a framework now known to be incomplete.
• IN svm-hard-margin-requires-linear-separability
  Hard-margin SVM requires linearly separable data and permits no misclassifications; it minimizes (1/2)||w||² subject to y_i(w^T x_i - b) >= 1 for all i.
• IN svm-hinge-loss-target-is-bayes-optimal-classifier
  The hinge loss target function recovers exactly the Bayes-optimal classifier f*(x) (outputs ±1 based on whether p_x >= 1/2), unlike square loss (conditional expectation) or log-loss (logit) which estimate the full conditional distribution.
• IN svm-hyperparameter-selection-grid-search-cv
  SVM hyperparameters (kernel choice, gamma, lambda/C) are typically selected via grid search with exponentially growing sequences (e.g., 2^-5 to 2^15) evaluated by cross-validation, or alternatively by Bayesian optimization.
• IN svm-invented-vapnik-chervonenkis-1963
  The linear SVM was proposed by Vapnik in 1963; the kernelized version was introduced by Boser, Guyon, and Vapnik in 1992; the soft margin formulation was published by Cortes and Vapnik in 1995.
• IN svm-kernel-methods-superset
  SVMs are kernel-based learning methods; all SVMs use kernels (even if just the linear kernel), but not all kernel methods are SVMs.
• IN svm-kernel-trick-avoids-explicit-high-dim-mapping
  The kernel trick enables nonlinear classification by computing dot products in a high-dimensional feature space via a kernel function k(x,y) without explicitly computing the high-dimensional mapping phi(x).
• IN svm-kernel-trick-boser-guyon-vapnik-1992
  Boser, Guyon, and Vapnik (1992) proposed the kernel trick for optimal margin classifiers, predating the 1995 soft-margin paper.
• IN svm-liblinear-linear-time-iterations
  LIBLINEAR's coordinate descent for linear SVMs has iterations that are linear in data read time with Q-linear convergence, making it extremely fast.
• IN svm-limitations-labeled-data-binary-uncalibrated
  Key SVM limitations: requires fully labeled data, produces uncalibrated probabilities, is natively binary-only (multiclass requires reduction strategies), and has difficult-to-interpret parameters.
• IN svm-margin-formula-2-over-norm-w
  The SVM geometric margin equals 2/||w||, derived from the separating hyperplane formulation.
• IN svm-margin-width-formula
  The geometric width of the SVM margin is 2/||w||, so maximizing the margin is equivalent to minimizing ||w||.
• IN svm-mathematical-coherence-sparsity-and-equivalence
  SVMs exhibit notable mathematical coherence — the model is fully determined by a sparse subset of training points (support vectors), and the soft-margin optimization admits three equivalent formulations (slack variables with margin constraints, hinge loss ERM with Tikhonov regularization, and the C-parameter tradeoff), providing both computational sparsity and multiple theoretical perspectives on the same underlying optimization.
• IN svm-mathematical-coherence-three-dimensional
  SVMs exhibit mathematical coherence across three independent dimensions — sparsity and equivalence in the model structure (support vector determination, three equivalent soft-margin formulations), elegance in the optimization landscape (convex objective, kernel trick, dual formulation), and systematic extensibility beyond binary classification (SVR, transductive, Bayesian) — making SVMs uniquely principled across formulation, optimization, and scope.
• IN svm-mathematical-elegance-convex-kernel-dual
  SVMs achieve mathematical elegance through three interlocking properties: the convex objective guarantees global optimality, the dual formulation exposes dot products, and the kernel trick maps those dot products into high-dimensional spaces without explicit computation.
• IN svm-max-margin-hyperplane-minimizes-generalization-error
  SVMs find the hyperplane that maximizes the margin between classes; larger margins correlate with lower generalization error and reduced overfitting.
• IN svm-methodology-codification-mirrors-optimization-topology
  The contrast between SVMs' prescriptive methodology (standardize, RBF default, grid-search) and GANs' ad-hoc training recipes appears to be significantly influenced by their optimization topologies — convexity enables more codifiable methodology while non-convex minimax games resist systematization, suggesting that optimization landscape is an important factor in methodology maturity, though the evidence from these two cases alone is insufficient to establish it as the sole or root determinant.
• IN svm-model-fully-determined-by-support-vectors
  The entire SVM model — decision boundary, weight vector, and bias — is fully determined by the support vectors alone; all other training points are irrelevant to the learned classifier.
• IN svm-multiclass-one-vs-one-outperforms-one-vs-all
  For multiclass SVM, one-vs-one generally outperforms one-vs-all (Hsu & Lin 2002; Duan & Keerthi 2005).
• IN svm-multiclass-ova-vs-ovo
  Multiclass SVM uses one-versus-all (OVA: K classifiers, winner-takes-all on calibrated scores) or one-versus-one (OVO: K(K-1)/2 classifiers, max-wins voting).
• IN svm-multiclass-requires-architectural-extension
  SVMs' binary-native design requires substantial architectural extension for multiclass problems — decomposition into one-vs-all or one-vs-one subproblems, Platt scaling for probability calibration in OVA, or the unified Crammer-Singer formulation — with OVO generally outperforming OVA despite training more classifiers.
• IN svm-objective-is-convex
  The SVM optimization objective is convex, guaranteeing that any local minimum is the global minimum.
• IN svm-pegasos-sgd-solver
  Pegasos (Shalev-Shwartz et al. 2007) is a stochastic gradient descent SVM solver operating in the primal with O(1/t) convergence.
• IN svm-platt-scaling-probability-calibration
  Platt scaling is used to calibrate SVM outputs into probabilities, which is important for one-versus-all multiclass SVM.
• IN svm-practical-guide-rbf-default-grid-search
  The practical SVM guide (Hsu et al. 2003) recommends: scale features, use RBF kernel as default, and grid-search C and gamma via cross-validation.
• IN svm-preprocessing-standardization-critical
  Data preprocessing via standardization (Z-score: subtract mean, divide by standard deviation per feature) is critical for SVM accuracy and highly recommended before training.
• IN svm-rare-complete-ml-framework
  SVMs represent a notably coherent framework in ML — three decades of incremental development produced convex optimization with global optimality guarantees, kernel-enabled nonlinearity, and a model fully determined by a sparse subset of training points — a degree of mathematical closure that few other learning paradigms achieve.
• IN svm-sigmoid-kernel-valid-only-some-params
  The sigmoid kernel k(x,y) = tanh(kappa*x·y + c) is only a valid Mercer kernel for certain parameter values (kappa > 0, c < 0), not universally.
• IN svm-sklearn-svc-api
  In scikit-learn, SVMs are used via sklearn.svm.SVC with parameters kernel, C, and gamma; e.g., SVC(kernel='rbf', C=1.0, gamma='scale').
• IN svm-smo-algorithm-2d-subproblems
  The SMO (Sequential Minimal Optimization) algorithm decomposes the SVM quadratic programming problem into 2-dimensional sub-problems solved analytically, requiring no numerical optimizer or large matrix storage.
• IN svm-smo-platt-1998
  Sequential Minimal Optimization (SMO), introduced by Platt in 1998, decomposes the SVM quadratic programming problem into minimal 2-variable subproblems.
• IN svm-soft-margin-cortes-vapnik-1995
  Cortes and Vapnik (1995) introduced soft-margin support-vector networks in the foundational SVM paper.
• IN svm-soft-margin-hinge-loss-slack-variables
  Soft-margin SVM handles non-separable data by introducing slack variables zeta_i >= 0 and hinge loss max(0, 1 - y_i(w^T x_i - b)), minimizing ||w||² + C * sum(zeta_i).
• IN svm-soft-margin-three-equivalent-views
  The soft-margin SVM admits three equivalent mathematical formulations — slack variables with margin constraints, empirical risk minimization with hinge loss and Tikhonov regularization, and the C-parameter tradeoff between margin width and classification errors — all describing the same optimization from different theoretical perspectives.
• IN svm-solver-ecosystem-scale-dependent
  SVMs have a rich ecosystem of specialized solvers whose optimal choice depends on problem geometry — SMO decomposes into 2D subproblems for general QP, Pegasos uses stochastic gradient descent in the primal for large-scale problems, LIBLINEAR achieves linear-time iterations for high-dimensional sparse data, and the crossover point between sub-gradient and coordinate descent depends on whether n or d dominates.
• IN svm-subgradient-large-n-coorddescent-high-dim
  Sub-gradient descent is preferred for SVMs with large n (many training examples); coordinate descent is preferred for high-dimensional feature spaces.
• IN svm-svr-epsilon-insensitive-loss
  Support Vector Regression (SVR), introduced by Drucker et al. (1997), uses an epsilon-insensitive loss function.
• IN svm-theory-practice-unity-unmatched
  SVMs exhibit an unusual degree of theory-practice coherence in ML — their three-dimensional mathematical coherence (sparsity, equivalence, elegance across formulation, optimization, and scope) corresponds to an unusually prescriptive practical methodology (standardize features, default to RBF, grid-search C and gamma), suggesting one of the more complete theory-to-practice pipelines in the field.
• IN svm-three-decade-development-arc
  The SVM framework evolved incrementally over three decades: linear classification (Vapnik 1963), kernelized nonlinearity (Boser-Guyon-Vapnik 1992), and soft-margin error tolerance (Cortes-Vapnik 1995).
• IN svm-transductive-joachims-1999
  Transductive SVMs (Joachims 1999) bridge supervised and semi-supervised learning by using unlabeled test data during training.
• IN svm-transductive-semi-supervised-learning
  Transductive SVMs (Vapnik, 1998) extend SVMs to semi-supervised learning by jointly optimizing the separating hyperplane and the labels of unlabeled test data, where test labels y* are decision variables in the optimization.
• IN svm-weight-vector-linear-combination-support-vectors
  The SVM weight vector w = sum(c_i * y_i * x_i) is a linear combination of support vectors only; most dual coefficients c_i are zero.
• IN td-lambda-interpolates-mc-and-td
  TD(λ) with λ=0 relies entirely on Bellman equations (pure TD), while λ=1 is equivalent to Monte Carlo with no Bellman reliance; λ provides continuous interpolation between the two
• IN theoretical-completeness-no-guarantee-of-paradigm-durability
  Theoretical completeness does not guarantee paradigm durability — GANs had a notably complete analytical characterization (closed-form optimal discriminator, JSD minimization proof, unique equilibrium) yet were largely supplanted by diffusion models from approximately 2022 onward, suggesting that factors beyond theoretical elegance — possibly including training reliability — may play a significant role in determining which paradigms persist.
• IN unsupervised-generative-evolution-validates-paradigm-succession
  The evolution of unsupervised neural network architectures (Hopfield → Boltzmann → RBM → VAE) and the succession of generative model families (GANs → diffusion models) illustrate a recurring pattern in which dominant ML paradigms can be displaced by successors that may draw on predecessor ideas but achieve prominence through different approaches. This is consistent with the observation that even widely adopted paradigms appear to be locally optimal practices liable to succession rather than permanent foundations.
• IN vc-dimension-half-information-capacity
  The VC dimension for arbitrary inputs equals half the information capacity of a perceptron.
• IN wgan-earth-mover-distance-training-stability
  Wasserstein GAN addresses GAN training instability by using the Earth Mover's (Wasserstein) distance as the loss function; theoretical motivation by Arjovsky & Bottou (2017)
• IN wgan-uses-wasserstein-distance
  Wasserstein GAN (WGAN) replaces the JS divergence with the Wasserstein (Earth Mover's) distance, constrains the discriminator's Lipschitz norm, and the discriminator outputs unbounded real values rather than probabilities