gradient-descent-effective-global-optimizer

OUT derived (depth 1)

Created 2026-06-21T09:59:01+00:00

Gradient descent with backpropagation effectively finds global optima for neural network training in practice, despite lacking global guarantees.

Justifications

SL — GD is effective when local minima and saddle points are not pathological

Antecedents (all must be IN):

IN gradient-descent-first-order-negative-gradient — Gradient descent is a first-order iterative optimization algorithm that moves in the negative gradient direction (steepest descent) to find local minima of differentiable functions
IN backprop-computes-gradient-not-learning — Backpropagation only computes the gradient; a separate optimization algorithm (e.g., SGD, Adam) performs the actual parameter updates.
IN svm-objective-is-convex — The SVM optimization objective is convex, guaranteeing that any local minimum is the global minimum.

Unless (any of these IN defeats this justification):

IN gradient-descent-finds-local-not-global-minima — Gradient descent with backpropagation finds local minima only, with no guarantee of reaching a global minimum; however, LeCun et al. argued this is often acceptable in practice for neural networks.
IN gd-saddle-point-limitation — Gradient descent can slow down or get stuck near saddle points, which is a known limitation for non-convex optimization