gradient-descent-conditionally-reliable

IN derived (depth 1)

Created 2026-06-21T13:38:09+00:00 · Reviewed 2026-06-21T15:37:01+00:00

Gradient descent convergence is conditionally reliable — linear convergence is achievable under strong convexity and Lipschitz smoothness, but non-convex landscapes present saddle points that can trap or slow optimization, with momentum methods providing partial mitigation by accumulating velocity across the loss surface.

Justifications

SL — Convergence guarantees are conditional on landscape geometry with partial remedies for non-convexity

Antecedents (all must be IN):

IN gd-linear-convergence-conditions — Gradient descent with fixed step size achieves linear convergence (error decreases by a constant factor each iteration) when the objective is both strongly convex and has Lipschitz continuous gradients
IN gd-saddle-point-limitation — Gradient descent can slow down or get stuck near saddle points, which is a known limitation for non-convex optimization
IN gd-momentum-update-rule — The heavy ball (momentum) update rule is x^(k+1) = x^(k) - η·∇f(x^(k)) + β·(x^(k) - x^(k-1)), adding a fraction of the previous step to dampen oscillation