gdpp-eigenvalue-transformation-reduces-condition-number

IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s0-ix-chunk-1.md

Created 2026-08-24T17:11:04+00:00

GD++ transforms the loss Hessian eigenvalues from λᵢ to λᵢ − 2γλᵢ² + γ²λᵢ³ via the input transformation xᵢ ← (I − γXXᵀ)xᵢ, reducing the condition number κ = λ₁/λₙ; optimal γ ≈ 0.099 for N=25 and γ ≈ 0.029 for N=100.

Summary

GD++ reshapes the curvature of the loss landscape by applying a clever transformation to the input, squeezing the gap between the steepest and flattest directions so the surface is more uniformly curved. In practice this means gradient descent can take larger, more confident steps without overshooting, leading to noticeably faster convergence, with the right amount of reshaping scaling down as the problem gets bigger.