Gradient Descent vs Gradient Descent with Momentum

Watch how momentum helps escape local minima and speeds up convergence

Learning rate α 0.05

Momentum β 0.70

Loss function

GD Gradient Descent

GDM GD + Momentum

GD Step

GD Loss

—

GD θ

—

GDM Step

GDM Loss

—

GDM θ

—

Loss curve

GD ball

GDM ball

Global minimum

Gradient (slope)

GDM velocity arrow

Vanilla GD
θ ← θ − α · ∇L(θ)

GD + Momentum
v ← β · v − α · ∇L(θ)
θ ← θ + v

GD traps immediately in a local minimum (loss=2.05). GDM builds momentum to escape and reach the global minimum (loss=-1.36). Hit Start!