Optimizer Convergence Race — GD · GDM · AdaGrad · RMSProp · Adam · AdamW

Three scenarios, each showing a different optimizer's strength. Click legend items to show/hide.

Loss landscape + ball positions
Convergence curve — Loss vs Step
OptimizerStepLossθStatus
GD & GDM θt = θt−1 − η·gt
GDM: mt=β1mt−1+gt  θt=θt−1−ηmt Fixed lr. GDM adds momentum to smooth noise.
AdaGrad & RMSProp Ada: Gt=Gt−1+gt²
RMS: Gt=ρGt−1+(1−ρ)gt²
θt=θt−1−η√Gt+ε·gt Adaptive lr. AdaGrad stalls on long runs.
Adam & AdamW m̂t=mt/(1−β1t)  v̂t=vt/(1−β2t)
θt=θt−1−η√v̂t+ε·m̂t
AdamW adds: −ηλθt−1 (decoupled WD) Bias-corrected momentum + adaptive lr.
Hit Start to race all optimizers.