AdaGrad vs RMSProp — Three Honest Scenarios

Each scenario is numerically verified to show a genuine algorithmic difference — not a tuning artifact

RMSProp rho 0.90

Scenario

GDGradient Descent

Effective lr over time

AGAdaGrad

Effective lr over time

RMRMSProp

Effective lr over time

Effective lr: GD (orange) AdaGrad (blue) RMSProp (purple)

GD Steps

GD Loss

—

GD theta

—

AG Steps

AG Loss

—

AG theta

—

RM Steps

RM Loss

—

RM theta

—

Gradient Descent θ_t = θ_t−1 − η · g_t

Fixed effective lr = η every step

AdaGrad G_t = G_t−1 + g_t²
θ_t = θ_t−1 − η √G_t + ε · g_t

G_t grows monotonically → η/(√G_t + ε) → 0

RMSProp G_t = ρ · G_t−1 + (1−ρ) · g_t²
θ_t = θ_t−1 − η √G_t + ε · g_t

EMA keeps G_t bounded → η/√G_t stays healthy