A Nonlinear Time-Series Theory for Constant-step SGD with Momentum
The paper studies the stationary behavior and distributional limits of constant-step stochastic gradient descent with momentum (SGDM) as a nonlinear time series.
This paper provides a new framework for understanding the stationary behavior and distributional limits of SGDM, which is a novel contribution compared to prior work.
Before reading this…
Applications
- →Machine learning
- →Optimization algorithms
To understand this paper, make sure you know these concepts first:
- Stochastic gradient descentfind papers →
- Momentum-based optimizationfind papers →
Abstract
More Like ThisConstant-step stochastic gradient descent with momentum (SGDM) augments the parameter recursion with a momentum state that carries geometrically decaying memory of past stochastic gradients. We study this coupled recursion as a nonlinear time series and develop a framework for its stationary behavior and distributional limits. The joint parameter-momentum process forgets its initialization geometrically under a finite $p$th moment for $p > 1$, yielding a unique stationary causal solution. Under a second-moment condition, this contraction leads to an invariance principle and quenched Gaussian limit for averaged SGDM, while a small-learning-rate analysis gives a Gaussian limit for the last iterate. In a stochastic-curvature quadratic model, we further study how the rate of initialization forgetting relates to stationary variance and averaged fluctuations.