Definition. Optimizer state [kingma2015adam, Algorithm 1] [ftip-001H]
Definition. Optimizer state [kingma2015adam, Algorithm 1] [ftip-001H]
An optimizer state \(s_k\) is the collection of persistent variables, other than the model parameters \(\theta _k\), that an update rule uses after step \(k\). For Adam, \(s_k\) contains the step index and exponential moving averages of the gradient and squared gradient.
The state is initialized by a declared value \(s_0\). Two runs with the same checkpoint \(\theta _k\) but different optimizer states need not have the same next update.