Definition. Parameter update [kingma2015adam, Algorithm 1] [ftip-001I]
Definition. Parameter update [kingma2015adam, Algorithm 1] [ftip-001I]
A parameter update rule is a specified map \(U_k\) that takes the current parameters \(\theta _k\), optimizer state \(s_k\), gradient estimate \(g_k\), and scheduled hyperparameters \(\eta _k\), and returns \[ (\theta _{k+1},s_{k+1}) =U_k(\theta _k,s_k,g_k,\eta _k). \] The learning rate, momentum coefficients, numerical stabilizers, clipping, and weight decay belong to \(\eta _k\) or to the declared form of \(U_k\); they are not determined by the loss alone.