Definition. Generalized advantage estimator [schulman2016gae, Section 3, equations (10) and (16)] [ftip-003E]
Definition. Generalized advantage estimator [schulman2016gae, Section 3, equations (10) and (16)] [ftip-003E]
Given an approximate value function \(V\), with \(V(\mathsf H_N)=0\) at a terminal information state, define the temporal-difference residual
\[ \delta _t^V=R_{t+1}+\gamma V(\mathsf H_{t+1})-V(\mathsf H_t). \]For \(\lambda \in [0,1]\), the finite-episode generalized advantage estimator is
\[ \widehat A_t^{\mathrm {GAE}(\gamma ,\lambda )} =\sum _{l=0}^{N-t-1}(\gamma \lambda )^l\delta _{t+l}^V. \]The parameters \(\gamma \) and \(\lambda \) trade temporal reach against the variance and approximation error induced by bootstrapping. The estimator also depends on the fitted value function.