Definition. PPO rollout and minibatch update [schulman2017proximal, Section 5] [ftip-003H]
Definition. PPO rollout and minibatch update [schulman2017proximal, Section 5] [ftip-003H]
A PPO iteration collects a batch of trajectories under the behavior policy, computes returns or advantage estimates, and then performs several epochs of minibatch optimization on the same collected batch. The updated policy becomes the behavior policy for a later collection round.
The number of trajectories, epochs, minibatches, and optimizer steps governs how often one batch is reused. The clipped objective does not determine those protocol choices by itself.