Remark. PPO is an optimizer, not an alignment definition [ftip-003K]
Remark. PPO is an optimizer, not an alignment definition [ftip-003K]
RLHF is a feedback-and-training pipeline; PPO is one optimizer used within that pipeline. PPO specifies a sampled update surrogate; it does not specify whose preferences are collected, what the reward means, or which independent behavior counts as aligned.
Changing the comparison population or reward model can change the alignment target while leaving PPO unchanged. Conversely, replacing PPO by another optimizer changes update geometry without necessarily changing the declared preference data or evaluation criterion.