← All methodologies

Proximal Policy Optimization

PPO
Improve without changing too much at once.

PPO is an update algorithm. It uses experience and rewards to improve a model while discouraging some overly large changes.

The idea

When practicing a basketball shot, you adjust your movement rather than replacing your whole technique after one lucky basket. The analogy is about controlling changes, not a promise that every adjustment helps.

Try → score → adjust → try again.

Starting point

A model has a current way of answering.

Before the update

Generate practice answers with the current policy.

After the update

Make rewarded choices more likely, with limits on the incentive to change too far.

Feedback

Estimate which choices did better than expected.

PPO's clipped objective restrains the update incentive.

Collect fresh experience with the updated policy.

A conceptual loop, not a numeric simulation. Clipping is not a hard guarantee that the whole model changes only a little.

How PPO works

  1. Try the current policy

    A policy is the model's rule for choosing an action or the next token.

  2. Estimate what helped

    Compare observed outcomes with an estimate of expected reward.

  3. Make restrained updates

    The common clipped version reduces the incentive for excessively large probability changes.

  4. Collect new practice

    Repeat with the updated model.

Useful for

Updating policies in games, robotics, and reward-model-based language-model training.

The catch

PPO cannot fix a bad reward. A model can become better at earning the score without becoming better at the real task.

A little more detail

PPO is not a feedback source

RLHF can use PPO to learn from human-derived rewards. A verifier can also provide rewards. PPO tells the model how to update, not what deserves a high score.

Read the research

These guides simplify the methods. The papers describe the full training recipes. Examples on this page are illustrative, not experimental results.