Proximal Policy Optimization
PPO
Improve without changing too much at once.
PPO is an update algorithm. It uses experience and rewards to improve a model while discouraging some overly large changes.
The idea
When practicing a basketball shot, you adjust your movement rather than replacing your whole technique after one lucky basket. The analogy is about controlling changes, not a promise that every adjustment helps.
Try → score → adjust → try again.
Step 1 of 6
The model picks its next token.
A language model writes one token at a time; a token is a word or a piece of one. The model has written “Rain happens when water” and must choose what comes next. It picks “ drops”, which it gave a 40% chance. In PPO, this model is called the policy.
Step 2 of 6
The finished answer earns a reward.
The model keeps writing until the answer is done. A reward model, like the one trained in RLHF, scores the whole answer: 0.9. Each token choice in that answer shares the credit for the result.
Step 3 of 6
A critic predicted the score in advance.
PPO trains a second network, the critic (or value model). Looking only at the words so far, it predicted that this answer would score about 0.5. The critic learns too, so its predictions improve.
Step 4 of 6
Better than expected? That is the advantage.
Advantage = reward − prediction = 0.9 − 0.5 = +0.4. A positive advantage means “do this more often”. A negative one means “do this less often”. An answer that scored exactly what the critic expected teaches nothing.
Step 5 of 6
Push the chance up, but stop at 1.2 times.
PPO compares the new chance with the old one; this is the ratio. Once the ratio passes 1.2, PPO stops giving extra credit for pushing higher. So “ drops” can rise from 40% to at most 48% in this round, and the other tokens shrink to make room.
Step 6 of 6
Collect fresh practice with the updated model.
After a few passes over this batch, the updated model becomes the new “old” model. It writes new answers, they get new rewards, and the loop starts again. The 1.2 limit applies to each round, not to training as a whole.
Rain happens when waterRain happens when water drops in clouds grow heavy and fall.
Picked: “ drops”, at 40%.
Scores the finished answer
0.9
Predicted it from the words so far
0.5
reward 0.9 − prediction 0.5 = +0.4 better than expected → do it more
0.48 ÷ 0.40 = 1.2: the chance of “ drops” stops rising at 48% this round.
Task or dataModel being trainedJudge or scorerCritic (predicts the score)Good / chosenTraining update
jsonrollout buffer · one rownew at this step
{
"prompt": "Explain rain to a younger student.",
"context": "Rain happens when water",
"action": " drops",
"p_old": 0.40,
"logprob_old": -0.92
}Trainers store the logarithm of the chance (ln 0.40 = -0.92), because adding logs is easier than multiplying many small chances.
jsonrollout buffer · one rownew at this step
{
"context": "Rain happens when water",
"action": " drops",
"p_old": 0.40,
"reward": 0.9
}jsonrollout buffer · one rownew at this step
{
"context": "Rain happens when water",
"action": " drops",
"p_old": 0.40,
"reward": 0.9,
"value": 0.5
}jsonrollout buffer · one rownew at this step
{
"context": "Rain happens when water",
"action": " drops",
"p_old": 0.40,
"reward": 0.9,
"value": 0.5,
"advantage": 0.4
}pythonppo_loss.pynew at this step
# ratio = 0.48 / 0.40 = 1.2
ratio = p_new / p_old
# keep the ratio between 0.8 and 1.2
clipped = min(max(ratio, 0.8), 1.2)
objective = min(ratio * advantage, clipped * advantage)The 0.8 and 1.2 come from ε = 0.2, the clip range in the PPO paper. With a negative advantage the same rule stops the push at 0.8 instead.
jsonrollout buffer · next roundnew at this step
{
"context": "Rain happens when water",
"action": " drops",
"p_old": 0.48
}How PPO works
Try the current policy
A policy is the model's rule for choosing an action or the next token.
Estimate what helped
Compare observed outcomes with an estimate of expected reward.
Make restrained updates
The common clipped version reduces the incentive for excessively large probability changes.
Collect new practice
Repeat with the updated model.
Useful for
Updating policies in games, robotics, and reward-model-based language-model training.
The catch
PPO cannot fix a bad reward. A model can become better at earning the score without becoming better at the real task.
A little more detail
PPO is not a feedback source
RLHF can use PPO to learn from human-derived rewards. A verifier can also provide rewards. PPO tells the model how to update, not what deserves a high score.
Who works with PPO
RL environment builders and AI benchmarks with public evidence that they use PPO: their own website says so, or two independent sites do. Only confirmed evidence is shown; each page lists the quotes.
AI benchmarks 14
- Agent-environment papers
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - DrugDiscoveryBench with Phylo
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - HiL-Bench
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - Humanity's Last Exam
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - LifeSciBench
Evidence on its own site
“We’re releasing a new class of reinforcement learning algorithms, Proximal Policy Optimization (PPO), which perform comparably or better than state-of-the-art approaches while being much simpler to implement and tune. PPO has become the default reinforcement learning algorithm at OpenAI because of its ease of use…”
openai.com - MCP Atlas
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - PRBench
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - Production-debugging RL environment (hud-ops-diagnostics) - 2026-01-20
Evidence on its own site
“A practical guide to Group Relative Policy Optimization. Learn how GRPO works, why it replaced PPO for verifiable-reward training, and how to run a GRPO training loop on HUD.”
hud.ai - READY or Not enterprise agent deployment benchmarks
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - Remote Labor Index
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - RSI Bench
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - Rubric-based RL papers
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - SWE-Bench Pro - Private Dataset
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com - SWE-Bench Pro v1 (2025-09-19) and v2
Evidence on its own site
“…lly incorrect and adjusts the importance sampling ratio for the "off-policy" trajectories in order to optimize the policy for contexts in which the hints are no longer present. We describe variants of Guide for GRPO and PPO and empirically show that Guide-GRPO on 7B and 32B parameter models improves generalization over its vanilla counterpart with up to 4% macro-average improvement across math benchmarks. We include careful ablations to an…”
labs.scale.com
Read the research
These guides simplify the methods. The papers describe the full training recipes. Examples on this page are illustrative, not experimental results.