How RL training
fits together.

Some methods choose the feedback. Others decide how to update the model. They can work together.

Recipe map · how the methods connect7 steps

Step 1 of 7

Two questions decide every recipe.

Where does the feedback come from? People, an AI judge, contrasting prompts, or a program that checks answers. How does the model update? Usually PPO or GRPO. DPO is a shortcut that skips the practice loop. Press play to trace the common recipes one at a time.

Step 2 of 7

Most recipes start with worked examples.

Before any reinforcement learning, SFT teaches the model by example: here is a prompt, here is a good answer, copy the pattern. The result is a model that can already chat, which the later stages improve.

Step 3 of 7

The classic recipe: people's choices, a reward model, then PPO.

People compare pairs of answers (RLHF), a reward model learns their taste, and PPO uses its scores to update the model, over and over. InstructGPT, an early ChatGPT-style assistant, was trained this way.

Step 4 of 7

The reasoning recipe: a checker, then GRPO.

For math and code, a program can check the answer, so no reward model is needed (RLVR). GRPO compares a group of attempts at the same task. DeepSeek-R1 trained its reasoning with this pairing.

Step 5 of 7

An AI judge replaces most human labels.

People write principles; an AI judge applies them to pairs of answers (RLAIF). From there it is the RLHF route: reward model, then PPO. Anthropic’s Constitutional AI used AI feedback for its RL stage.

Step 6 of 7

Contrasting prompts make the labels.

RLCD asks the same model for a helpful and an unhelpful answer, and labels the pair by construction, with no judge. A reward model learns from those pairs, then PPO trains as usual.

Step 7 of 7

The shortcut: learn straight from the pairs.

DPO takes preference pairs, from people or an AI judge, and updates the model directly. There is no reward model and no practice loop, which makes it simpler and cheaper, but it can only learn from the pairs it is given.

SFT start here Learn from worked examples first.

The model

Writes answers, gets feedback, updates.

Where the feedback comes from

RLHF People pick the better answer RLAIF An AI judge picks, using rules RLCD Opposite prompts make the pairs RLVR A program checks the answer

Reward model

Learns from the labeled pairs, then scores new answers.

How the model updates

PPO Small, clipped updates with a critic. GRPO Compares a group of attempts.

A separate route, no practice loop

Preference pairs

Chosen and rejected answers, from people or an AI judge.

DPO Updates the model directly from the pairs.

Task or dataModel being trainedJudge or scorerCritic (predicts the score)Training updateFrozen or hidden

textthe piecesnew at this step

feedback sources:   RLHF · RLAIF · RLCD · RLVR
update algorithms:  PPO · GRPO
related methods:    SFT · DPO

jsonsft.jsonl · one rownew at this step

{"messages": [
  {"role": "user", "content": "Explain 6 × 7."},
  {"role": "assistant", "content": "Six groups of seven make 42."}
]}