How RL training
fits together.

Some methods choose the feedback. Others decide how to update the model. They can work together.

SFT often provides a starting model. In a typical RL loop, RLHF, RLAIF, or RLCD supply preferences for a reward model; RLVR supplies rewards from checks. PPO or GRPO uses the rewards to update the model, which tries again. Standard DPO is a separate route using preference pairs without a reward model or online RL loop.
Typical recipes, simplified. Not every method requires SFT; RLAIF also has direct-feedback variants. Open full diagram ↗ Edit in Excalidraw ↓

Two ways to combine them

RLHF → reward model → PPOPeople’s preferences guide the updates.

RLVR → code tests → GRPOTest results guide the updates.

DPO can use preference pairs too, but learns from them directly, without a separate reward model.