Direct Preference Optimization
DPO
Learn directly from “this answer is better.”
Train on pairs of preferred and rejected answers, directly. Standard DPO skips a separately trained reward model and the online RL practice loop.
The idea
A writing tutor gives you pairs of paragraphs and marks the stronger one. You learn from those comparisons directly, instead of first training another tutor to assign scores to new paragraphs.
Preference pair → direct update.
Explain a difficult word to a beginner.
Use a familiar example and one short definition.
Give a circular definition with more unexplained jargon.
A training dataset already says which answer is preferred.
DPO loss → update the model directly
Shift relative likelihood toward preferred answers, using a reference model.
How DPO works
Collect preference pairs
Each prompt has a preferred and a rejected answer.
Keep a reference
A reference model anchors the comparison.
Train directly
Use the DPO objective to change the relative likelihood of the two answers.
Evaluate independently
Check whether the trained model improved on new prompts.
Useful for
Preference tuning when a suitable comparison dataset is available.
The catch
DPO learns the preferences in its data, including their mistakes. Standard offline training does not automatically explore new answers.
A little more detail
Is DPO reinforcement learning?
DPO is derived from a reward-regularized preference-learning objective, but its standard training procedure avoids the usual RL rollout-and-reward loop. We label it a related preference method, not another reward source like RLHF or RLVR.
Read the research
These guides simplify the methods. The papers describe the full training recipes. Examples on this page are illustrative, not experimental results.