← All methodologies

Direct Preference Optimization

DPO
Learn directly from “this answer is better.”

Train on pairs of preferred and rejected answers, directly. Standard DPO skips a separately trained reward model and the online RL practice loop.

The idea

A writing tutor gives you pairs of paragraphs and marks the stronger one. You learn from those comparisons directly, instead of first training another tutor to assign scores to new paragraphs.

Preference pair → direct update.

Worked example · DPO6 steps

Step 1 of 6

Start with a labeled pair.

Each DPO training row holds one prompt and two answers. People, or an AI judge, already marked the better one: the chosen answer. The other is the rejected answer.

Step 2 of 6

Freeze a copy of the model as the reference.

Training starts from a model, usually one that already went through SFT. DPO copies it and freezes the copy. This reference model never changes; it is the “before” picture that the trainable model is measured against.

Step 3 of 6

Both models score both answers.

Each model reports a log-probability for each answer: how likely it is to write exactly that text. Closer to 0 means more likely. At the start the two models are identical, and both slightly prefer the jargon answer.

Step 4 of 6

The loss checks which answer gained more.

DPO asks: compared with the reference, did the trainable model raise the chosen answer more than the rejected one? Right now neither moved, so the margin is 0 and the loss is 0.69. The loss only gets small when the chosen answer pulls ahead.

Step 5 of 6

Training moves the two answers apart.

After some updates the trainable model’s score for the chosen answer rises from −14.0 to −12.5, and the rejected answer falls from −12.0 to −14.5. The margin grows to 0.4 and the loss drops to 0.51. The reference stays where it was.

Step 6 of 6

No reward model and no practice loop.

That is the whole recipe: labeled pairs in, a direct update out. Unlike RLHF, standard DPO never trains a separate reward model and never samples new answers while it trains. The trade-off: it can only learn from the pairs it is given.

Prompt

What does “photosynthesis” mean? Explain it to a beginner.

Chosen

It's how plants make food: they use sunlight to turn water and air into sugar.

Rejected

Photosynthesis is the photosynthetic process of photoautotrophs.

Trainable model

Starts as an exact copy. DPO changes this one.

Reference model frozen

The “before” picture. Never updated.

Log-probability of writing each answer
chosen ref -14.0 -12.5 +1.5
rejected ref -12.0 -14.5 −2.5

−16 · less likelymore likely · −10

DPO loss
0.51

margin = 0.1 × (+1.5 − −2.5) = 0.40

What DPO skips

RLHFpairs→reward model→sample answers→score→PPO update→repeat

DPOpairs→DPO loss→update

Task or dataModel being trainedGood / chosenBad / rejectedFrozen or hiddenTraining update

jsonprefs.jsonl · one rownew at this step

{
  "prompt": "What does 'photosynthesis' mean? Explain it to a beginner.",
  "chosen": "It's how plants make food: they use sunlight to turn water and air into sugar.",
  "rejected": "Photosynthesis is the photosynthetic process of photoautotrophs."
}

jsondpo_config.jsonnew at this step

{
  "policy": "assistant-sft (trainable)",
  "reference": "assistant-sft (frozen copy)",
  "beta": 0.1
}

beta (β) sets how strongly the comparison counts. A small β lets the model move further from the reference.

jsonscores.jsonnew at this step

{"answer": "chosen",   "logp_policy": -14.0, "logp_ref": -14.0}
{"answer": "rejected", "logp_policy": -12.0, "logp_ref": -12.0}

pythondpo_loss.pynew at this step

chosen_gain = logp_c - ref_c        # -14.0 - (-14.0) = 0.0
rejected_gain = logp_r - ref_r      # -12.0 - (-12.0) = 0.0
margin = beta * (chosen_gain - rejected_gain)    # 0.0
loss = -math.log(sigmoid(margin))                # 0.69

pythondpo_loss.pynew at this step

chosen_gain = -12.5 - (-14.0)      # +1.5
rejected_gain = -14.5 - (-12.0)    # -2.5
margin = 0.1 * (1.5 - (-2.5))       # 0.4
loss = -math.log(sigmoid(0.4))     # 0.51

textpipelinesnew at this step

RLHF: pairs → reward model → sample answers → score → PPO update → repeat
DPO:  pairs → DPO loss → update
A toy pair with illustrative log-probabilities. The loss is the standard DPO objective from Rafailov et al. Standard offline DPO trains on supplied pairs; it does not sample and score new answers inside each training step like an online RL loop.

How DPO works

  1. Collect preference pairs

    Each prompt has a preferred and a rejected answer.

  2. Keep a reference

    A reference model anchors the comparison.

  3. Train directly

    Use the DPO objective to change the relative likelihood of the two answers.

  4. Evaluate independently

    Check whether the trained model improved on new prompts.

Useful for

Preference tuning when a suitable comparison dataset is available.

The catch

DPO learns the preferences in its data, including their mistakes. Standard offline training does not automatically explore new answers.

A little more detail

Is DPO reinforcement learning?

DPO is derived from a reward-regularized preference-learning objective, but its standard training procedure avoids the usual RL rollout-and-reward loop. We label it a related preference method, not another reward source like RLHF or RLVR.

Who works with DPO

RL environment builders and AI benchmarks with public evidence that they use DPO: their own website says so, or two independent sites do. Only confirmed evidence is shown; each page lists the quotes.

RL environments 7

  • Collinear
    Evidence on its own site
    “4. Preference Optimization: Iteratively improve the judge model by DPO, resulting in an improved judge model at each iteration.”
    blog.collinear.ai
  • Revelo
    Evidence on its own site
    “Experts compare, rank, and rate your model’s outputs for RLHF, DPO, and reward models.”
    revelo.com
  • Scale AI
    Evidence on its own site
    “…rovides targeted, natural language guidance based on the specific error, suggesting a plan or pointing to relevant files. The student agent then re-attempts the task with this guidance. The policy is updated via offline DPO, learning from the successful guided trajectories to prefer correct solutions. - What We Found: - Agent-RLVR is highly effective, elevating the performance of a 72B parameter model on the SWE-Bench benchmark from a 9.4%…”
    scale.com
  • Snorkel AI
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • Turing
    Evidence on its own site
    “If you're looking to optimize your LLMs for real-world applications, Turing offers comprehensive LLM training services that integrate SFT, RLHF, and direct preference optimization (DPO). By leveraging high-quality proprietary human data, extensive STEM domain expertise, and advanced training techniques, we’ve helped leading LLM companies and research organizations such as OpenAI, Google Gemini, M…”
    turing.com
  • Uber AI
    Evidence on its own site
    “Leverage domain specialists for SFT, RLHF, direct preference optimization (DPO), and adversarial red teaming to help improve model safety and reasoning.”
    uber.com
  • Ulam
    Evidence on its own site
    “Training views: SFT, DPO, critic and repair data, RLVR/GRPO tasks, process-reward steps, private evaluation, and regression tests.”
    ulam.ai

AI benchmarks 35

  • Agents' Last Exam
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • Arena AI Document
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • Arena AI Image Edit
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • Arena AI Image-to-Video
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • Arena AI Text-to-Image
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • Arena AI Text-to-Video
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • Arena AI Video Edit
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • Arena AI WebDev Arena
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • Benchmarking Agents in Insurance Underwriting Environments
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • BigLaw Bench: Research
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • Cats Confuse Reasoning LLM
    Evidence on its own site
    “4. Preference Optimization: Iteratively improve the judge model by DPO, resulting in an improved judge model at each iteration.”
    blog.collinear.ai
  • Continual Learning Bench
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • CWE-bench
    Evidence on its own site
    “4. Preference Optimization: Iteratively improve the judge model by DPO, resulting in an improved judge model at each iteration.”
    blog.collinear.ai
  • CWE-bench v1
    Evidence on its own site
    “4. Preference Optimization: Iteratively improve the judge model by DPO, resulting in an improved judge model at each iteration.”
    blog.collinear.ai
  • FinQA / SnorkelFinance environment and benchmark
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • Learning from Less: RLVR in Low Data Regimes
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • LibraryDesignBench
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • LifeSciBench
    Evidence on its own site
    “Vision fine-tuning\\ \\ Learn to fine-tune for computer vision with image inputs. Direct preference optimization\\ \\ Fine-tune a model using direct preference optimization (DPO). Reinforcement fine-tuning\\ \\ Fine-tune a reasoning model by grading its outputs.”
    developers.openai.com
  • LMArena Search Arena
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • LMArena Text Arena
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • LMArena Vision Arena
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • LMArena WebDev Arena
    Evidence on its own site
    “…ing LLMs. To do this, we RLHF-tune a base LLM using several reward models and evaluate the resulting LLMs by measuring real-world human preference (Arena Scores). The experiment uses Llama-3.1-8B-Instruct, with RLHF via Direct Preference Optimization (DPO). Results are collected from 12,190 human votes in Chatbot Arena, and final Arena Scores depend solely on the reward model used.”
    arena.ai
  • Math RL Tasks
    Evidence on its own site
    “Training views: SFT, DPO, critic and repair data, RLVR/GRPO tasks, process-reward steps, private evaluation, and regression tests.”
    ulam.ai
  • OSWorld 2.0
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • RIFT / Hitchhiker's Guide to Rubric Quality
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • Self-rationalization improves LLM as a fine-grained judge
    Evidence on its own site
    “4. Preference Optimization: Iteratively improve the judge model by DPO, resulting in an improved judge model at each iteration.”
    blog.collinear.ai
  • Senior SWE-bench
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • Shrinking the Generation-Verification Gap with Weak Verifiers
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • SlopCodeBench
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • Snorkel: Rapid Training Data Creation with Weak Supervision
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • Terminal-Bench
    Evidence on its own site
    “Our work explored two techniques for model alignment: RLHF and direct preference optimization (DPO).”
    snorkel.ai
  • The Valley of Code Reasoning
    Evidence on its own site
    “4. Preference Optimization: Iteratively improve the judge model by DPO, resulting in an improved judge model at each iteration.”
    blog.collinear.ai
  • TraitBasis / Impatient Users Confuse AI Agents
    Evidence on its own site
    “4. Preference Optimization: Iteratively improve the judge model by DPO, resulting in an improved judge model at each iteration.”
    blog.collinear.ai
  • VERITAS
    Evidence on its own site
    “4. Preference Optimization: Iteratively improve the judge model by DPO, resulting in an improved judge model at each iteration.”
    blog.collinear.ai
  • YC-Bench
    Evidence on its own site
    “4. Preference Optimization: Iteratively improve the judge model by DPO, resulting in an improved judge model at each iteration.”
    blog.collinear.ai

Read the research

These guides simplify the methods. The papers describe the full training recipes. Examples on this page are illustrative, not experimental results.