← All methodologies

Reinforcement Learning from Contrastive Distillation

RLCD
Learn from deliberately contrasting examples.

Generate answers with opposite instructions. Treat them as a preference pair, train a scoring model, then use its scores for reinforcement learning.

The idea

A writing exercise asks for one clear explanation and one deliberately confusing explanation. Putting them side by side makes the intended difference easier to spot. The labels can still be wrong if the writer ignores the instructions.

Two prompts create the contrast.

Worked example · RLCD6 steps

Step 1 of 6

Write the same question two ways.

RLCD starts from one question and adds opposite instructions. The positive version asks for a clear, helpful answer. The negative version asks for a confusing, unhelpful one.

Step 2 of 6

The same model answers both versions.

Same model, same question; only the instruction differs. The positive prompt produces o₊ and the negative prompt produces o₋.

Step 3 of 6

The label comes from the setup, not a judge.

RLCD marks o₊ as chosen and o₋ as rejected automatically, because of which prompt produced them. No person or AI judge compares this pair. The instructions are dropped, so the pair is stored under the plain question.

Step 4 of 6

Train a reward model on many such pairs.

Thousands of contrast pairs teach a reward model what a clear, helpful answer looks like. It learns to score o₊-style answers above o₋-style answers.

Step 5 of 6

Use the reward model for RL practice.

As in RLHF, the assistant answers new prompts, the reward model scores them, and PPO makes higher-scoring answers more likely.

Step 6 of 6

The catch: a bad contrast makes a wrong label.

If the model ignores an instruction, the “positive” answer can come out worse than the “negative” one, and the pair still says positive wins. Noisy pairs like this teach the reward model the wrong lesson.

Question

Explain how a bicycle brake works.

Positive prompt

…works. (Give a clear, helpful answer.)

Negative prompt

…works. (Give a confusing, unhelpful answer.)

Same model, both prompts

Writes one answer for each version.

o₊ · from the positive prompt

Squeezing the lever pulls a cable. Rubber pads grip the wheel rim, and friction slows it down.

chosen · by construction
o₋ · from the negative prompt

Deceleration follows the actuation of a frictional interface.

rejected · by construction
Reward model
o₊ before 1.5
o₋ before -1.0
RL practice

New prompt: “How does a zipper work?” → reward 1.3 → PPO ▲

The catch · a noisy pair

chosen Brakes are complicated. They just work.

rejected Pulling the lever presses rubber pads against the wheel to slow it.

The model ignored its instructions, so this label is backwards.

Task or dataModel being trainedJudge or scorerGood / chosenBad / rejectedTraining update

jsoncontrast_prompts.jsonnew at this step

{
  "prompt": "Explain how a bicycle brake works.",
  "positive": "Explain how a bicycle brake works. (Give a clear, helpful answer.)",
  "negative": "Explain how a bicycle brake works. (Give a confusing, unhelpful answer.)"
}

jsonoutputs.jsonnew at this step

{
  "o_plus": "Squeezing the lever pulls a cable. Rubber pads grip the wheel rim, and friction slows it down.",
  "o_minus": "Deceleration follows the actuation of a frictional interface."
}

jsonprefs.jsonl · one rownew at this step

{
  "prompt": "Explain how a bicycle brake works.",
  "chosen": "Squeezing the lever pulls a cable. Rubber pads grip the wheel rim, and friction slows it down.",
  "rejected": "Deceleration follows the actuation of a frictional interface.",
  "label_source": "prompt contrast"
}

pythonreward_model_loss.pynew at this step

loss = -math.log(sigmoid(r_chosen - r_rejected))
# after training: r(o_plus) = 1.5, r(o_minus) = -1.0

jsonrollouts.jsonl · one rownew at this step

{
  "prompt": "How does a zipper work?",
  "response": "Each side has a row of teeth. The slider presses them together so they lock.",
  "reward": 1.3
}

jsonprefs.jsonl · a noisy rownew at this step

{
  "prompt": "Explain how a bicycle brake works.",
  "chosen": "Brakes are complicated. They just work.",
  "rejected": "Pulling the lever presses rubber pads against the wheel to slow it.",
  "label_source": "prompt contrast"
}

The label says the first answer is better, but a person would disagree.

An illustrative pair, not a measured result. The prompt wording is simplified from the RLCD paper, which appends short descriptions such as “(helpful, honest response)” and their opposites. Unlike a typical RLAIF comparison, no judge chooses the winner.

How RLCD works

  1. Pick a principle

    For example, make an explanation helpful or easy to understand.

  2. Generate a contrast

    Use a positive instruction and an opposing negative instruction.

  3. Learn the preference

    Train a reward model on the resulting preferred and dispreferred pairs.

  4. Train with rewards

    Improve the assistant using that reward model and an RL algorithm.

Useful for

Generating preference data around a written principle when human comparisons are scarce.

The catch

The positive-prompt answer is not automatically better. Noisy generated pairs can teach the reward model the wrong lesson.

A little more detail

What the name means here

This guide uses RLCD as defined in Yang and colleagues’ paper: reinforcement learning from contrastive distillation. It is a specific research method, not a name for every kind of synthetic training data.

Read the research

These guides simplify the methods. The papers describe the full training recipes. Examples on this page are illustrative, not experimental results.