Reinforcement Learning from Contrastive Distillation
RLCD
Learn from deliberately contrasting examples.
Generate answers with opposite instructions. Treat them as a preference pair, train a scoring model, then use its scores for reinforcement learning.
The idea
A writing exercise asks for one clear explanation and one deliberately confusing explanation. Putting them side by side makes the intended difference easier to spot. The labels can still be wrong if the writer ignores the instructions.
Two prompts create the contrast.
Explain how a bicycle brake works.
Squeezing the lever pulls the brake, which slows the wheel.
Deceleration follows the actuation of a frictional interface.
Prefer the positive-prompt output by construction, not because a person checked this pair.
Synthetic preference pairs → train a reward model
That reward model scores new answers during RL training.
How RLCD works
Pick a principle
For example, make an explanation helpful or easy to understand.
Generate a contrast
Use a positive instruction and an opposing negative instruction.
Learn the preference
Train a reward model on the resulting preferred and dispreferred pairs.
Train with rewards
Improve the assistant using that reward model and an RL algorithm.
Useful for
Generating preference data around a written principle when human comparisons are scarce.
The catch
The positive-prompt answer is not automatically better. Noisy generated pairs can teach the reward model the wrong lesson.
A little more detail
What the name means here
This guide uses RLCD as defined in Yang and colleagues’ paper: reinforcement learning from contrastive distillation. It is a specific research method, not a name for every kind of synthetic training data.
Read the research
These guides simplify the methods. The papers describe the full training recipes. Examples on this page are illustrative, not experimental results.