Reinforcement Learning from AI Feedback

RLAIF
An AI helps judge the practice.

An AI evaluator supplies feedback. A common setup trains a reward model on its comparisons, then uses that model for reinforcement learning.

The idea

A teacher writes a grading rubric. An automated teaching assistant uses it to compare essays. More essays can be graded, but a mistake in the rubric or grader can affect many students.

The judge is a model following a rubric.

Worked example · RLAIF6 steps

Step 1 of 6

People write the rules first.

Instead of labeling every pair by hand, people write short principles, sometimes collected into a “constitution”. Here there is one rule: choose the more helpful and respectful response.

Step 2 of 6

The model writes two answers.

The assistant being trained answers the same prompt twice. One answer gives specific, kind feedback. The other is blunt and gives the student nothing to act on.

Step 3 of 6

An AI judge applies the rule.

A separate AI model reads the principle, the prompt and both answers, then answers “A” or “B”. Its confidence becomes a soft label: 93% for A. The judge’s prompt is ordinary text, shown on the right.

Step 4 of 6

AI labels pile up fast.

The judge can label far more pairs than people could by hand. Each label is saved in the same format as a human preference, plus the judge’s confidence.

Step 5 of 6

Train a reward model, then practice with RL.

From here the recipe matches RLHF: a reward model learns the judge’s preferences, and an RL algorithm such as PPO trains the assistant against it.

Step 6 of 6

People still check.

An AI judge can be biased, fooled or confidently wrong, and the same mistake repeats in every label it writes. People review a sample of the judge’s labels and evaluate the trained assistant themselves.

Principle written by people

Choose the response that is more helpful and respectful.

Prompt

Give feedback on a student's draft essay.

Answer A

Your example is clear. Add one sentence explaining how it supports your point.

chosen

Answer B

This is bad. Rewrite it.

rejected

AI judge · reads the principle and both answers

“A is better”93%

“B is better”7%

AI-labeled pairs

Same format as human labels, plus "labeler": "ai-judge".

Reward model → RL

Learns the judge's preferences, then scores the assistant's practice answers: 1.4 PPO ▲

People check

Spot-check a sample of the judge's labels against human choices, and evaluate the trained assistant. One judge mistake can repeat in thousands of labels.

Task or dataModel being trainedJudge or scorerGood / chosenBad / rejectedFrozen or hidden

yamlprinciples.yamlnew at this step

principles:
  - Choose the response that is more helpful and respectful.

jsonto_label.jsonl · one rownew at this step

{
  "prompt": "Give feedback on a student's draft essay.",
  "answer_a": "Your example is clear. Add one sentence explaining how it supports your point.",
  "answer_b": "This is bad. Rewrite it."
}

textjudge promptnew at this step

Principle: Choose the response that is more helpful and respectful.
Prompt: Give feedback on a student's draft essay.
(A) Your example is clear. Add one sentence explaining how it supports your point.
(B) This is bad. Rewrite it.
Which response is better? Answer with A or B.
Judge: A   (p = 0.93)

jsonai_comparisons.jsonl · one rownew at this step

{
  "prompt": "Give feedback on a student's draft essay.",
  "chosen": "Your example is clear. Add one sentence explaining how it supports your point.",
  "rejected": "This is bad. Rewrite it.",
  "labeler": "ai-judge",
  "p_chosen": 0.93
}

jsonrollouts.jsonl · one rownew at this step

{
  "prompt": "Review my cover letter's opening line.",
  "response": "Strong start. Name the role you're applying for in the first sentence.",
  "reward": 1.4
}

jsonlabel_audit.jsonl · one rownew at this step

{
  "label_id": "ai-000917",
  "judge_choice": "A",
  "human_choice": "A",
  "agree": true
}

One common RLAIF pipeline with an illustrative judge confidence. Some variants use the judge's scores directly during RL instead of first training a separate reward model.

How RLAIF works

Set the judging rules
People specify the behavior they want.
Get AI feedback
An evaluator compares candidate answers using those rules.
Turn feedback into rewards
For example, train a model to predict the evaluator's preferences.
Train and check
Use RL to improve the assistant, then check its behavior independently.

Useful for

Collecting more preference feedback than people can feasibly label by hand.

The catch

An AI judge can be biased, fooled, or confidently wrong. AI feedback does not remove the need for human choices and evaluation.

A little more detail

Is this the same as Constitutional AI?

Not exactly. Constitutional AI is a broader recipe guided by written principles. Its original version includes both a critique-and-revision stage and an RLAIF stage.

RLAIF names the use of AI feedback; it is not one fixed set of principles.

Read the research

These guides simplify the methods. The papers describe the full training recipes. Examples on this page are illustrative, not experimental results.