Reinforcement Learning from Human Feedback
RLHF
People show what “better” means.
People compare answers. A scoring model learns those preferences. The assistant then practices getting higher scores.
The idea
A teacher compares two essays and picks the clearer one. After seeing many choices, a teaching assistant learns how to grade. The student can now practice with that assistant's feedback.
First learn the score. Then learn from it.
Explain rain to a younger student.
Rain happens when water drops in clouds grow heavy enough to fall.
Atmospheric precipitation results from hydrometeor growth.
A person prefers A for this audience.
Many comparisons → train a reward model
The assistant writes new answers → gets scores → updates → tries again.
How RLHF works
Collect comparisons
Ask people which of two answers they prefer.
Train a scoring model
A reward model learns to predict those choices.
Practice and update
The assistant generates answers. An RL algorithm uses the scores to adjust it.
Check with people again
Test whether higher scores actually produced more helpful answers.
Useful for
Helpful explanations, tone, and other goals without a single answer key.
The catch
A higher predicted preference score is not proof of truth. People and reward models can both favor a convincing but wrong answer.
A little more detail
Two words worth knowing
Reward means a training score, not a feeling. A reward model is another model trained to give that score.
One important distinction
RLHF describes the role of human feedback. PPO is one algorithm that can perform the updates. They are not competing names for the same thing.
Read the research
These guides simplify the methods. The papers describe the full training recipes. Examples on this page are illustrative, not experimental results.