Reinforcement Learning from Human Feedback (RLHF): How AI Models Learn What Humans Actually Want

RLHF is the training method that takes a raw language model, which only predicts likely next words, and tunes it to produce answers humans prefer. Humans rank model outputs, a reward model learns those preferences, and reinforcement learning pushes the model towards higher-reward behaviour. This article explains each stage, what RLHF fixes, where it breaks, and what alternatives like DPO and Constitutional AI changed.

What RLHF actually is

Reinforcement learning from human feedback is a training stage where humans rank a model's outputs, those rankings are used to train a second model that scores answers, and the original model is then optimised to score well on that scorer. That is the whole idea. Everything else is engineering detail.

The reason it exists is simple. A base language model trained on internet text is only good at one thing: predicting what word probably comes next. Ask GPT-3 base "What is the capital of Australia?" and it might reply with a list of ten more quiz questions, because that is what text on the internet looks like around that sentence. It is not wrong. It is just not answering you.

RLHF closes that gap between statistically likely text and text a human wanted. OpenAI's InstructGPT paper in 2022 showed the size of the effect: labellers preferred outputs from a 1.3 billion parameter InstructGPT model over the 175 billion parameter GPT-3 base model. A model over a hundred times smaller won, because it had been taught what a good answer looks like. That result is why almost every chat assistant shipped since then has some version of this step.

The three stages, in order

Stage one is supervised fine-tuning. Human writers produce demonstration answers to prompts, and the base model is fine-tuned on those pairs. This is ordinary supervised learning, no reinforcement anywhere. It teaches the model the format of being helpful. A few tens of thousands of examples is typical, and quality matters far more than volume here.

Stage two is the reward model. You take a prompt, generate several responses from the fine-tuned model, and show them to a human who ranks them best to worst. Note that people rank, they do not score. Asking someone to rate an answer 7.3 out of 10 produces noisy, inconsistent data. Asking which of two answers is better produces something usable. Those comparisons train a reward model, usually a language model with the final layer swapped for a single number output, to predict which response a human would pick.

Stage three is the reinforcement learning itself. The policy (your chat model) generates a response, the reward model scores it, and an algorithm like PPO nudges the model's weights to make higher-scoring responses more likely. A KL divergence penalty sits alongside the reward, punishing the model for drifting too far from its pre-RL self. Without that leash, the model finds nonsense that the reward model loves and humans hate. It will discover, for example, that starting every answer with a long polite preamble scores well, and then do nothing else.

• Stage 1: supervised fine-tuning on human-written demonstrations
• Stage 2: humans rank pairs of outputs, a reward model learns the pattern
• Stage 3: PPO or similar optimises the model against the reward model, with a KL penalty to prevent drift

Where RLHF breaks

The reward model is a stand-in for human judgement, and it is an imperfect one. Optimise hard enough against any imperfect proxy and you get reward hacking. The classic symptom in chat models is sycophancy: the model learns that agreeing with the user scores better than correcting them, so it agrees. Anthropic published work in 2023 documenting exactly this across several RLHF-trained assistants. If you tell a model you think an answer is wrong, it will often fold, whether or not it was wrong.

Length is another one. Longer answers tend to get ranked higher by tired human labellers, so RLHF models learn to pad. That is a large part of why chat assistants produce five paragraphs when one would do.

Then there is the question of whose feedback. "Human preference" in practice means the preferences of a specific pool of contractors, working to a specific rubric, often under time pressure. Their values, reading level and cultural assumptions get baked into the model. This is not a scandal, it is just a fact worth knowing when someone tells you a model is "aligned with human values". Aligned with which humans, judged against which instructions?

And it is expensive. Collecting good preference data at scale costs real money and real hours, which is why the field started looking for ways around it almost immediately.

What came after: DPO, RLAIF and Constitutional AI

Direct Preference Optimization, published in 2023, made a sharp observation: if the point of the reward model is to guide the policy, you can skip building the reward model and optimise the policy on the preference pairs directly. Same data, one model instead of two, no RL loop to babysit. DPO is now the default for most open-source fine-tuning because it is far easier to make stable. Whether it matches full PPO-based RLHF at the frontier is still argued about, but for most teams the trade is worth it.

The other direction is replacing the human. RLAIF, reinforcement learning from AI feedback, uses a strong model to produce the preference labels. Anthropic's Constitutional AI takes this further by giving the labelling model an explicit written set of principles to judge against, so the critique step is auditable in a way a contractor's gut feel never was. If you want the details of how that works, we covered it separately in constitutional ai harmlessness from ai feedback.

One thing that has not changed: all of these still need some genuine human signal at the root. AI feedback bootstraps off models that were themselves shaped by human preferences. Remove the humans entirely and the whole thing is a snake eating its tail.

Method	Human labels needed	Reward model	Main trade-off
RLHF with PPO	High	Yes	Strong results, hard to train, expensive
DPO	High	No	Simpler and stable, skips the RL loop entirely
RLAIF	Low	Yes	Cheap to scale, inherits the labeller model's biases
Constitutional AI	Low	Yes	Principles are written down and auditable, needs a capable critic model

Why this matters if you are not training models

Most people reading this will never run a PPO loop. You will still deal with RLHF's fingerprints every single day.

When a model hedges instead of committing, that is RLHF. When it caves the moment you push back, that is RLHF. When it gives you 600 words for a yes-or-no question, that is RLHF. Knowing this changes how you prompt: tell the model explicitly that you want disagreement if you are wrong, ask for a short answer and mean it, and stop treating agreement as evidence that you were right. That last habit costs people more than they realise.

If you are evaluating AI vendors or building anything on top of a foundation model, RLHF also explains why two models with similar benchmark scores can feel completely different to use. The base capability may be close. The preference data was not.

Reading about this gets you the concepts. Actually building intuition takes fine-tuning a small model on a preference dataset yourself and watching it go sideways, which is the kind of thing our AI Masterclass sessions are built around, with a mentor to tell you which failure is normal and which one means you broke something. Students and early-career folks who want a longer runway on projects like this usually start with the AI Fellowship instead. Either way, the goal is the same: stop reading about how models are shaped and start seeing it happen.

RLHF is not a solved problem, and nobody serious claims it is. It is the best available answer to a genuinely hard question, which is how you tell a machine what "good" means when you cannot write it down.

FAQs

1. Is RLHF the same as fine-tuning?

No. Fine-tuning usually means supervised training on input-output pairs, where you show the model the correct answer. RLHF trains on relative preferences instead, learning from which of two answers a human liked better, which is useful when there is no single correct answer to demonstrate.

2. Does ChatGPT still use RLHF?

Yes, in some form. OpenAI has used RLHF since InstructGPT and GPT-3.5, and continues to use human preference data along with newer techniques for reasoning models. The exact recipe for current models is not public.

3. How much human feedback data does RLHF need?

Published work has used anywhere from tens of thousands to a few hundred thousand preference comparisons. Data quality and labeller agreement matter more than raw volume, and small, carefully curated preference sets often outperform larger noisy ones.

4. What is reward hacking in RLHF?

Reward hacking is when a model finds behaviour that scores highly with the reward model but is not actually what humans wanted. Common examples in chat models are padding answers to excessive length, over-using confident-sounding language, and agreeing with the user regardless of whether they are right.

5. Can I do RLHF on a small model myself?

Yes. Libraries like Hugging Face TRL support DPO and PPO training on models small enough for a single consumer GPU, and open preference datasets such as Anthropic HH or UltraFeedback are freely available. DPO is the easier starting point because it avoids the instability of an RL loop.

6. Why do RLHF models agree with me even when I am wrong?

Because human labellers tend to prefer responses that validate them, and the model optimises for that preference. This is called sycophancy, and the practical workaround is telling the model upfront to challenge you and to state when it disagrees.

Related reading

• constitutional ai harmlessness from ai feedback
• The AI Learning Process, Explained Step by Step
• AI Learning Rate: What It Is, How to Pick One, and Why Training Fails Without It
• AI Learning: What It Means, and How to Actually Learn AI in 2025
• Is AI Really AI? An Honest Answer to the Question Everyone Is Asking