AI Learning
Constitutional AI: How Anthropic Trained Harmlessness From AI Feedback
A plain-language explanation of Anthropic's Constitutional AI method: the supervised critique-revision stage, the RLAIF stage that replaces human harmlessness labels with model-generated preferences, what sits inside the constitution, where the method breaks down, and how teams building with LLMs can reuse the same self-critique and LLM-as-judge patterns in production.
CSEWhy ·
A plain-language explanation of Anthropic's Constitutional AI method: the supervised critique-revision stage, the RLAIF stage that replaces human harmlessness labels with model-generated preferences, what sits inside the constitution, where the method breaks down, and how teams building with LLMs can reuse the same self-critique and LLM-as-judge patterns in production.
The short answer
Constitutional AI (CAI) is a training method Anthropic published in December 2022, in a paper titled Constitutional AI: Harmlessness from AI Feedback, led by Yuntao Bai. It replaces human labels for harmfulness with feedback the model generates about itself, guided by a short written list of principles called a constitution. The model critiques its own answers against those principles, rewrites them, and is then trained on preferences that another model instance produced, instead of preferences produced by human contractors.
That last part is the whole point. In standard RLHF, if you want a model to stop giving instructions for building a weapon, humans have to sit and rank thousands of responses to red-team prompts, reading the worst of them all day. CAI moves that judgment into the model and moves the human role upstream, into writing the rules.
The two stages, in plain terms
CAI is not one trick. It is a supervised stage followed by a reinforcement learning stage, and they do different jobs.
Stage one starts with a model that has only been trained to be helpful, deliberately not trained to be harmless. You prompt it with red-team questions, it produces something bad, and then you ask it to critique that response using a randomly sampled principle from the constitution. The paper worked with sixteen such principles. The model writes the critique, then rewrites its own answer. Repeat the loop a few times, collect the final revisions, and fine-tune the original pretrained model on them. You now have a model whose default output distribution is already much closer to where you want it, which makes the next stage cheaper and more stable.
Stage two is the part people mean when they say RLAIF, reinforcement learning from AI feedback. Sample two responses to a harmful prompt from the stage-one model. Show both to a feedback model along with a constitutional principle and ask which one is better. The answer comes back as a probability distribution over the two options, not a hard label, and those soft preferences train a preference model. Then you run ordinary RL against that preference model. Adding chain-of-thought reasoning to the feedback step, letting the model think through why one answer is better before choosing, improved the quality of the labels noticeably.
| Stage | What happens | Who provides the signal |
|---|---|---|
| Supervised (SL-CAI) | Model critiques and revises its own harmful responses against sampled principles; fine-tune on the revisions | The model, prompted with the constitution |
| RL (RL-CAI) | Model compares response pairs per principle; preferences train a preference model; RL against it | The model, for harmlessness |
| Helpfulness data | Standard preference comparisons carried over from earlier work | Humans, still |
What people get wrong: humans did not disappear
A common misreading is that CAI removes human feedback from the pipeline. It does not. Harmlessness preferences came from AI feedback. Helpfulness preferences were still human-labelled, and the final system is described as RLHF plus RLAIF, not a human-free process. Anthropic named the combination RLAIF specifically for the harmlessness half.
The interesting result was not that the AI-labelled model was safer. It was that it was safer without becoming useless. Earlier RLHF models trained hard on harmlessness tended to go evasive, answering anything sensitive with a flat refusal and no explanation. CAI models pushed the helpfulness-harmlessness frontier outward: they engaged with the question, said what they objected to and why, and refused less bluntly. Anyone who has argued with an over-tuned chatbot knows why that matters.
What is actually in a constitution
It is a document, in English, that you can read. Anthropic published Claude's version in May 2023 and has revised it since. The principles were drawn from a mix of sources, including the UN Universal Declaration of Human Rights, Apple's terms of service, safety rules from DeepMind's Sparrow work, Anthropic's own research, and a set written specifically to consider non-Western perspectives. Individual principles read like plain instructions: choose the response that is least likely to be viewed as harmful by people from a non-Western cultural tradition, or pick the answer that is least dishonest.
The governance argument here is stronger than the technical one. When behaviour is encoded in thousands of unpublished human labels, nobody outside the lab can inspect it. When it sits in a document, you can argue with it. Anthropic tested that idea directly in October 2023 with Collective Constitutional AI, running a public deliberation with around a thousand Americans through the Polis platform and training a model on the resulting constitution. Roughly half the ideas overlapped with the in-house version. The half that did not is the interesting part.
Where it breaks
CAI inherits the judgment of the model doing the judging. If the feedback model is weak, or holds a bias, the training loop amplifies it and nobody sees a human label go wrong along the way. Principles also conflict constantly, and the constitution does not tell the model how to rank them. A short document cannot cover the actual mess of real requests.
And there is the obvious circularity: a model deciding what counts as harmless output is a model grading its own homework, with the lab choosing the rubric. CAI does not solve alignment. It makes one expensive, unpleasant, slow labelling process cheaper, faster and easier to audit. That is a real contribution and a smaller claim than the paper's popularity suggests.
How to use the pattern, even if you never train a model
Most people reading this will never run RLAIF. The technique underneath it is still directly useful, because it is just structured self-critique plus model-graded evaluation, and both work at the prompt and pipeline level.
A support bot that drafts an answer, checks it against five written rules your compliance team owns, and rewrites before sending is stage one of CAI with a different budget. An evaluation harness where a stronger model scores outputs against an explicit rubric, with reasoning before the verdict, is stage two. The lesson from the paper that transfers best: write the rules down as a short document with a real owner, sample from them rather than stuffing all of them into every prompt, and make the judge explain itself before it scores.
The gap for most teams is not understanding this, it is building it and knowing when the judge is lying to you. If that is where you are, a mentor-led AI Masterclass that works through your own use case is more useful than another paper summary, and you can tell us what you are building when you enquire.
Read the original paper if you have an afternoon. It is unusually readable, and the appendices with the actual principles and sample critiques teach more than any explainer, including this one.
FAQs
1. Is Claude still trained with Constitutional AI?
Yes, Constitutional AI remains part of how Anthropic trains Claude, alongside human feedback for helpfulness and additional safety layers added later, such as the constitutional classifiers work published in 2025. The constitution itself has been revised more than once since the first public version in May 2023.
2. What is the difference between RLAIF and RLHF?
In RLHF, humans compare pairs of model responses and those rankings train a reward model. In RLAIF, a language model makes those comparisons instead, guided by written principles, which is far cheaper and lets you regenerate the whole preference dataset when the rules change.
3. Can I apply Constitutional AI to an open-source model?
The supervised stage is very doable: generate responses, have a capable model critique and revise them against your principles, and fine-tune on the revisions with LoRA or full fine-tuning. The RL stage needs a preference model and a working RL setup, so many teams stop after the supervised stage or use DPO on the AI-generated preference pairs instead.
4. Who writes the constitution?
In Anthropic's case, the lab did, drawing on sources like the UN Universal Declaration of Human Rights and Apple's terms of service. Anthropic later tested a public alternative through Collective Constitutional AI in October 2023, where about a thousand Americans deliberated on principles via Polis and a model was trained on the result.
5. Does Constitutional AI make a model safe?
No. It makes harmlessness training cheaper, more transparent and less dependent on humans reading disturbing content, and it reduces evasive refusals. Jailbreaks, principle conflicts and biases inherited from the feedback model all survive it.
6. Where can I read the original paper?
It is on arXiv as paper 2212.08073, published in December 2022 by Yuntao Bai and colleagues at Anthropic. The appendices contain the actual constitutional principles and sample critique-revision chains, which are the most instructive part for practitioners.