Can a model help critique training examples without becoming the final authority on whether those examples are correct? Constitutional AI uses written principles to produce feedback, but the feedback still needs checking.
Before you begin: Understand supervised fine-tuning, reward models, and preference optimization.
Constitutional AI explores that idea using a written set of principles, called a constitution, to guide critique and comparison. The name describes a training approach. It does not mean that the model understands laws, possesses moral authority, or becomes safe by reading a policy once.
Follow the original training idea
The 2022 Constitutional AI paper describes a supervised phase in which model-generated responses are critiqued and revised using principles. Revised responses become training data. A later phase uses AI-generated comparisons to train a preference model and guide reinforcement learning.
Keep these signals distinct: a revised demonstration supplies a target response, while a comparison supplies a relative preference. AI feedback reduces some labeling work, but people still choose the principles, data, models, and evaluation process.
This is historical research, not a claim that every current assistant uses that exact pipeline.
Work through a harmless miniature example
Suppose a workshop assistant follows two application principles: preserve source facts, and state when a required detail is missing.
The input record says “Pottery, Saturday, time unannounced.” The draft says “Come on Saturday at 2 pm.”
A useful critique identifies the precise unsupported field: 2 pm does not appear in the evidence. A revision says that pottery is on Saturday and that the start time is unannounced. The change is testable against the record.
This exercise borrows the idea of principle-guided revision. It is not a reproduction of the paper's harmlessness training. The important habit is to tie criticism to a concrete defect rather than asking for a vaguely “better” answer.
Which principle governs when two instructions conflict?
Written principles can guide critique and revision, but they still need interpretation in a task. “Be helpful” can conflict with a privacy boundary when a requested detail belongs to another person. State how those principles should be applied to the actual case instead of assuming their names settle the decision.
Compare the initial answer with a revision that respects the controlling boundary while offering an allowed alternative. The challenge asks you to explain the conflict and evaluate the resulting behavior. Principles and model critique remain parts of a broader system of access controls and task evaluation.
A volunteer asks for another member's private contact details.
Apply the privacy constraint while looking for an authorized way to help with the underlying task.
The answer can protect the data and offer a permitted contact route or escalation.
Turn principles into observable criteria
A principle such as “be helpful” leaves many decisions open. More specific criteria can distinguish useful detail from invented detail, a necessary refusal from an unnecessary one, and an honest limitation from evasion.
Principles can also conflict. “Be concise” may conflict with explaining an important exception. A rubric should say how to prioritize those goals in the task. Collect examples where reviewers disagree and inspect what the system learns from them.
Do not assume a longer constitution is automatically better. Redundant or inconsistent rules can make judgments harder to interpret. The chosen principles reflect human decisions even when a model produces the individual labels.
Make two principles disagree
Give a critic these priorities: preserve every material condition, then keep the answer concise. The source says that members attend free and guests pay a fee. Candidate A says, “Workshops are free.” Candidate B says, “Members attend free; guests pay a fee.”
Ask which candidate a brevity-only judge might prefer, then which the stated priority requires. Swap the candidates' display order and judge again. If the preference changes, inspect whether position influenced the result rather than the principle.
Find the defect in the feedback
B retains the membership condition. A's shorter length does not compensate for its changed meaning under the stated priorities. A judge that rewards A has produced a defective training signal for this task. Repeating that label across synthetic data can teach the generator to remove important conditions.
One order-swap test does not measure every judge bias. It gives you a controlled probe: the evidence and candidates stay fixed while presentation changes. Preserve disagreements for review instead of treating agreement with a single critic as independent proof of quality.
Check the feedback model itself
A model critic can miss the same mistake as the generator. A preference judge may favor its own writing style or reward confidence. If generated feedback is the only evaluation, shared errors can look like improvement.
Keep independent source-grounded checks and human-reviewed cases. Evaluate both harmful compliance and over-refusal of harmless requests. Measure how behavior changes across languages and task types that matter to the application.
A successful training result still needs runtime permissions, careful tool design, and data boundaries. Training an assistant to respect private records does not justify removing the database's ownership checks.
Practice a critique
Draft: “All workshops are free.” Evidence: “Members attend free; guests pay the listed fee.” Write a critique and a corrected answer.
Compare your revision
The draft removes the membership condition. A corrected answer preserves that members attend free while guests pay the listed fee. If the actual fee is absent, the assistant should not invent it. “Be safer” is a less useful critique than identifying the lost condition.
To assess a principle-guided training experiment, preserve the principles, generated examples, filtering rules, model revisions, and held-out evaluations. Report gains and regressions separately. Agreement with the feedback model is evidence about that model's judgment, not universal correctness.
The original Constitutional AI paper describes the supervised and reinforcement-learning phases. InstructGPT offers a useful comparison where human demonstrations and preferences play a different role.
Practice with feedback
Make two principles disagree
Principle A: "be maximally helpful and complete". Principle B: "never share a member's contact details". A volunteer asks for a member's phone number to arrange a lift.
Turn principles into observable criteria and see what happens when they conflict.
Check your understanding
Your task
Write two principles with observable criteria, resolve their conflict, and audit the checker.
These notes stay on this page. Download them before leaving.
What to include
- Criteria are mechanically checkable
- The conflict case has a concrete resolved response
- Precedence is stated as a rule, not case by case
- The critique audit has a sample size and a correction path
Compare with a worked answer
Here is one way to answer. Check how it uses the information in the task.
Principle A: be complete and useful. Observable criteria: the response either answers the question, or names exactly what it cannot do and offers a concrete next step. A response that refuses without an alternative fails. Pass: 'I cannot share phone numbers. I can message them on your behalf.' Fail: 'I cannot help with that.' Principle B: never disclose member contact details. Observable criteria: the response contains no string matching a phone, email, or postal address pattern, and no partial form of one. Pass: 'I cannot share contact details.' Fail: 'Their number ends in 4471, ask Priya for the rest.' - which passes a naive regex and violates the principle, which is why the criteria say 'no partial form'.
Precedence: B over A, always. Where they conflict, satisfy B fully and then A as far as B allows. The resolved response: 'I cannot share a member's phone number. I can send them a message asking whether they would like a lift, and pass on their reply if they agree.' That contains no contact detail and still gives the volunteer a route to the outcome they wanted, which is what A asks for.
Critique audit sample size: 100 critiques a week, stratified so half are cases the critic marked as violations and half it passed. Sampling only the flagged ones measures precision and never recall. what I compare: my own judgement against the written criteria, not against my instinct. If I disagree with the critic and cannot point to a criterion, the criterion is the thing that is unclear. what I do when the critic is wrong: if it is a criteria problem, rewrite the criterion and re-run the sample. If the criterion is clear and the critic misapplies it, that case goes into the critic's own evaluation set. Our first audit found the critic passing 'their number ends in 4471' every time, which is exactly the failure that would have been trained into the policy.
When you are signed in, opening the challenge carries your edited working notes into its draft in this browser. The challenge has its own completion record. Practising here does not award points or mark it complete.
, where you can connect these ideas and plan a focused reading sequence.