Constitutional AI: use principles as a training signal

Can a model help critique training examples without becoming the final authority on whether those examples are correct? Constitutional AI uses written principles to produce feedback, but the feedback still needs checking.

Before you begin: Understand supervised fine-tuning, reward models, and preference optimization.

Constitutional AI explores that idea using a written set of principles, called a constitution, to guide critique and comparison. The name describes a training approach. It does not mean that the model understands laws, possesses moral authority, or becomes safe by reading a policy once.

Follow the original training idea

The 2022 Constitutional AI paper describes a supervised phase in which model-generated responses are critiqued and revised using principles. Revised responses become training data. A later phase uses AI-generated comparisons to train a preference model and guide reinforcement learning.

Keep these signals distinct: a revised demonstration supplies a target response, while a comparison supplies a relative preference. AI feedback reduces some labeling work, but people still choose the principles, data, models, and evaluation process.

This is historical research, not a claim that every current assistant uses that exact pipeline.

Work through a harmless miniature example

Suppose a workshop assistant follows two application principles: preserve source facts, and state when a required detail is missing.

The input record says “Pottery, Saturday, time unannounced.” The draft says “Come on Saturday at 2 pm.”

A useful critique identifies the precise unsupported field: 2 pm does not appear in the evidence. A revision says that pottery is on Saturday and that the start time is unannounced. The change is testable against the record.

This exercise borrows the idea of principle-guided revision. It is not a reproduction of the paper's harmlessness training. The important habit is to tie criticism to a concrete defect rather than asking for a vaguely “better” answer.

Turn principles into observable criteria

A principle such as “be helpful” leaves many decisions open. More specific criteria can distinguish useful detail from invented detail, a necessary refusal from an unnecessary one, and an honest limitation from evasion.

Principles can also conflict. “Be concise” may conflict with explaining an important exception. A rubric should say how to prioritize those goals in the task. Collect examples where reviewers disagree and inspect what the system learns from them.

Do not assume a longer constitution is automatically better. Redundant or inconsistent rules can make judgments harder to interpret. The chosen principles reflect human decisions even when a model produces the individual labels.

Make two principles disagree

Give a critic these priorities: preserve every material condition, then keep the answer concise. The source says that members attend free and guests pay a fee. Candidate A says, “Workshops are free.” Candidate B says, “Members attend free; guests pay a fee.”

Ask which candidate a brevity-only judge might prefer, then which the stated priority requires. Swap the candidates' display order and judge again. If the preference changes, inspect whether position influenced the result rather than the principle.

Check the feedback model itself

A model critic can miss the same mistake as the generator. A preference judge may favor its own writing style or reward confidence. If generated feedback is the only evaluation, shared errors can look like improvement.

Keep independent source-grounded checks and human-reviewed cases. Evaluate both harmful compliance and over-refusal of harmless requests. Measure how behavior changes across languages and task types that matter to the application.

A successful training result still needs runtime permissions, careful tool design, and data boundaries. Training an assistant to respect private records does not justify removing the database's ownership checks.

Practice a critique

Draft: “All workshops are free.” Evidence: “Members attend free; guests pay the listed fee.” Write a critique and a corrected answer.

To assess a principle-guided training experiment, preserve the principles, generated examples, filtering rules, model revisions, and held-out evaluations. Report gains and regressions separately. Agreement with the feedback model is evidence about that model's judgment, not universal correctness.

The original Constitutional AI paper describes the supervised and reinforcement-learning phases. InstructGPT offers a useful comparison where human demonstrations and preferences play a different role.

Continue: Research paper roadmap, where you can connect these ideas and plan a focused reading sequence.