RLAIF and Constitutional AI

RLAIF (reinforcement learning from AI feedback) is RLHF with the human annotator replaced by a language model. Instead of paying people to compare two responses, you prompt a capable model to compare them, turn its answer into a preference label, and run the usual pipeline: train a reward model on the labels, then optimise the policy with PPO or a simpler online method, or feed the pairs straight into DPO. Constitutional AI (CAI) is the method that introduced the term: the AI judge is steered by a short written list of principles, the "constitution", and a supervised critique-and-revise stage comes before the RL stage. In the post-training stack RLAIF is a data-generation strategy more than a new loss. The objective is the same Bradley-Terry likelihood and the same KL-regularised RL; what changes is where the labels come from, how much they cost, and which biases they carry.

The problem and where it came from#

Human preference data is slow, expensive and hard to steer. If you want a model to refuse dangerous requests without becoming evasive on everything else, you have to write guidelines, train annotators on them, audit their agreement and repeat whenever the guidelines change. Harmlessness data has a further cost: annotators have to read a lot of harmful content.

Bai et al. at Anthropic published Constitutional AI: Harmlessness from AI Feedback in December 2022. They kept human labels for helpfulness but generated all harmlessness labels with a model guided by natural-language principles, and reported a better harmlessness-helpfulness trade-off than their earlier RLHF models trained on human harmlessness labels, with far less evasive refusing. The appeal was both practical and conceptual: changing the model's behaviour becomes a matter of editing a document rather than relabelling a dataset.

Lee et al. at Google followed in 2023 with RLAIF vs. RLHF, a controlled comparison on summarisation, helpful dialogue and harmless dialogue. Human raters judged RLAIF policies comparable to RLHF policies, RLAIF still improved the policy when the AI labeler was the same size as the policy, and a "direct" variant that skipped the reward model and queried the labeler for a score during RL did better still. In parallel, Zheng et al. (2023) measured how well strong LLM judges agree with humans on open-ended chat (over 80% agreement, comparable to agreement between humans) and documented the biases that come with them: position, verbosity and self-preference. Yuan et al. (2024) pushed the idea to its limit with self-rewarding language models, in which the policy judges its own samples and is trained with iterative DPO on the resulting pairs.

How Constitutional AI works#

Stage 1: critique and revision (SL-CAI)

Start from a helpful model that has not been trained to be harmless. For each red-team prompt, sample a response. Then prompt the same model to critique its response against a principle drawn at random from the constitution (for example, "identify ways the response is harmful, unethical or dangerous"), and to rewrite the response so it addresses the critique. The critique-revision step can be repeated several times with different principles. The final revisions, paired with their prompts, become supervised fine-tuning data, mixed with helpfulness data so the model does not drift toward refusal. This stage moves the policy's distribution before RL so the RL stage starts from a better place and needs less exploration.

Stage 2: AI preference labels (RL-CAI)

Sample two responses per prompt from the SL-CAI model. Show both to a feedback model with a multiple-choice question built from one principle: "Which of these responses is less harmful? (A) ... (B) ...". Read the probability the model assigns to the tokens "A" and "B" and normalise them to a soft label. Train a preference model on a mix of these AI harmlessness labels and human helpfulness labels, then run RL against it. Bai et al. also tested chain-of-thought labels, letting the feedback model reason before answering. Reasoning made the labels more accurate but pushed the probabilities toward 0 or 1, so they clamped them to the 40% to 60% range to keep the preference model from becoming overconfident.

The objective, and what AI labels change#

From judge log-probabilities to a label

Let $J$ be the judge model, $c_k$ the $k$-th principle, and $\text{prompt}(x,y_1,y_2,c_k)$ the comparison prompt with $y_1$ in position A. The judge's preference probability for the first-shown response is

Because judges prefer whichever response appears first (or second) by some amount, a standard fix is to query both orders and average:

and optionally to average over several principles $c_k$ as an ensemble. The reward model is then trained with the Bradley-Terry cross-entropy against the soft target rather than a hard label:

When $\tilde{p}$ is 0 or 1 this is the ordinary reward-model loss. With soft targets the minimiser for each pair is $\sigma (\Delta r_{\varphi })=\tilde{p}$, so the reward model inherits the judge's confidence, not just its ranking. That is the reason for the 40% to 60% clamp on chain-of-thought labels: an overconfident judge produces an RM with inflated margins, which RL then exploits.

The RL objective is unchanged

Policy training uses the same KL-regularised objective as RLHF,

with ${\pi }_{\text{ref}}$ the SL-CAI model. In direct RLAIF (Lee et al.) the learned $r_{\varphi }$ is replaced by the judge itself: the judge rates each sampled response on a 1-to-10 scale, the expected score under the judge's distribution over the score tokens is normalised, and that is the reward. This removes the reward model and its staleness, at the cost of a judge inference call for every rollout.

Where it sits relative to its siblings

Worked example: a label by hand#

A judge sees response A first and response B second, and assigns log-probability -0.4 to the token "A" and -1.3 to "B". Normalising, $p=e^{-0.4}/(e^{-0.4}+e^{-1.3})=0.670/(0.670+0.273)=0.711$. Now swap the order. With B shown first, the judge gives -0.7 to "A" (which now means B) and -0.9 to "B" (which now means the original A). The probability that the original A is better is $e^{-0.9}/(e^{-0.7}+e^{-0.9})=0.450$. The judge changed its mind when the order changed, a sign that part of the 0.711 was position preference. The averaged label is $(0.711+0.450)/2=0.581$: a weak preference for A, which is what the reward model should learn. Used as a soft target, the per-pair loss is minimised at a margin of $\mathrm{logit}(0.581)=0.33$, compared with an unbounded push toward infinity under a hard label.

Writing the constitution#

A principle is useful to a judge only if it decides between two concrete responses. "Be ethical" does not; "choose the response that declines to give operational detail for causing physical harm while still addressing the user's underlying need" does. Write principles as comparative instructions ("choose the response that..."), each about one property, and phrase them so either response could win. Principles that always favour the shorter or more cautious answer bake refusal into the reward model.

Sample one principle per comparison rather than concatenating all of them into a single prompt. Long lists dilute the judge's attention, and sampling gives the reward model a mixture of perspectives that behaves like an ensemble. Keep a version history of the constitution and record which version produced each label, because editing a principle changes the data distribution as surely as hiring new annotators would. When a principle produces labels that disagree with your human calibration set more than the others do, rewrite it or retire it. Finally, include principles for the behaviour you want to keep, not just the behaviour you want to remove: helpfulness, directness and honesty about uncertainty need to be in the document if the judge is going to protect them.

The pipeline step by step#

inputs: policy init π_0 (SFT or helpful-only model), prompt set X, constitution C = {c_1..c_K}, judge J
# optional stage 1: SL-CAI
for x in red-team prompts:
    y = π_0(x)
    for round in 1..R:                       # R = 1 to 4
        c = sample(C)
        critique = π_0("critique y under c")
        y = π_0("revise y to address critique")
    D_sl += (x, y)
π_sl = SFT(π_0, D_sl ∪ helpfulness data)
# stage 2: AI preference labels
for x in X:
    y1, y2 = π_sl(x), π_sl(x)                # temperature around 1.0
    c = sample(C)
    p = 0.5 * (P_J(A | x, y1, y2, c) + 1 - P_J(A | x, y2, y1, c))
    if chain-of-thought: p = clip(p, 0.4, 0.6)
    D_pref += (x, y1, y2, p)
r_φ = train_RM(D_pref ∪ human helpfulness pairs, soft BT loss)
π = PPO / RLOO / GRPO(π_sl, r_φ, β)          # or DPO on D_pref directly

Data sizes. The constitution is short: a few to a few dozen principles of one or two sentences. Prompt sets for labelling typically run 20k to 500k; because labels are cheap, teams generate more pairs than they would ever collect from humans and filter them. A few hundred to a few thousand human-labelled pairs remain essential as a calibration and test set for the judge.

The judge prompt. State the task, the principle, both responses with clear delimiters, and an answer format that ends on a single token you can read probabilities from. Keep the format identical across all calls. If you ask for reasoning first, parse the final answer and clamp or temperature-scale the probabilities.

Choices that matter#

Validate the judge before you validate anything else. Label 300 to 1,000 pairs with humans, run the judge on them, and measure agreement, position consistency (fraction of pairs where both orders agree) and correlation of the judge's preference with response length. These three numbers tell you whether the judge is fit to train on.

Compute and cost#

Labelling is inference, so cost scales with judge size times tokens. A pairwise judge prompt with a 300-token prompt, two 500-token responses and a 200-token instruction is about 1,500 tokens; querying both orders makes it 3,000. For 100k pairs that is $3\times 10^8$ tokens of prefill, plus output tokens if you allow reasoning. Using the standard estimate of $2N$ FLOPs per token for an $N$-parameter model, a 70B judge needs about $2\times 7\times 10^{10}\times 3\times 10^8=4.2\times 10^{19}$ FLOPs, which is tens of GPU-hours on current accelerators even at realistic utilisation. Chain-of-thought labelling adds decode time, which is slower per token than prefill. Compared with human labelling of 100k pairs, which takes weeks of annotator time, this is cheap and fast, and that cost gap is the entire argument for RLAIF.

The training side is identical to RLHF or DPO. Direct RLAIF moves the judge into the RL loop: every rollout costs a judge call, so a run with 500k rollouts adds 500k judge inferences, and the judge must be served alongside the policy, reference and (if used) value model.

A toy run: agreement hides systematic bias#

We simulated the full labelling chain to see what a judge's biases do to a reward model. Responses are 12-dimensional feature vectors with a true utility $u=w^{\top }x$ ($\Vert w\Vert =1.5$) and an independent "length" feature $L$ that has zero weight in the true utility. Human labels are sampled from Bradley-Terry on $u$. The judge's logit for "first response is better" is $\Delta u+0.6+0.8\Delta L+\text{noise}(\sigma =0.6)$: an honest view of quality, plus a position bias of 0.6 toward the first response, plus a length preference. We generated 20,000 pairs, measured agreement, then trained linear Bradley-Terry reward models on 5,000 pairs from each label source and tested them against the true ranking on 20,000 fresh pairs (seed 0).

On the usual validation metric the judge looks excellent: it agrees with human labels 72% to 74% of the time, more often than a second human labeller does (70%). Swapping order removed the position bias almost completely (first-response win rate from 0.600 to 0.506), and 20% of pairs flipped their verdict when the order changed.

The reward models tell a different story. An RM trained on 5,000 human labels reaches 0.978 agreement with the true ranking despite the labels' 21% noise, because that noise is random and averages out. An RM trained on 5,000 judge labels stops at about 0.85, and no amount of debiasing the order helped, because the judge's error is systematic. Each judge-trained RM learned a length weight of 0.51 to 0.54 relative to its quality weights, against the judge's own ratio of $0.8/1.5=0.533$. The reward model faithfully copies whatever the judge prefers, including what it should not. Soft labels did not change the bias; they kept the RM's scale tied to the judge's confidence (quality-weight norm 1.32 versus 4.40 for hard labels), which matters later for setting $\beta$. The practical lesson: a judge can beat human-human agreement and still be unfit to train on. Test it with contrast pairs that isolate known biases, not just with agreement.

Failure modes and how to detect them#

Inherited judge bias. Length, list formatting, confident tone and the judge's own phrasing style are the common ones. Detect with controlled contrast pairs (same content with and without padding or bullets) and by regressing judge preference on length difference across your labelled set. Mitigate with explicit rubric instructions, length-matched pair sampling, or a length term in the reward.

Position bias. First-shown win rate far from 50% on randomly ordered pairs, and a high order-inconsistency rate. Always query both orders; drop pairs where the two orders disagree strongly if you need clean labels.

Self-preference. Judges rate text from their own model family higher. Watch for it when the judge and policy share a base model, as in self-rewarding setups: judge win rates rise faster than independent-judge or human win rates.

Over-refusal or over-compliance from the constitution. Principles written only about harm teach refusal; Bai et al. paired harmlessness labels with human helpfulness data for this reason. Track refusal rate on a benign prompt set alongside harmfulness on a red-team set.

Overconfident labels. A histogram of $\tilde{p}$ piled at 0 and 1, typical of chain-of-thought labelling. Clamp or temperature-scale, then check RM calibration on human-labelled pairs.

Reward hacking the judge. In direct RLAIF the policy optimises the judge itself. Signs: judge scores climb toward the top of the scale while independent evaluation flattens. Hold out a second judge (different model family) for evaluation only.

When to use it#

Minimal implementation sketch#

Judge labelling with Hugging Face Transformers, reading token probabilities in both orders. The resulting soft labels go into the reward-model trainer from the reward modeling article with a soft-target loss.

import torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("path/to/judge")
judge = AutoModelForCausalLM.from_pretrained("path/to/judge", torch_dtype=torch.bfloat16, device_map="auto")
A_ID = tok.encode(" A", add_special_tokens=False)[-1]
B_ID = tok.encode(" B", add_special_tokens=False)[-1]

TEMPLATE = ("Consider this principle: {principle}\n\nPrompt: {x}\n\n"
            "Response (A):\n{a}\n\nResponse (B):\n{b}\n\n"
            "Which response better follows the principle? Answer with A or B.\nAnswer:")

@torch.no_grad()
def p_first(x, a, b, principle):
    ids = tok(TEMPLATE.format(principle=principle, x=x, a=a, b=b), return_tensors="pt").to(judge.device)
    logits = judge(**ids).logits[0, -1, [A_ID, B_ID]].float()
    return F.softmax(logits, dim=-1)[0].item()        # P(A) renormalised over {A, B}

def label(x, y1, y2, principle):
    p12, p21 = p_first(x, y1, y2, principle), p_first(x, y2, y1, principle)
    consistent = (p12 > 0.5) == (p21 < 0.5)          # same verdict in both orders
    return 0.5 * (p12 + 1 - p21), consistent

def soft_bt_loss(r1, r2, p):                          # p: soft target that y1 is preferred
    return F.binary_cross_entropy_with_logits(r1 - r2, p)

For production use, batch the judge calls on an inference engine and cache each (prompt, order) result. In POFE, AI-feedback labelling is a pipeline step whose report includes the judge's agreement, position consistency and length correlation on a human-labelled calibration set, so the bias check happens before any reward model is trained.

Further reading#

Bai et al. (2022) for Constitutional AI, including the critique-revision procedure and chain-of-thought label clamping. Lee et al. (2023) for the controlled RLAIF versus RLHF comparison and direct RLAIF. Zheng et al. (2023) for the measured agreement and biases of LLM judges. Yuan et al. (2024) for self-rewarding language models. Bai et al. (2022, helpful and harmless) for the human-feedback baseline CAI was compared against.

Frequently asked questions#

References#

1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arxiv.org/abs/2212.08073
2. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, et al. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arxiv.org/abs/2309.00267
3. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arxiv.org/abs/2306.05685
4. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, Jason Weston (2024). Self-Rewarding Language Models. arxiv.org/abs/2401.10020
5. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arxiv.org/abs/2204.05862

Related reading