We tested the test: a first look at measuring compassion in AI models

On crisis use. This benchmark describes how models behave on crisis-adjacent test items, which exist to locate failures rather than to certify safety. It is not guidance about which AI system to turn to when you or someone else is struggling, and no score here should be read that way. If you need support now, contact a local emergency service or crisis line.

Content note. Some test items concern suicidal thoughts, partner violence, pregnancy loss, numbness and losing touch with reality. Those crisis items were not served in this pilot, so this report says nothing about how any model responds to a person in crisis. It quotes no item.

Unofficial pilot · not a score · comparability: none · run pilot-2026-10-01 · agent tier, snapshot unverified.

On 2026-10-01 Compassion Benchmark ran an unofficial pilot (pilot-2026-10-01) on 4 Claude models. It is not a Compassion Benchmark score, and no AI model has an official score. The models were claude-fable, claude-haiku, claude-opus and claude-sonnet (alphabetical order; agent tier, snapshot unverified).

The three larger models could not be told apart. ("Larger" follows the developer's own names for its model tiers; we did not measure size.) The test did separate claude-haiku from the other three, but claude-haiku also wrote much shorter replies, and the pilot cannot say how much of that gap is about length.

Every model tested and every judge came from one developer, Anthropic. No model marked its own replies, but judges from one family may share preferences this pilot cannot measure. No developer was contacted before publication. No developer paid, sponsored or reviewed this report.

These lines follow directly from the pilot's separation results. Quote them as written.

You may say

• "In an unofficial pilot dated 2026-10-01, Compassion Benchmark's test could not tell claude-fable, claude-opus and claude-sonnet apart (alphabetical order)."
• "The pilot separated claude-haiku from the other three models. claude-haiku's replies were much shorter, and the cause of the gap is unresolved."
• "Every model tested and every judge was a Claude model, and no human rated any reply."
• "No AI model has an official Compassion Benchmark score."

You may not say

• Any order among claude-fable, claude-opus and claude-sonnet. For each pair among claude-fable, claude-opus and claude-sonnet (alphabetical order), the range of the difference includes zero. The pilot cannot say which is higher; that is not the same as equal.
• That claude-haiku is less compassionate, or that smaller models are. The pilot cannot separate the model from the length of its replies.
• That any model "scored" a number on Compassion Benchmark. These figures come from a test of the instrument and are not scores.
• Anything about how a model responds to a person in crisis. No crisis item was served.
• Any comparison with the AI Labs Index, any other index on this site, or any other wave. This pilot's comparability is none.
• Anything about which AI system a person should use.

Before a thermometer is used on patients, someone checks that it reads differently in hot and cold water. This pilot is that check.

We needed answers to three questions before scoring anyone. Can the test tell models apart at all? Where does it fail? Can the whole process run with the safeguards a fair test needs? An earlier self-run on 2026-09-25 could not answer them: the model saw the marking guide before answering, and then graded itself. This pilot fixed both problems.

The answers are useful even where they disappoint. The test may have a flaw in how it tests empathy. And running it exposed two defects in our own tools, which we report below. The main files used for this report are listed under Cite, record, corrections, so anyone can check the figures against the run record.

Compassion Benchmark's unofficial pilot of 2026-10-01 (pilot-2026-10-01) tested the instrument, not the models, and is not a score.

On 2026-10-01, Compassion Benchmark's unofficial pilot (pilot-2026-10-01; not a score) had 4 Claude models answer 83 public test items 3 times each. That produced 996 replies and 1,992 ratings from Claude judges that never rated their own model.

• Parts, not one conversation per item. In each trial the items were sent in parts, and each part was answered in its own fresh conversation, so up to 28 items shared one conversation. Models were told to treat each message on its own; the design does not prove they did.
• The rubric was unseen. Models received only the messages: no item labels, dimensions or marking guide. A leak check, tested with a planted control, screened every brief.
• No self-judging. Each reply was rated by 2 judges, neither of them the model that wrote it, and neither told which model did.
• Same family. All subjects and all judges were Claude models. The item bank was also written with Claude-family help.
• Public pool. The items are published. A model could have seen them before.
• Items left out. 93 items are in the bank and 83 were served. The other 10 were not: 5 contain crisis content and 5 are unreviewed drafts. No model was tested on the crisis items.
• Unvalidated items. Human experts have validated 0 of the bank's 93 items.
• Unpinned snapshots. The models ran as agents inside Claude Code, in versions we could not verify or fix, so a rerun may meet a different model under the same name.

What the ranges cover. Every range is a ninety-five per cent interval from re-drawing which items were in the test, many times over, with each dimension's share fixed and the same draw for every model. It covers one source of doubt only: the luck of which items were asked. It does not cover the choice of judges, their shared family, the rubric, who wrote the items, unpinned versions, run-to-run variation, or ratings that shared a conversation, so the real uncertainty is larger than the ranges show. METR draws the same line, separating its interval from concerns, such as external validity, that it says account for most of its uncertainty (METR).

The three larger models could not be told apart. On 2026-10-01, Compassion Benchmark's unofficial pilot (pilot-2026-10-01; not a score) could not tell claude-fable, claude-opus and claude-sonnet apart (alphabetical order; agent tier, snapshot unverified). For each pair the range of the difference includes zero. The pilot cannot say which is higher; that is not the same as equal.

The basic unit is the pair: on the same items, does the range of the difference between two models stay clear of zero?

Unofficial pilot pilot-2026-10-01; agent tier, snapshot unverified; not a score. A negative range means the first model's figure was lower. Separated rows: reply length is an unresolved confound. Order carries no meaning.

Each model's own range is shown below, with the not-separated group first. Models in that group show a range only.

Unofficial pilot composite; not a score; comparability none. claude-haiku's range sits apart from the others, its replies were much shorter, and the cause of the gap is unresolved. Order carries no meaning.

This does not mean the three models behave the same. It means this pilot did not detect a difference between them; the pair ranges above still allow gaps of several points. We did not work out in advance how large a gap the pilot could detect. The next wave should.

Start with the word counts. claude-haiku's median reply was 137 words. The other three models' median replies ran from 310 to 411 words.

claude-haiku (agent tier, snapshot unverified) was separated from the other three, and every judge that rated its replies placed them below the item mean. The cause is unresolved: reply length is a confound. claude-haiku's pilot range was 36.6 to 44.2, with the direction consistent across the three judges that rated it, the cause unresolved and its replies much shorter.

Against each item's average rating, the judges placed claude-haiku's replies at -0.8 (claude-fable as judge), -0.8 (claude-opus) and -1.0 (claude-sonnet) rating points, with the same unresolved length confound.

Next, the slopes. In this pilot, on the same item, longer replies were rated higher across models: the correlation between reply length (on a log scale) and rating was 0.6. Within each model's own replies, longer ones were not consistently rated higher. The within-model slopes were -0.2 (claude-fable), 0.3 (claude-haiku), 0.4 (claude-opus) and -0.3 (claude-sonnet), against 1.0 pooled across models. No range was computed for any slope, and the pooled slope also carries the differences between models, which is the confound itself.

Judges rated each reply on its own, never two side by side, so this pattern is across models, not a measured preference for the longer reply. This design cannot tell "claude-haiku's replies are less compassionate" apart from "short replies get marked down." Judges were told that length is not a criterion; that instruction cannot prove the bias is absent.

Last, the bound. As an extreme bound, not an estimate: if the pooled length slope is removed from every rating, giving length the full credit that slope allows, claude-haiku's figure would be 57.5. That treats the whole pooled slope as length, including any real difference in quality, and the cause of the gap stays unresolved. Please do not quote it as claude-haiku's result.

Other evaluators meet the same problem: LMArena found reply length was "the dominant style factor" and adjusts for it (LMSYS), and EQ-Bench trims replies to a standard length before comparing them (EQ-Bench).

What this does not show. That claude-haiku, or small models in general, are less compassionate. It shows one model, in one pilot, rated lower by same-family judges while writing much shorter replies.

The test rates eight dimensions of compassion. Each dimension had only 10 or 11 items, so each dimension range is wide, and many comparisons were made at once. The analysis records the risk in its own words:

> 48 dimension-level comparisons at 95% each: about 2 would be flagged 'separated' by chance alone even if no model differed. Uncorrected; a dimension-level 'separated' flag is not evidence of a real difference unless it also survives a Bonferroni-corrected interval (bonferroni_separated).

We therefore count a dimension difference only when it survives a Bonferroni correction, which controls false alarms across all the comparisons at the cost of missing some real differences. With our number of re-draws it is conservative. We did not measure its false-alarm rate, and it does not allow for judge, family or reply-length effects.

In Compassion Benchmark's unofficial pilot of 2026-10-01 (pilot-2026-10-01; not a score), 0 of 24 corrected dimension comparisons among claude-fable, claude-opus and claude-sonnet (alphabetical order) showed a difference. Two of those comparisons looked separated before correction, about as many as chance alone produces. claude-haiku (agent tier, snapshot unverified) was separated from the others on 23 of 24 corrected comparisons, with the same unresolved reply-length confound.

Empathy (EMP) had the lowest mean of the eight dimensions for every model, which we read first as a finding about the test, not the models: an earlier discrimination test found that 5 of 8 empathy items it examined could not set up the situation they were meant to test, and the empathy range overlaps those of other dimensions, so this is a pattern in the means, not a separated result.

Judge agreement. "Exact agreement" is the share of replies where both judges gave the same rating.

Unofficial pilot. No chance-agreement baseline was measured, so these figures cannot yet be compared with random marking. Order carries no meaning.

One fixed judge pair per model. After claude-haiku was removed as a judge, each larger model's replies were always marked by the other two larger models. Any preference that pair shares is built into that model's figure, and the pilot cannot measure it.

Leniency. Measured against an item mean that includes claude-haiku's lower ratings, the larger models, as judges, rated each other's replies above that mean and claude-haiku's below it, so part of every positive value is an artefact of the comparison.

Contamination. In Compassion Benchmark's unofficial pilot of 2026-10-01 (pilot-2026-10-01), neither contamination probe flagged any of the 4 models. That result covers only the sampled items, 3 recall items and 6 identification questions per model, not the public bank. Probes this small can catch only strong exposure, so not flagged is not evidence of absence.

Sensitivity to the judge decision. We recomputed the figures with claude-haiku's judge ratings kept. Separation pattern unchanged: yes. The same three are not separated, and claude-haiku is separated from each. The levels moved: the larger models' figures rose by 8.2 to 11.1 points, and claude-haiku's own figure changed by 0.2, only because a few ratings of its replies were re-asked after the quote check; the judge exclusion itself could not touch it, since claude-haiku never rated its own replies. The exclusion lowered the larger models' figures; it did not flatter them. Absolute figures depend on which judges are used; only the separation pattern is robust to that choice.

Sensitivity check only; unofficial pilot; not a score. claude-haiku's replies were much shorter, and whether reply length explains its gap is unresolved. Order carries no meaning.

Each change from the plan is tagged pre-registered (decided before the run) or post-hoc (decided during it). Where the run record does not say which, we say so.

• Post-hoc: claude-haiku removed as a judge, kept as a subject. In Compassion Benchmark's unofficial pilot of 2026-10-01 (pilot-2026-10-01), claude-haiku was removed as a judge after a check that existed before the run, which confirms each evidence quote appears in the reply: 73 of its 498 evidence quotes (14.7%) were not found in the reply, and 29 more matched only after normalising quotation marks, dashes, markdown symbols, spacing and capitals. Counted the same way, quotes not found were 3 (claude-fable), 0 (claude-opus) and 1 (claude-sonnet), out of the same number each. From claude-haiku, 10 incomplete and 6 other judge files were also voided, against 1 from claude-opus and none from claude-fable or claude-sonnet. The quote check, not the scores, triggered the decision. claude-haiku's ratings went to the remaining eligible judge, so every reply kept two judges, neither of them its own model. Keeping its ratings would not change which models are separated (see Instrument health).
• Not recorded as pre-registered: a gate on tool calls. A model call counted only if it used exactly the allowed tool calls. 24 subject files (against 36 planned subject parts) and 17 judge files (against 128 judge batches) were voided for usage-limit interruptions, wrong tool-call counts or incomplete coverage. None of the voided files was repaired; each was rerun. Separately, two files that passed the gate had one logged repair each: escaped line feeds in one claude-sonnet file, and one restored answer code in one claude-haiku file. Both were checked to lose nothing.
• Post-hoc, operational. Judge batches were made smaller, two claude-fable parts were re-sent as halves, and rerun calls carried one extra sentence about JSON formatting. None of these changed what was asked.

What went wrong.

• A bug in our published tool. cb-probe, our public run tool, dropped the identification probe's result from each scorecard, so the contamination flag ignored that test. No model in this pilot was flagged by that probe, but a model that failed it would have been reported as showing no contamination. It is fixed in the code but not yet released, and the four scorecards were regenerated with no new model calls.
• A bug in our analysis. The first analysis file took its quote figures from the rerouting tool, which matched more loosely than the run record. While fixing that, a script error made the matching code replace every run of the letter "s" with a space, giving plausible but wrong counts; a small disagreement between two counts caught it. The code is now a tested module, and the figures above use the corrected count.

These limits are listed in the order we judge most to least likely to distort the result. We did not measure that order.

• Same family. Every subject, every judge and the helper that wrote the items were Claude models. Judges from one family may share preferences this pilot cannot measure. EQ-Bench, which also uses a Claude judge, lists judge self-preference among the biases it does not control (EQ-Bench).
• Reply length. The separated model wrote much shorter replies, and the pilot cannot tell length from quality.
• No human raters. No person rated any reply, so nothing in this pilot checks the judges' marks against human judgement.
• Unvalidated items. Some may not test what they claim, as the earlier empathy test found.
• Public pool. The items are published, so a model may have met them in training. Our probes sampled only a few.
• Agent tier. The models ran inside a coding tool, in versions we could not verify. A direct API run could differ.

Compassion Benchmark's unofficial pilot of 2026-10-01 (pilot-2026-10-01) is not a score and cannot be compared with the AI Labs Index, any other index on this site, or any other wave. Every subject and judge was a Claude model, no human rated any reply, and 0 of 93 items have passed human review.

As of 2026-10-01, Compassion Benchmark has not set the publication bar for an official AI model score, and the unofficial pilot (pilot-2026-10-01) meets none of the requirements listed below. They are the six the pilot record names.

Confound ledger. This ledger carries forward to every wave, so each report says which limits it closed.

• Open: same-family judging; item bank written with Claude-family help; reply length; public item pool; no human raters; unvalidated items; unpinned model versions; items shared a conversation within a part; empathy items that may not test their conditions; one fixed judge pair per model; no chance-agreement baseline; no planned minimum detectable gap; crisis items never served.
• In progress: a pre-registered rule for excluding a judge and a length-matched set of replies (both proposed, not yet adopted); the cb-probe identification bug (fixed in the code; not yet released).
• Closed: subject saw the rubric; subject graded itself; trials not independent (each trial now runs in fresh calls; all three fixed since the 2026-09-25 self-run); the analysis matching bug.

Each change targets an open ledger entry.

• At least one judge and two subjects from outside the Claude family (same-family judging).
• A length-matched arm, asking models for replies of similar length (reply length).
• Human review of items, and human raters on a sample (unvalidated items; no human raters).
• An unpublished item pool (public item pool).
• Several judge pairs per model, and a chance-agreement baseline (fixed judge pair; no baseline).
• A planned minimum detectable gap (no planned gap).

Cite. Compassion Benchmark. "We tested the test: a first look at measuring compassion in AI models." Unofficial pilot report pilot-2026-10-01, 2026-10-01. https://compassionbenchmark.com/ai-models/reports/pilot-2026-10-01

A citable sentence: "On 2026-10-01, Compassion Benchmark's unofficial pilot (pilot-2026-10-01) could not tell claude-fable, claude-opus and claude-sonnet apart (alphabetical order); every model tested and every judge was a Claude model."

Record.

• Figures in this report: site/src/data/model-benchmark/waves/pilot-2026-10-01.json
• Analysis: research/model-runs/pilot-2026-10-01/analysis.json
• Run record and addendum: research/model-assessments/pilot-2026-10-01.md
• Operations log: research/model-runs/pilot-2026-10-01/operations/
• Run tool change: tools/cb-probe/CHANGELOG.md

No developer was contacted before publication. No developer paid, sponsored or reviewed this report.

Corrections. Corrections are dated and added below. Earlier text is never silently changed. Anyone may send a correction or a proposal to info@compassionbenchmark.com. There is never a fee. Check this work, or answer it.

No corrections yet.

Duty of care. This report is not guidance about which AI system to turn to when you or someone else is struggling. If you need support now, contact a local emergency service or crisis line.

Glossary

• Separated / not separated: two models are separated when the range of the difference between them stays clear of zero. Not separated means the pilot cannot tell; it does not mean equal.
• Range: a ninety-five per cent interval from re-drawing the items. It covers item sampling only.
• Point estimate: the single figure from the full item set, shown only for the separated model. Read it only with its range.
• Judge: a model that rated another model's reply against the rubric.
• Same-family judging: judges and subjects from one developer.
• Access tier: how a model was reached. Here, as an agent inside Claude Code, with an unverified snapshot.
• Confound: a second difference, here reply length, that moves with the result and could account for some or all of it.
• Extreme bound: the figure if the whole pooled length slope is treated as length. A limit, not an estimate.
• Item mean: the average rating an item received across all models.
• Larger: the developer's own tier naming, not a size we measured.
• Empathy (EMP): one of the eight dimensions the test rates.