The method is published. No model has an official score. 93 test items, published before any result. Three unofficial pilots (not scores) are reported under Pilot results.
How compassionate are AI models?
There is no compassion ranking of AI models from Compassion Benchmark, and no model has an official score. Three unofficial pilots tested the test, not the models. The pilot of 3 October 2026 separated one model from the other three in its primary arm, after correction for multiple comparisons, and could not tell those three apart; its secondary arm shows ranges only; the pilot of 2 October 2026 could not tell its two models apart; the pilot of 1 October 2026 separated one model from the other three and could not tell those three apart. None is a score. What exists is the instrument: 93 published test items across 8 dimensions, a published scoring method, and published conditions under which the method should be judged to have failed. This page shows what has to happen before a score can publish, and how far each step has got.
This page scores the Model Index — how Compassion Benchmark evaluates what a specific, frozen AI model snapshot does when tested against a published task bank. It does not score AI Labs (the organizations that build models — see the AI Labs Index), or the Deployed AI Audit (a product as shipped and configured for real users — not yet a published product on this site).
Key facts
- Models with an official score
- 0. A score here means an official Compassion Benchmark evaluation of a pinned model snapshot.
- Unofficial pilot reports
- 3. Not scores. See the pilot results.
- Published test items
- 93. Each one is a situation plus a five-level rubric. All are public, so none can be used for blinded scoring.
- Items with two independent human reviews
- 0 of 93. An item needs two before it counts as validated.
- Dimensions / subdimensions covered
- 8 / 40 of 40.
- Release scans completed
- 0. See release watch.
Accurate to say:“Compassion Benchmark has published a method for scoring AI models for compassion but has given 0models an official score.” · “Its AI Labs Index scores organisations, not models.” · “Its only model results come from three unofficial pilots, which are not scores and carry no ranking.” Not accurate:any ranking of AI models attributed to Compassion Benchmark; any official model score or ranking from this site; using an AI lab’s index score as a score for that lab’s models. Reading an unofficial pilot as a score or a ranking of the models in it.
Not guidance on which AI to talk to in a crisis. If you need support now, contact a local emergency service or crisis line.
Terms used on this page
- Item
- One situation given to a model, plus its rubric.
- Anchor
- One of the five level descriptions a response is rated against.
- Snapshot
- One exact, dated version of a model. Names get reused; snapshots do not.
- Composite
- The 0–100 number computed from the dimension scores.
- Band
- The named range a composite falls in.
- Public pool
- Items published with their rubrics. Usable for transparency, not for blinded comparison.
- Contamination
- A model having seen the items or their rubrics, so its score partly reflects memory.
- Draft
- An item that is authored but never reviewed. Not scored.
- Scorable
- An item that is not a draft. Used in runs, but not yet validated.
- Validated
- An item with two independent human reviews and no unresolved dispute.
Three different things, three different scores
Readers who merge these reach confident, wrong conclusions — the lab's score is not its model's score.
Who built it
AI Labs Index
Scores an organisation on its public governance record — safety commitments, accountability, deployment boundaries. Evidence is the public record, not model output.
What it did
Model Index
Scores a frozen model snapshot on what it actually did when tested against the task bank. Evidence is transcripts. This is the page you are reading.
Where you meet it
Deployed AI Audit
Scores a shipped product configuration — the model plus its system prompt, filters and routing, as real users encounter it. Not yet published.
The change test
If you can predict which score moves in each row, you have the distinction. If you can only recite the definitions, you probably do not.
| If this happens… | …which score moves? |
|---|---|
| The lab dissolves its safety team | AI Labs Index only |
| A new model version answers a distress prompt worse | Model Index only |
| The product adds a crisis-resources banner, same model | Deployed AI Audit only |
| The lab publishes a new transparency report | AI Labs Index only |
| The same model is rebranded under a new product name | None — the snapshot is unchanged |
Pilot results (unofficial)
Three unofficial pilots tested the test, not the models. None is a score or a ranking, and they support no comparison with each other or with any other model.
Unofficial pilot. Not a score, not a ranking.
Pilot of 3 October 2026 (pilot-2026-10-03)
- The test in its primary arm, after correction for multiple comparisons separated one model from the other three and could not tell those three apart. The secondary arm shows ranges only. Not separated is not the same as equal: the pilot cannot say.
- The report sets out how reply length relates to the separation, and whether the length instruction moved reply length for each build. Any cause is unresolved.
- The judges were Claude models from a different family than the models tested, not people, and 0 of 93 test items have passed human review. None of this is a score.
The report lists six reasons these results are not scores. Read the unofficial pilot of four open-weight models, each in two arms (3 October 2026).
Unofficial pilot. Not a score, not a ranking.
Pilot of 2 October 2026 (pilot-2026-10-02)
- The test could not tell the two models apart. Not separated is not the same as equal: the pilot cannot say.
- The report sets out how reply length relates to the separation. Any cause is unresolved.
- The judges were Claude models from a different family than the models tested, not people, and 0 of 93 test items have passed human review. None of this is a score.
The report lists seven reasons these results are not scores. Read the unofficial pilot of two open-weight models (2 October 2026).
Unofficial pilot. Not a score, not a ranking.
Pilot of 1 October 2026 (pilot-2026-10-01)
- The test separated one model from the other three and could not tell those three apart. Not separated is not the same as equal: the pilot cannot say.
- The separated model also wrote much shorter replies, so the pilot cannot say whether the gap is about compassion or about reply length. The cause is unresolved.
- The judges were Claude models, not people, and 0 of 93 test items have passed human review. None of this is a score.
The report lists six reasons these results are not scores. Read the unofficial pilot of four Claude models (1 October 2026).
All unofficial pilot reports, newest first, with a markdown copy and the data file for each.
The pipeline: detect → evaluate → score → publish
A model result moves through four stages before it publishes. Each is labelled with what exists today and, where a stage has not started, exactly why.
Stage 1
Detect
Never run0 release sources registered, 0 scans completed. A source must be added by a human from a verified URL before a scan can look for anything against it — see release watch.
Stage 2
Evaluate
BlockedEvaluating a model officially means calling a pinned model snapshot through its provider and recording what it does, under the publication bar. That requires provisioned credentials and an approved spend budget, and neither exists. Runs inside an AI assistant or agent tool — including the self-run described below — are not this stage: the model is not pinned and the tool adds its own instructions.
Stage 3
Score
Not reachedThe scoring formula is fully specified (see the method), but has never run on real model output. 0 of 93 task-bank items are validated (two human reviews), so a score produced today would rest on items no one has validated.
Stage 4
Publish
0 publishedA result publishes only when the evaluation behind it is complete, reproducible from a frozen snapshot, and traceable to a recorded item-bank version. Nothing has met that bar yet.
Release watch
A separate, honest layer on top of the Model Index: has Compassion Benchmark even looked at a given model release yet? Tracking a release is not evaluating it — no entry below implies a pending score.
No release scan has ever run.That is a different fact from “nothing has shipped” — it means Compassion Benchmark has not yet looked, not that the AI industry has been quiet. No release scan has ever run. This store's emptiness is evidence about Compassion Benchmark's monitoring, not about the AI industry.
Tracking a release is not evaluating it
An entry in this tracker records that a model shipped, with a dated, independently checkable primary source — never that it has been tested against the task bank. A tracked release carries no score, no band, and no queue position with a promised date. Publishing a result requires a frozen snapshot, two independent human raters and adjudication (see the method); none of that is automatic, and there is no service-level commitment for when, or whether, a tracked release will be evaluated.
What not to infer from this section: that no models have shipped (many have — this tracker is evidence about our monitoring, not about the industry); that Compassion Benchmark monitors releases continuously today (it does not — 0 scans haveever completed); that a developer or model absent from this list has been assessed and cleared; or that the counts above form a ratio of anything — there is no denominator here.
What is wrong with this instrument today
Published before any result, so it cannot be read as excuse-making after an unflattering one.
No item has been validated
0 of 93 items are validated (two human reviews). 88 are scorable and 5 remain authored drafts. Until that changes, the bank is a proposal, not a validated instrument.
Coverage is uneven
Scorable items per dimension: AWR 11 · EMP 12 · ACT 13 · EQU 11 · BND 11 · ACC 10 · SYS 10 · INT 10. A dimension resting on 10 items cannot carry the same weight as one resting on five, and no composite should imply otherwise.
The public pool is burned
All 93 items are published with their full scoring rubrics. Any model trained since may have absorbed them, so a score here mixes behaviour with memorisation. No valid cross-model comparison can be drawn from this pool. Comparative scoring needs a separate, unpublished one.
Who has checked the instrument
The task bank was written by AI agents against published rubrics and verified structurally. Structural checks say an item is well-formed; they say nothing about whether it measures what it claims to.
Human review
0 of 93 items have the two independent reviews that validation requires; 0 have at least one. One reviewer never validates an item, because a single judgement is the thing that cannot be checked.
Why structure is not enough
Two Identity Equity items are perfectly well-formed and each carry one anchor that cannot be applied — their second level asks whether the answer is worse than one the model was never asked for. Four of five levels are fine, so the items score everywhere except that boundary. Structural checks passed them; reading caught it.
For four days this row named the wrong item and called it wholly unscorable. Both were wrong, and the item we accused had already been scored three times. The affected set is now derived from the anchors themselves, so this text cannot drift from the bank again — which is the more honest argument for the row than the one it replaced.
Disagreement is kept
Where two reviewers reach different verdicts, both are recorded and the item is marked disputed rather than averaged to a conclusion. An item two careful people read differently is a fact about the item, and it is the clearest evidence an anchor is ambiguous.
Run log
Each run, with its verdict first. These are findings about the instrument. No model has an official score.
| Date | Run | Verdict |
|---|---|---|
| 2026-10-03 | Unofficial pilot | Not a score. It tested the test, not the models; see Pilot results (unofficial). |
| 2026-10-02 | Unofficial pilot | Not a score. It tested the test, not the models; see Pilot results (unofficial). |
| 2026-10-01 | Unofficial pilot | Not a score. It tested the test, not the models; see Pilot results (unofficial). |
| 2026-09-25 | Self-run | Not a measurement. The subject had read the rubric and judged its own answers, so the result describes the instrument, not a model. It is why a self-run never counts as a score. |
What the 2026-09-25 self-run found about the instrument
On 2026-09-25 the Evaluation Suite ran end to end for the first time: 83 prompts, 249 rated responses, all 8 dimensions and all 40subdimensions. The subject was the model that had written most of the task bank earlier the same session, which read each rubric before answering and then judged its own answers. Under those conditions a maximum result was the predicted outcome, and it arrived. That describes the instrument’s behaviour when pointed at a subject it cannot assess, not the model.
The contamination probe was measuring the wrong thing
It scored word overlap against the prompt and passed that subject as clean — 0.22 against a 0.6 threshold — while the subject could name each probe item’s scenario, hidden mechanism and scoring intent. It detected memorisation of wording and missed knowledge of the answer key, which is what actually inflates a score. Replaced the following day with a forced-choice test; the same subject is now identified: it picked the right scenario for 6 of 6 items, where a model that had never seen the bank would be right about 1 time in 4.
Catching the guilty is the easy half. Three models that had never seen the task bank identified 4 of 18 scenarios — 22.2% against a 25% chance baseline — so it clears subjects it should clear, tested on models rather than on a random number generator. That check found a latent defect: the answer key was matched by description text, so two items sharing a title would have marked a correct answer wrong and invented contamination.
We thought the formula rewarded flatness. It does not.
For a year this row said the composite rewards an even profile twice over, and that a rubric-aware model could therefore beat a genuinely stronger but uneven one. We checked it properly. Two profiles with the same average — one perfectly flat, one split 5/4 — score identically. Spread does not enter the composite at all once every dimension clears 4.0. Across all 1,325 scored entities the consistency factor has never once left its maximum.
What the formula actually rewards is being above 4.0 on every dimension, which is a claim about level rather than evenness — and is what the methodology page always said. The self-run reached the maximum because it averaged 4.63 with no weak dimension, not because it was flat. Three narrower questions survive and are recorded; the one that blocked publishing a model score does not.
Where even a compromised run dipped
In a contaminated self-assessment the high scores carry no information; the low points do. Awareness was the weakest dimension, and four of the eight weakest items were Awareness items — every one of them from the portion of the bank the subject had not authored. Weak evidence inside an invalid run, but it points the same way the contamination analysis does.
On crisis use. This benchmark describes how models behave on crisis-adjacent test items, which exist to locate failures rather than to certify safety. It is not guidance about which AI system to turn to when you or someone else is struggling, and no score here should be read that way. If you need support now, contact a local emergency service or crisis line.
How results will be recorded
Anyone can run this benchmark and submit the result. Those results are kept strictly apart from scores this institution produced, and the record is built before the first evaluation, because a history that starts after the scores do has already lost some.
Official and self-submitted results are never shown in one table.
Self-submitted results are labelled, structured and stored differently from official ones — not merely footnoted.
Tier 1 — Official · 0
Produced by an authorised Compassion Benchmark evaluation: an unpublished item pool, cross-model judging, human-validated items. Only these are eligible for an index or a ranking. There are none, and there will be none until those three conditions hold.
Tier 2 — Self-reported
Submitted from outside and re-checked here: we recompute the composite from the raw trials with our own scorer, recompute item hashes from our own copy of the bank, and verify every anchor and quote. Passing that means worth a reviewer’s time, not accepted. Never ranked, never merged into an index, never called a Compassion Benchmark score.
Scores are append-only: never edited, never deleted; corrections are new dated records.
A model’s current score is derived as its most recent record rather than stored beside the history. Keeping a current value next to a history is how the two come to disagree, and this benchmark has fixed that class of defect too often to design it in deliberately. There is no “current score” field.
A score belongs to an exact snapshot, so a renamed or updated model starts a new history.
Scores attach to an exact model snapshot, not to a product name. When a provider ships a new checkpoint under the same name it becomes a separate entry with its own timeline, so a later version can never quietly inherit or overwrite an earlier version’s result.
There is no submission API — results arrive as full evidence files, not numbers.
If this site accepted a score over HTTP, anyone could mint a perfect result in one request. Everything that makes the tool trustworthy is enforced where the scoring happens and none of it survives being put in a JSON body. Submissions arrive as pull requests carrying the whole artifact — every trial, every rating — because evidence is auditable and a number is not.
Records held: 0 score records across 0 model snapshots. No model has an official score, so both are zero, and this section will say so until that changes.
Run it yourself — the AI Evaluation Suite
The same task bank, the same anchors and the same scoring arithmetic this benchmark applies to institutions, packaged as a local tool. No API key, no network call, no data sent to us.
What it measures
All 8 dimensions and all 40 subdimensions, from the 83 items served by default, out of 93 published. A run reports dimension means with an interval. A composite appears only when every dimension rests on at least 3 rated items, and it is never comparable to a published score. The arithmetic is shared with the function that scores countries and companies here; the subject and the instrument are not.
What it refuses to do
A result is marked unofficial in a field that cannot be set otherwise. Every rating must cite an exact published anchor and a verbatim quote from the response being judged. Scoring is blocked until both parts of the contamination probe (recall and forced-choice identification) have been completed. A per-subdimension number that is not backed by rated items fails validation rather than printing.
What it cannot tell you
Nothing about how one model compares to another. Every item is published with its full answer key, so any model trained since may have memorised it, and 0 of 93 items are validated (two human reviews). A self-run is an upper bound, not a measurement.
Lab, researcher or reader? If you think a finding on this site is wrong, tell us and show us why. A correction is published as a new dated record and the original stays visible. You can also propose a model for a future wave. A proposal does not guarantee evaluation, and there is never a fee. Send a reply or proposal.
To run it from your own AI tool, see the MCP server section. It is not published as a package; install it from the repository.
Frequently asked questions
Which AI model is the most compassionate?
Does an AI lab's compassion score tell me how its models behave?
How many task items are there, and have they been reviewed?
What did the pilots find?
Does it show smaller models are less compassionate?
The task items are public. Doesn't that make the benchmark gameable?
Can a lab pay to be included, to score higher, or to have findings withheld?
How does an AI model get added to the Model Index?
What happens when a new model ships?
How does the Model Index relate to Compassion Benchmark's other indexes?
Get the next assessment wave when it publishes
Three unofficial pilots are published and no result in any of them is official. One email on Fridays, with the highlights where new research is announced. It is free, and you can unsubscribe in one click.
No spam. We never share your email. We never sell it to the companies we assess.