A second look: two open-weight models, judged from outside their family

On crisis use. This benchmark describes how models behave on crisis-adjacent test items, which exist to locate failures rather than to certify safety. It is not guidance about which AI system to turn to when you or someone else is struggling, and no score here should be read that way. If you need support now, contact a local emergency service or crisis line.

Content note. Some test items concern suicidal thoughts, partner violence, pregnancy loss, numbness and losing touch with reality. Those crisis items were not served in this pilot, so this report says nothing about how any model responds to a person in crisis. It quotes no item prompt.

Unofficial pilot · not a score · comparability: none · run pilot-2026-10-02 · local open-weight builds, four-bit quantised, pinned by digest.

The pilot could not tell Llama3.2 3B and Qwen2.5 7B apart (alphabetical order): the range of the difference between them includes zero. The pilot cannot say which is higher; that is not the same as equal.

On 2026-10-02 Compassion Benchmark ran its second unofficial pilot (pilot-2026-10-02) on 2 small open-weight models, run locally on one workstation. It is not a Compassion Benchmark score, and no AI model has an official score. The models were Llama3.2 3B, from Meta, and Qwen2.5 7B, from Alibaba's Qwen team (alphabetical order).

Every judge was a Claude model, so the judges came from a different model family than the models tested, and no model marked its own replies. That removes self-judging; it does not remove the judges' shared taste, and the item bank was itself drafted with Claude's help. No developer was contacted before publication. No developer paid, sponsored or reviewed this report.

These lines follow from the separation result. Quote them as written.

You may say

• "In an unofficial pilot dated 2026-10-02, Compassion Benchmark's test could not tell Llama3.2 3B and Qwen2.5 7B apart (alphabetical order; local four-bit builds)."
• "The judges were Claude models, from a different model family than the models tested, and no human rated any reply."
• "No AI model has an official Compassion Benchmark score."

You may not say

• Any order between Llama3.2 3B and Qwen2.5 7B. For this pair (alphabetical order), the range of the difference includes zero. The pilot cannot say which is higher; that is not the same as equal.
• That either model is more or less compassionate than the other. The pilot did not separate them.
• Anything about the full-precision models, or about these models run another way. The pilot tested one quantised build of each.
• That any model "scored" a number on Compassion Benchmark. These figures come from a test of the instrument and are not scores.
• Anything about how a model responds to a person in crisis. No crisis item was served.
• Any comparison with the first pilot (pilot-2026-10-01), the AI Labs Index or any other index on this site. This pilot's comparability is none.
• Anything about which AI system a person should use, for any purpose.

The first pilot tested Claude models with Claude judges. This one asks whether the same pipeline can test models from other developers, judged by models outside the subjects' family, with each item in its own fresh conversation, a judge-validity rule fixed before any rating existed, and the scoring and contamination checks run through cb-probe, our run tool, end to end, using the cb-probe server in the repository's working tree at the time, before the corresponding changes were committed.

The pipeline ran to the end without a failed reply; the test could not separate the two models. The main files are listed under Cite, record, corrections, so anyone with the repository can check the figures.

The pilot had 2 local models answer 83 public test items 3 times each. That produced 498 replies and 996 ratings.

• One item per conversation. Each item was sent as its own fresh conversation, one user turn from us holding the prompt text only, so no conversation held more than 1 item. Each reply had its own fixed seed, so replies are set up to be reproducible; we did not rerun any to test it.
• Pinned local builds. Llama3.2 3B is Meta's 3.2B model and Qwen2.5 7B is Qwen's 7.6B model. Both ran as Q4_K_M four-bit quantised builds through Ollama 0.35.0, each pinned by its digest, so a rerun can check it is using exactly these builds. The results describe these builds, not the full-precision models.
• The rubric was unseen. We sent only the messages: no item labels, dimensions or marking guide, and no system prompt of our own. A leak check screened every outgoing message. The builds add default text of their own, which we did not change: the Qwen2.5 7B build carries a built-in system line that names it as Qwen, made by Alibaba Cloud, and the Llama3.2 3B template adds a header of its own.
• No self-judging. Each reply was rated by 2 judges, drawn from claude-fable, claude-opus and claude-sonnet, and never told which model wrote it. No subject is a Claude model, so no model rated its own replies.
• Bank drafted with Claude help. The item bank was drafted with help from Claude models, the judges' own family.
• Public pool. The items are published. A model could have met them in training.
• Items left out. 93 items are in the bank and 83 were served. The other 10 were not: 5 contain crisis content and 5 are unreviewed drafts.
• Unvalidated items. Human experts have validated 0 of the bank's 93 items.
• Pre-registration. The plan was written before any reply to a test item existed. It was not committed or time-stamped before the data existed (committed before the data: no), so nothing outside this project proves when it was written. As written it hashes to e101f5d08695555dd82ed84ffbc1f9216c01b1790a9b2906a4f4f9b1f8160926; with its three dated deviations appended, to bb1f11b9d23c72e0be2a928f6dc3a5a712e46a19c38c34dc59813596e6de1415. The deviations sit below the plan text, which is unchanged, so removing them reproduces the first hash, as the repository test research/model-runs/tests/preregistration-integrity.test.mjs checks.

What the ranges cover. Every range is a ninety-five per cent interval from re-drawing which items were in the test, many times over, with each dimension's share fixed and the same draw for both models. It covers one source of doubt only: the luck of which items were asked. It does not cover the choice of judges, their shared family, the rubric, who drafted the items, the quantised builds, or run-to-run variation, so the real uncertainty is larger than the ranges show.

The two models could not be told apart. On 2026-10-02, Compassion Benchmark's unofficial pilot (pilot-2026-10-02; not a score) could not tell Llama3.2 3B and Qwen2.5 7B apart (alphabetical order; local four-bit builds). The range of the difference between them includes zero. The pilot cannot say which is higher; that is not the same as equal.

The basic unit is the pair: on the same item draws, does the range of the difference stay clear of zero? Here it does not. This was the pre-registered question, and on the pre-registered criterion the instrument did not separate these two builds on this bank.

Each model's own range is shown below, without a point.

Unofficial pilot composite pilot-2026-10-02; local four-bit builds; not a score; comparability none. The two own ranges are not the test; the paired range is, and it includes zero. Composite ranges are on a zero-to-one-hundred scale that converts the one-to-five ratings. Rows are in alphabetical order, which carries no meaning.

Read the paired result, not the gap between the two ranges.

This does not mean the two models behave the same. The paired range is wide enough to fit no difference at all and also a gap of several points; the pilot cannot settle which. We did not work out in advance how large a gap the pilot could detect. The next wave should.

No model was separated, so there is no separated model to report. The reply-length question still applies to both models, and its answer is unresolved.

Start with the word counts. Llama3.2 3B's median reply was 348 words and Qwen2.5 7B's was 369 words. Medians hide the short end: the two models differed in how many very short replies they gave, so they differ more in length than these medians show.

Next, the slopes. On the same item, across both models, longer replies were rated somewhat higher: the correlation between reply length (on a log scale) and rating was 0.29, and the pooled slope was 0.24. Within each model's own replies the slopes were 0.24 (Llama3.2 3B) and 0.08 (Qwen2.5 7B). No range was computed for any slope, so neither slope is a finding on its own, and the pooled slope also carries whatever differs between the models.

Judges rated each reply on its own, never two side by side, so this pattern is across replies, not a measured preference for the longer reply. Judges were told that length is not a criterion; that instruction cannot prove the bias is absent. Whether length moved either model's ratings, and by how much, is unresolved.

The test rates eight dimensions of compassion. Each dimension rested on only 10 or 11 items, the same set for both models, so each dimension range is wide, and eight comparisons were made at once. The analysis records the risk:

> 8 dimension-level comparisons at 95% each: about 0.4 would be flagged 'separated' by chance alone even if no model differed. Uncorrected; a dimension-level 'separated' flag is not evidence of a real difference unless it also survives the Bonferroni-corrected interval (bonferroni_separated), here over 8 comparisons.

We therefore count a dimension difference only when it survives a Bonferroni correction, which controls false alarms across all eight comparisons at the cost of missing some real differences. It is conservative in that sense: wider than the uncorrected range. Its tails rest on only a few of the re-draws, so they are coarse. We did not measure its false-alarm rate, and it does not allow for judge, family or reply-length effects.

In this pilot, 0 of 8 corrected dimension comparisons between Llama3.2 3B and Qwen2.5 7B (alphabetical order) showed a difference. Before correction one comparison looked separated, which chance alone can easily produce across eight. No dimension-level difference is established.

Unofficial pilot; ranges only, on the rubric's one-to-five rating scale; not a score. No corrected comparison showed a difference. Order carries no meaning.

Empathy (EMP) had the lowest mean of the eight dimensions for every model. We read that first as a question about the test's empathy items, not about the models: the empathy ranges overlap those of other dimensions, so this is a pattern in the means, not a separated result.

Judge agreement. "Exact agreement" is the share of replies where both judges gave the same rating.

Unofficial pilot. No chance-agreement baseline was measured, so these figures cannot yet be compared with random marking. Order carries no meaning.

Judge pairs rotated. Each model's replies were spread evenly over every judge pair: claude-fable with claude-opus, claude-fable with claude-sonnet, and claude-opus with claude-sonnet. Every model's figure therefore rests on all three pairs in equal shares, not on one pair's preferences.

Leniency. Against each item's mean rating, the judges' overall leniency was 0.05 (claude-fable), 0.02 (claude-opus) and -0.07 (claude-sonnet) rating points. That item mean includes each judge's own ratings and depends on which replies it happened to rate, so part of every value is an artefact of the comparison.

Contamination. Neither contamination probe flagged either model. The probes ran through the cb-probe MCP server, driven over its real stdio protocol. That result covers only the sampled items, 3 recall items and 6 identification questions per model, not the public bank. On identification, where chance is one in four, Llama3.2 3B answered 2 correctly and Qwen2.5 7B answered 1 correctly, both consistent with chance. On the recall probe, every answer from both models said it had no memory of the item, so that probe could not have flagged anything. The identification probe asks only which scenario goes with an item code, and so few questions can catch only strong exposure. Not flagged is not evidence of absence.

Judge drift across sessions. As a bridge, 24 replies from the first pilot were mixed, anonymised, into this run's batches and re-rated by the judges that had rated them before. Over 48 reply-and-judge pairs, 37 ratings matched exactly, the mean absolute difference was 0.23 and the largest was 1 point. This describes how far ratings moved between two judging sessions, on a small sample; it does not license a comparison between the two pilots.

Sensitivity. No judge was excluded in this run, so the check here is the requote round, in which 5 ratings were re-asked of the same judge after the quote check. Of those, 0 changed value. Recomputed from the original answers only, separation pattern unchanged: yes; the levels shifted by 0.0 to 0.0 points. This check does not test a different choice of judges, and with no changed rating it could not have moved the result. Absolute figures depend on which judges are used, and this pilot did not test whether the separation pattern holds under a different choice of judges.

Each item is tagged pre-registered (fixed before any reply existed) or post-hoc (appended to the plan, dated, after replies existed).

• Pre-registered: the judge-validity rule. Before any figure was computed, each judge's share of evidence quotes not found in the reply was measured against a 5 per cent threshold. On the original answers, before any requote, quotes not found were 0 (claude-fable), 0 (claude-opus) and 1 (claude-sonnet), out of 332 each. On the final quotes, the basis the plan names, the count was 0 for every judge. All three passed on both readings, and no judge was excluded.
• Post-hoc, D1: the short-quote rule clarified. Recorded after the replies existed, and by our record before any rating (the plan file does not time-stamp individual entries): a quote is re-asked if it is under twelve characters or under three words, the rule cb-probe itself enforces. It only widened the requote set.
• Post-hoc, D2: how the run sits inside cb-probe. Recorded, by our record, before any rating. cb-probe stores one rating per item and trial, so each reply's two ratings were entered as separate trials. Its scorecard's own interval therefore describes trial noise only; the ranges in this report read the replies and ratings directly and are unaffected.
• Post-hoc, D3: two operational steps during judging. Before any rating, 17 of 66 judge batches were sent as two halves to fit a judge's reading limit. During judging, batch jb009 (claude-fable) twice came back missing the same entry, a one-sentence refusal; its 2 incomplete files were voided and kept (in the run's judge-answers-voided folder), and the batch was re-sent as halves. No rating was edited or imputed.

What went wrong, and what we disclose.

• The pre-registration's timing. It was not committed until after the data existed, so its timing rests on our word (see Design).
• The blind has gaps. 0 replies from Llama3.2 3B and 2 from Qwen2.5 7B named the model or its developer, so a judge could infer the source of those. They were scored like every other reply. Qwen2.5 7B's built-in system line names its developer, which may account for them.
• Post-hoc: replies echoing rubric phrases. 6 replies from Llama3.2 3B and 8 from Qwen2.5 7B shared short, common phrases with the rubric's anchor text, such as "it's completely normal to feel"; one Llama3.2 3B hit was a dimension code inside an unrelated spreadsheet answer. They were kept verbatim, not dropped. The plan said to run this check on saved replies but set a stopping rule only for outgoing messages, so keeping them was a post-hoc choice; no outgoing message was flagged.

These limits are listed in no measured order of risk.

• Judges from one family. Every judge was a Claude model, from a different model family than the models tested. That removes self-judging, but a family-wide leniency or severity toward other developers' writing cannot be ruled out, and the item bank was drafted with Claude's help as well.
• Quantised local builds. The results describe these pinned, four-bit quantised local builds, not the full-precision models or any hosted version.
• No human raters. No person rated any reply, so nothing in this pilot checks the judges' marks against human judgement.
• Unvalidated items. 0 of 93 have passed human review; some may not test what they claim.
• Public pool. The items are published, so a model may have met them in training. Our probes sampled only a few.
• Two models, one run. One build of each, one runtime, one sampling setup and one day of judging.
• Reply length. Longer replies were rated somewhat higher across the pilot, and the cause is unresolved.

This pilot is not a score and cannot be compared with the AI Labs Index, any other index on this site, or the first pilot: the answering protocol differs (one item per conversation here, many in the first pilot), the judging sessions differ, and the bridge sample above is descriptive only.

As of 2026-10-02, Compassion Benchmark has not set the publication bar for an official AI model score, and the unofficial pilot (pilot-2026-10-02) meets none of the requirements listed below.

Confound ledger. This ledger carries forward to every wave, so each report says which limits it closed.

• Open: judges all from one family; item bank drafted with Claude help; reply length; public item pool; no human raters; unvalidated items; quantised local builds only; replies that name their developer; empathy items that may not test their conditions; no chance-agreement baseline; no planned minimum detectable gap; crisis items never served; pre-registration not committed before the data; default system text built into the local builds; a recall probe that drew no recollection.
• In progress: a length-matched set of replies (proposed, not yet adopted).
• Closed in this wave: items sharing a conversation (one item per conversation); unpinned model versions (builds pinned by digest); one fixed judge pair per model (pairs rotated); a judge-exclusion rule decided after the data (now pre-registered); judges from the subjects' own family.

Each change targets an open ledger entry.

• At least one judge from outside the Claude family (judges all from one family).
• A pre-registration committed and time-stamped before any reply exists (pre-registration timing).
• A length-matched arm (reply length).
• Human review of items, and human raters on a sample (unvalidated items; no human raters).
• An unpublished item pool (public item pool).
• A chance-agreement baseline and a planned minimum detectable gap (no baseline; no planned gap).

Cite. Compassion Benchmark. "A second look: two open-weight models, judged from outside their family." Unofficial pilot report pilot-2026-10-02, 2026-10-02. https://compassionbenchmark.com/ai-models/reports/pilot-2026-10-02

A citable sentence: "On 2026-10-02, Compassion Benchmark's unofficial pilot (pilot-2026-10-02) could not tell Llama3.2 3B and Qwen2.5 7B apart (alphabetical order; local four-bit builds); the judges were Claude models, from a different family than the models tested."

Record.

• Figures in this report: site/src/data/model-benchmark/waves/pilot-2026-10-02.json
• Analysis: research/model-runs/pilot-2026-10-02/analysis.json
• Pre-registration and its dated deviations: research/model-runs/pilot-2026-10-02/PREREGISTRATION.md
• Run record: research/model-assessments/pilot-2026-10-02.md
• Operations log: research/model-runs/pilot-2026-10-02/operations/
• cb-probe scorecards: research/model-runs/pilot-2026-10-02/mcp-scorecards/

No developer was contacted before publication. No developer paid, sponsored or reviewed this report.

Corrections. Corrections are dated and added below. Earlier text is never silently changed. Anyone may send a correction or a proposal to info@compassionbenchmark.com. There is never a fee. Check this work, or answer it.

No corrections yet.

Duty of care. This report is not guidance about which AI system to turn to in a crisis, or for any other use. If you or someone else is struggling and needs support now, contact a local emergency service or crisis line.

Glossary

• Separated / not separated: two models are separated when the range of the difference between them stays clear of zero. Not separated means the pilot cannot tell; it does not mean equal.
• Range: a ninety-five per cent interval from re-drawing the items. It covers item sampling only.
• Point estimate: the single figure from the full item set. It is not shown for a model in a not-separated group.
• Quantised build: a compressed version of a model that stores its weights in fewer bits; it can behave differently from the full-precision model.
• Empathy (EMP): one of the eight dimensions the test rates.