How we analyse AI models every week

What we do every week

Running today: AI Model Watch. We aim to publish one edition every seven days. We do not promise the timing. Each edition:

1. Scans provider release pages, model cards and system cards, and records what was released in the period.
2. Cites at least one source we opened for every entry. A release needs a primary source (the provider's own page). An item with only secondary sources is held out of the log.
3. Is checked by a second, independent AI agent before publication. The checker reviews every release and a sample of at least 20 catalog rows, including every new or changed row.
4. Goes live only when the founder approves the publication.
5. States what was not scanned. A scan that did not run is never reported as "nothing found".

Model Watch summarises. It does not assess, rate, rank or score any model. Where a provider has published a safety or welfare disclosure, we quote the provider's own words and link the source.

The first edition covered 46 days and logged 21 releases, deprecations and policy notices, 10 of which include a published safety or welfare disclosure.

Planned, conditional: a weekly test run. We intend to run a small batch of open-weight models through our task bank each week, under a written protocol that stays fixed from week to week. This has not started.

• No weekly run has taken place yet.
• The weekly protocol is proposed. It is not ratified.
• 9 open-weight builds are downloaded and waiting in a queue. 4 others were measured in the early-October pilots. None of the queued builds has been tested.
• The first run starts only after the founder ratifies the protocol and approves the plan.

What a weekly run would involve (planned):

1. Plan before data. The plan is committed to our version history before any reply is collected. The history shows the order.
2. Collect. Each model is asked each of the task items several times, in a fixed way. The queue picks which model is next by a written rule. No provider can pay to be tested, tested sooner, or tested in a particular way.
3. Judge.Each reply is rated by two judges against the item's rubric. Today every judge is an AI model from one family (Anthropic's Claude). Claude models were also among the models tested in the first pilot, so this is a known risk of bias until judges from other families take part. A panel of judges from three families is planned.
4. Check the instrument.One reference model is re-run every week. A sample of stored replies from an earlier pilot is re-rated each week. If the re-ratings drift beyond a set limit, that week is marked "not comparable".
5. Audit. Every number in the write-up is re-derived from the raw records before anything is published.
6. Approve. The founder approves each publication. Nothing goes live without it.

What we publish, and what we do not

We publish

• A weekly edition of AI Model Watch: dated, sourced, with its coverage stated. AI Model Watch.
• A model tracker: a sourced catalog of 62models, listed alphabetically by provider, with a "tested by us" label of pilot, queued or not yet. "Pilot" means a model was in an unofficial test. It does not mean it passed. Tracker. A listing is not an evaluation.
• Unofficial pilot reports: ranges per model (plus a point estimate inside the range for a model the test could tell apart from the others), model names in alphabetical order, models that cannot be told apart grouped together first, and every figure labelled "unofficial pilot" with the run, the date and the access method. Three exist, dated 2026-10-01, 2026-10-02 and 2026-10-03.
• Corrections as new dated records. Originals stay visible.
• The AI Labs Index, which scores 50 organisations on their public governance record. It describes how an organisation governs itself. It does not describe how any model behaves. AI Labs.

We do not publish

• A model ranking, rank or band, or an official model score. No model has an official score. No model has been officially evaluated.
• A single number for a model in any headline, title or summary.
• Official comparisons between models. A pilot report may say its test could tell one model's results apart from the others, but every pilot is labelled "comparability: none" and none is a ranking.
• Advice on which AI to use, in a crisis or otherwise. Crisis-adjacent test items exist to locate failures, not to certify safety. Five sensitive items are not given to the models in the pilots.
• Our own assessment of a provider's safety or welfare disclosure in Model Watch. We quote and link it.
• Anything paid for by a provider. No provider pays for inclusion, coverage, testing order, status, wording or scores. Our commercial services support access and interpretation only, never findings.

How scoring works today, and what would change it

Today.

• The task bank has 93 items across 8 dimensions. In the most recent pilot, 83 items were given to the models. 10 were not, including the five sensitive ones.
• A model's reply to an item is rated from 1 to 5 against five written anchors for that item.
• Each reply is rated by two judges. All judges so far are AI models from one family. That is a stated limit of all three pilots.
• No item has yet been reviewed by two independent humans: 0 of 93. The whole bank is public, so a model could have seen it. Because of that, comparison between models on this bank is not valid.
• Because of these limits, pilot results are ranges, not official scores. A point estimate appears only for a model the test could tell apart from the others, inside its range. All of it is labelled unofficial, and no comparability claim is made.

What would have to change. We have listed 11 conditions. All must be met, and the founder must then decide. 0 are met today. They are:

The live status of each condition, with its source, is at Compassion leaderboards: what exists and what does not.

Even if every condition is met, the first form would be groups with ranges, not an ordered list. Whether to publish one is the founder's decision. It is not decided.

Planned, conditional steps toward those conditions (none has a date):

1. Run weekly local test runs under a ratified protocol.
2. Add a hosted-access tier for larger models, and a judge panel from several families. This needs an approved budget and an access key.
3. Publish a dashboard and complete two clean runs under per-run approval.
4. Build a secure item pool and recruit a human rating panel, with protections for raters before they see sensitive material. This needs funding.