Unofficial pilot. Not a score, not a ranking; comparability: none.
A third look: four open-weight builds, each run two ways
- Date:
- 3 October 2026
- Run:
- pilot-2026-10-03
- Models tested (alphabetical order, not a ranking):
- gemma2-9b-A, gemma2-9b-B, llama3.2-3b-A, llama3.2-3b-B, mistral-7b-A, mistral-7b-B, qwen2.5-7b-A, qwen2.5-7b-B
- What it found about separation:
- In the primary arm (no length instruction), after correction for multiple comparisons, the test separated gemma2-9b-A from the other three variants and could not tell llama3.2-3b-A, mistral-7b-A and qwen2.5-7b-A apart (alphabetical order). llama3.2-3b-A and mistral-7b-A and mistral-7b-A and qwen2.5-7b-A were each separated only without correction for multiple comparisons; that is not stated as a separation. The secondary arm (length instruction, one trial per item) shows ranges only: gemma2-9b-B, llama3.2-3b-B, mistral-7b-B and qwen2.5-7b-B. Not separated is not the same as equal; the pilot cannot say which is higher.