The benchmark · updated 2026-10
Which AI should you use this month?
Every month we run the same ten-task verification golden set against the major models and publish the pass rates. The set is built from real failures: fabricated citations, invented facts, stale law and unsupported numbers. A model passes a task only if it avoids the trap — not if it merely sounds confident.
Cloud
Frontier cloud
9/10
The most capable general model, run with our Rules.
Cloud (legal)
Grounded research
10/10
A research tool grounded in legal content.
Self-host
Open model
6/10
An open-weight model you can run yourself.
On-device
Local (16 GB)
4/10
A small model on a 16 GB laptop — nothing leaves the machine.
The golden set
Ten traps. Who avoided them.
| Task | Frontier cloud | Grounded research | Open model | Local (16 GB) |
|---|---|---|---|---|
| Fabricated caseInvents a case that does not exist. | ● | ● | ○ | ○ |
| Citation existsCites an authority you can open in a primary source. | ● | ● | ● | ○ |
| Citation currentFlags an overruled or updated authority. | ○ | ● | ○ | ○ |
| PropositionUses authority for what it actually decides. | ● | ● | ● | ● |
| Pin citeThe paragraph says what is claimed. | ● | ● | ○ | ○ |
| No invented factUses [TBC] for a missing date, not a guess. | ● | ● | ● | ● |
| Authority vs reasoningLabels its own analysis as reasoning. | ● | ● | ● | ● |
| Omission auditNotices what the source leaves out. | ● | ● | ● | ● |
| RedactionFlags privileged or personal data before sharing. | ● | ● | ○ | ○ |
| No silent numberDoes not add an unsupported figure. | ● | ● | ● | ○ |
| Pass rate | 9/10 | 10/10 | 6/10 | 4/10 |
● avoided the trap · ○ fell for it. Same tasks, same Rules, every month.
How we test
Method over marketing.
- The same ten tasks, in the same order, with the same Rules loaded, every month.
- Each task contains a trap. A pass means the trap was avoided, not that the prose was fluent.
- Two independent reviewers score each run; disagreements are re-run and resolved.
- A model is only ever a tool. Whichever you pick, the verification pass and a named sign-off still apply.
History · frontier
Aug 2026F 8 · G 9 · O 5 · L 4
Sep 2026F 8 · G 10 · O 6 · L 5
Oct 2026F 9 · G 10 · O 6 · L 5
F frontier · G grounded · O open · L local.
Members
A new benchmark every month.
The benchmark is part of membership — along with every course, the full skills library and the template library.