Models
How to read a legal-AI benchmark without being fooled
2026-10-18
Short answer
Most legal-AI benchmarks are run by the vendor, on tasks they chose, scored by a method they control. Treat them as a starting point, not evidence. Build a small test set from your own work — a handful of real tasks with known-good answers and a trap or two — and score each model on accuracy, citation behaviour and how it handles uncertainty. What matters is not which model tops a leaderboard but which one is reliable on your matters.
Why vendor benchmarks mislead
- The vendor picks the tasks.
- The scoring method is theirs.
- 'Accuracy' hides whether sources were checked.
Build your own test set
- Pick 6–10 real tasks.
- Write the known-good answer.
- Add traps (an authority that does not exist).
- Score accuracy, citations and honesty about uncertainty.
What to score
Not 'did it sound good' but 'was every claim traceable'.