How to read a legal-AI benchmark without being fooled
Oct 18 2026 · Models
Vendor benchmarks are marketing. A one-afternoon test on your own matters beats any leaderboard.
Most legal-AI benchmarks are run by the vendor, on tasks they chose, scored by a method they control. Treat them as a starting point, not evidence. Build a small test set from your own work — a handful of real tasks with known-good answers and a trap or two — and score each model on accuracy, citation behaviour and how it handles uncertainty. What matters is not which model tops a leaderboard but which one is reliable on your matters.
Why vendor benchmarks mislead
- The vendor picks the tasks.
- The scoring method is theirs.
- 'Accuracy' hides whether sources were checked.
Build your own test set
- Pick 6–10 real tasks.
- Write the known-good answer.
- Add traps (an authority that does not exist).
- Score accuracy, citations and honesty about uncertainty.
Not 'did it sound good' but 'was every claim traceable'.
Written for legal professionals. Not a guarantee of accuracy — always verify every claim against its source, and let a named human sign off. Get the free handbook.