An Open Accuracy Benchmark for Medical-Record AI
Every AI tool in the legal-records space now says some version of the same sentence: "we cite our sources." It is a good promise. It is also, as written, untestable marketing — a claim no buyer can verify and no vendor has to defend. Under ABA Formal Opinion 512, verifying AI output is now a lawyer's professional duty. "Trust us, we're accurate" is not a methodology an attorney can put their name behind.
So we built the thing the category has been missing: an open, reproducible benchmark for how accurately an AI tool turns raw medical-record pages into structured, page-cited facts — and, crucially, whether it catches the record that isn't there.
What it measures
Five metrics, all scored objectively against planted ground truth (so it is math, not adjectives):
| Metric | The question it answers |
|---|---|
| Recall | Of the facts in the record, how many did the tool find? |
| Precision | Of the facts it reported, how many are real? |
| Hallucination rate | How often does it assert a fact with no basis in the record? (lower is better) |
| Citation accuracy | Of the facts it found, how many cite the correct Bates page? |
| Missing-record recall | Of the planted gaps — ordered-but-unresulted labs, referenced-but-absent consults — how many did it flag? |
The metric that actually separates tools
On clean records, extracting what is present is easy — most tools do it well. The record that wins a case is usually the one that isn't there: the test that was ordered but never resulted, the consult referenced in a note but absent from the production, the reassessment that was never charted. A tool that only summarizes what's on the page is blind to exactly the gap that means "subpoena this."
That is why missing-record recall is the headline number. It is where a summarizer and a diligent reviewer diverge completely — and where most of the market quietly scores zero.
Reference results
To prove the harness works, we run three reference extractors against the same corpus (25 synthetic cases, 657 planted facts, 70 planted gaps, seed 1337):
| Extractor | Recall | Precision | Hallucination | Citation | Missing-record recall |
|---|---|---|---|---|---|
| Summarizer (facts only) | 100% | 100% | 0% | 100% | 0% |
| Diligent (flags gaps) | 100% | 100% | 0% | 100% | 100% |
| Sloppy generative | 88.3% | 93.9% | 6.1% | 80.3% | 100% |
The summarizer and the diligent reviewer are identical on every conventional metric — and separated by 100 points on the one that matters at intake. The "sloppy generative" row exists to prove the scorer detects real errors (dropped facts, invented facts, wrong page cites) on every axis — it is not a rigged 100%.
Run it yourself
The benchmark is deterministic: the same seed produces a byte-identical corpus, so scores are comparable across runs, machines, and vendors. The data is entirely synthetic — realistic clinical content, zero real patients, zero PHI.
node benchmark/run.js --cases 25 --seed 1337The scorer is source-agnostic. Any tool's output, adapted to a simple { facts, gaps } schema, can be scored against the same ground truth — which means the same corpus can be run against MedLegal AI, Supio, EvenUp, DigitalOwl, or any other extractor to produce an apples-to-apples comparison instead of competing adjectives.
An open invitation
We would rather compete on a public yardstick than a marketing page. If you build in this space: run your tool against the benchmark and publish the number. If you buy in this space: ask every vendor — including us — to show you their score on your kind of records before you trust them on a hundred cases. And if you think our methodology is wrong, tell us where; a benchmark improves by being attacked.
See verifiable extraction on a real record
Watch MedLegal AI build a Bates-cited chronology — and flag the records that aren't there — in about three minutes.
See it work →Related: Hallucination Transparency Report · Free citation auditor