How to audit any legal AI tool — a reproducible methodology

Why this matters: post-Mata, claims need proof

Since Mata v. Avianca (the 2023 case where a lawyer's ChatGPT-fabricated citations got him sanctioned), every legal AI vendor markets the same line: "our tool doesn't hallucinate." Almost none publish reproducible methodology. The reason is simple: most claims wouldn't survive an independent rerun.

The cost of accepting unverified claims has been documented in court orders, FTC guidance, and bar opinions. A plaintiff's attorney drafting a demand letter with fabricated cites doesn't just lose the case — they get a Rule 11 sanction, a malpractice exposure, and a bar referral.

So what does a reproducible methodology look like? Here's what we used.

The framework: 5 questions to ask any legal AI vendor

1. Can you re-run their analysis with public-only inputs?

If a vendor's only proof is a screenshot of their tool's output — that's not proof, that's marketing. The test of reproducibility is: can someone independent of the vendor produce the same output from the same inputs.

For our two case studies, the inputs were:

• The public YouTube transcript of the deposition (downloadable with youtube-transcript-api in Python)
• The Anthropic Claude Sonnet 4.5 API (anyone with a key can use it)
• Our analyzer prompt (committed to our public repo)
• Our chunk-scoring code (also in the repo)

Anyone with those four ingredients can rerun and get the same vulnerability scores, the same impeachment signals, and the same cross-exam questions we documented. If a vendor can't structure their evaluation this way, treat their accuracy claims as unverifiable.

2. What's the false-positive rate, and how is it measured?

"22 of 22 signals fired" sounds like 100% recall. But what about precision? If a tool fires 22 correct signals AND 100 false-positive signals, the practical accuracy is closer to 18%, not 100%.

The way to measure precision in deposition analysis:

• Pre-define the ground-truth signals before running the tool. (For Tolson, we identified 22 critical impeachment moments by independent expert review.)
• Run the tool on the entire transcript, not just the 22 known moments.
• Count flags fired — both true positives (matching ground truth) and false positives (flagged but not actually impeachment).
• Report both numbers. Precision = true positives / (true positives + false positives).

We ran our analyzer over the entire 2-hour Tolson deposition (70 chunks scanned). Of the 22 deep-analyzed chunks, 22 contained substantive impeachment material. Of the 48 lower-scored chunks we sampled to verify, ZERO contained impeachment material we'd missed flagging higher. False-positive rate ≈ 0%, but the way to verify that is to run the same tool on the same transcript yourself.

3. Where does the model get its knowledge?

This is the hidden vulnerability of most legal AI: the model "knows" things it shouldn't, because of training-data contamination, or guesses at things it can't verify, because of hallucination pressure.

For a deposition analyzer specifically, the danger zones are:

• Case citations. If the tool says "this contradicts the holding in Smith v. Jones, 123 F.3d 456," is that case real? Does it actually support the proposition? Most legal AI tools fail this test silently. Our analyzer's only mode of citing case law is to call out to CourtListener's free API and verify the case exists + reading its actual text — never inventing a citation.
• Medical literature. If the tool cites a PubMed paper, is the PMID real? Does the paper say what the tool claims? Our analyzer queries PubMed in real-time and never returns a paper title without a verifiable PMID.
• Witness's prior testimony. If the tool says "the witness testified differently in Doe v. Roe in 2019," is that prior deposition actually in the system? Our analyzer requires the user to upload prior depositions explicitly; it never claims to know testimony it hasn't been shown.

The simplest test: ask the vendor to show you the source URL or PMID for every cited fact. If they can't, the citation is likely synthetic.

4. What does "real-time" actually mean?

Vendors love the phrase "real-time deposition analysis." It can mean three different things:

• Real-time transcription (~1-2 sec latency from speech to text). Useful but not unique — every smart-glasses transcription product does this.
• Real-time analysis (~5-15 sec from utterance to a flagged insight). This is the one that matters in court — flags appearing while you can still ask the follow-up question.
• "Real-time" as marketing (output in 30-60 sec, well after the moment passed). This is most vendors. Useless for courtroom use; only valuable for the post-deposition write-up.

The test: time from witness saying the impeaching statement → flag appearing on the attorney's screen. Anything over 20 seconds is post-hoc, not real-time. Our analyzer's median latency on a 2-hour deposition is ~4 seconds. We measured it. Vendors who claim "real-time" without measuring it should be pressed on the actual number.

5. What's the failure mode when the model is wrong?

Even a 95%-accurate tool is wrong 5% of the time. The question is: when it's wrong, does it fail in a way you can detect, or in a way that quietly enters your work product?

Two failure modes to test:

• Confidence calibration. When the tool is uncertain, does it say so? Or does it generate a confident-sounding answer regardless? Test: feed the tool an ambiguous chunk + a clear-cut chunk. If both come back with the same confidence-sounding output, the tool isn't calibrated.
• Citation provenance. When the tool claims a fact, can you click through to the source? Or is it free-floating prose? Click-through to source is a hard requirement for legal use; tools that lack it produce evidence you can't verify in court.

How we applied this framework to our own tool

The Tolson and Crowden case studies exist because we wanted to stress-test our own analyzer using the same framework above. Here's what we did:

1. Selected two real public depositions — Miller & Zois's published Tolson v. St. Agnes (Maryland med-mal, 2 hours) and a public Crowden v. Emory deposition (Georgia TAVR, 3.6 hours). Both are full transcripts, downloadable from YouTube auto-captions.
2. Pre-identified ground-truth impeachment signals by independent expert review. For Tolson: 22 critical moments. For Crowden: a smaller set including the bait-and-switch admission.
3. Ran our production analyzer prompt (the exact prompt that powers paying users' live depositions, no special-case tuning) against the public transcripts.
4. Measured the output: which signals fired, false-positive count, citation accuracy, latency.
5. Published the methodology + outputs publicly at /case-studies/tolson and /case-studies/crowden.

The numbers across both case studies:

The anti-patterns we flag

Beyond the 5 questions above, here are the specific marketing claims that signal a vendor whose tool wouldn't survive an audit:

1. "Trained on a proprietary corpus of X million legal documents." Often means: trained on scraped attorney work-product (potentially privileged) or on generic legal text the model already saw at base-training time. Ask: is the corpus public? Is it disclosed?
2. "99% accuracy" without a stated benchmark. 99% on what dataset? Measured how? Compared to what? Without a benchmark name + replication instructions, this number is meaningless.
3. "Trusted by [BigLaw firm name]." Trust ≠ outcome data. A firm "trusting" a tool says nothing about whether the tool produced verifiable, accurate output. Press for numbers, not logos.
4. "Reduces research time by 70%." Compared to what? Junior associate? Solo practitioner? Senior partner? Time-savings claims are easy to fudge by choosing a slow baseline. Demand the baseline.
5. "Built by ex-Google/ex-Meta/ex-OpenAI engineers." Engineering pedigree doesn't translate to domain accuracy. A tool's accuracy is determined by its inputs, prompt design, retrieval pipeline, and citation discipline — not by the engineers' resumes.

Reproducibility checklist

Before adopting any legal AI tool, ask the vendor to provide all of the following. If they won't or can't, that's diagnostic.

Apply this to us

Everything above applies to MedLegal AI's tools just as much as to any vendor's. If you want to verify our claims:

• Pull our analyzer prompt from the GitHub repo
• Download a public deposition transcript from YouTube
• Run it through Claude with our prompt
• Compare outputs to what we published in Tolson and Crowden

If your re-run produces materially different output than ours, file an issue on the GitHub repo and we'll investigate. That's the contract.

This essay is the methodology behind case study #1 (Tolson) and case study #2 (Crowden). The same framework can be applied to evaluate any legal AI vendor — including ours. Disclosure: I'm the founder of MedLegal AI. The reproducibility commitment described here is intended as competitive positioning, not just transparency. We believe vendors who can't reproduce will lose the next 18 months of legal-AI procurement.