Why we ran this test
We've published two reproducible case studies showing what the analyzer does on a single full deposition. This test takes the next step: can the same tool handle a cohort of 7 unrelated cases, including one where YouTube transcripts are disabled (so the only path is live audio)?
Two reasons this matters:
- Reproducibility. Single-case demos are easy to cherry-pick. A 7-case cohort drawn from public sources is harder to game.
- Live audio fidelity. Real depositions don't come with pre-loaded transcripts. The tool has to handle audio captured live from Deepgram, with all the medical-jargon ASR errors that entails.
Methodology
The cohort
7 public medical-malpractice depositions on YouTube — pediatric neurology, ER physician (×2), Johns Hopkins vascular surgery expert, triage nurse, ER nurse Doppler exam, defendant PA. Total ~12 hours of testimony.
Test A — closed-caption text
- Pull each video's English transcript via
youtube-transcript-api. - Bucket the 30-second ASR segments into 120-second chunks (~250-400 words each).
- Score each chunk by weighted regex hits across 8 impeachment patterns (memory failures, breach admissions, evasion, contradictions, qualifications gaps, FRE 702 vulnerabilities, etc.).
- Send the top 5 highest-scoring chunks per video through
POST /courtroom-ai/api/analyze— the production endpoint paying users hit.
Test B — live audio
- Mac output → MedLegal Multi-Output Device → BlackHole 2ch (virtual audio loopback).
- Mac default mic input set to BlackHole 2ch.
- Open courtroom-ai page in Chrome, select Deepgram source, click Start. Browser
getUserMediaopens BlackHole as the mic. - Open YouTube video in second tab, play 3 minutes per video.
- Audio routes from Chrome's YouTube playback → BlackHole → courtroom-ai's mic input → Deepgram WebSocket → analyzer.
- Capture analyzer events from the live transcript + signals panel.
Same analyzer prompt for both tests. Same Anthropic Haiku 4.5 model. The only difference is whether the input came from YouTube's CC API or from real audio playback.
Results
Per-video table
| Deposition | CC Daubert | CC Cross-Q | CC Adm | Live Adm | Live Sig |
|---|---|---|---|---|---|
| Pediatric neurologist (aneurysm) rb1GiUAX6aw |
74 | 24 | 4 | 4 | 14 |
| ER physician (Shady Grove) 9hjNW-eHvlg |
39 | 19 | 2 | 5 | 8 |
| ER physician (St Agnes — knee) Pq6o1fJFrmI |
54 | 18 | 2 | 0 | 6 |
| Vascular surgery EXPERT (Johns Hopkins) Emc1TaC3ceU |
75 | 23 | 1 | 2 | 8 |
| Triage nurse (Tolson) 9g1V1RV8F2A |
60 | 25 | 3 | 0 | 7 |
| ER nurse Doppler (Tolson) gUTPIOPkDHU |
74 | 25 | 2 | 0 | 18 |
| Defendant PA (live audio only — YouTube transcripts disabled) 4dZ9p8nC24U |
— | — | — | 0 | 6 |
What the numbers say
Both pipelines work end-to-end. Every CC chunk returned real model output (medical accuracy score, Daubert vulnerability score, cross-exam questions, admission/evasion detection, FRE foundation triggers). Every live-audio session captured real depo speech and routed it to the analyzer.
The vascular-surgery expert depo was the highest-impact case — Daubert vulnerability 75/100, 23 cross-exam questions, and 8 PubMed literature citations the analyzer pulled in real time on ischemia, myoglobin, and intimal-tear topics. Real PMIDs. The kind of thing that would take a human attorney 30 minutes per chunk to pull manually.
Where live audio underperformed CC — videos 3, 5, 6 caught zero admissions in the 3-minute live-audio window. Three reasons:
- Real-time-only by design. Live mode catches what plays in the 3-minute window. The CC pre-screen surveyed the FULL deposition (165 minutes for the pediatrician depo) and surgically picked the top-5 highest-signal chunks. Apples to oranges by definition.
- ASR errors on medical jargon. Deepgram occasionally mistranscribes "Doppler" → "doctor" or "intimal tear" → "internal terror," which suppresses signal detection.
- Non-clinical preamble heuristic. Currently evaluates each chunk in isolation; a procedural sentence after 60 seconds of clinical Q+A can suppress analysis. (Open issue, fix queued.)
The silent production bug this test caught
Two hours into testing, every analyze call across the entire production user base started returning {"error": "Analysis unavailable"} for every expert dimension — but the health endpoint stayed green. The only signal was firing a real chunk and seeing all-empty results.
Root cause: Anthropic Claude Haiku 4.5 began wrapping responses in ```json ... ``` markdown fences despite the prompt's explicit "no markdown" instructions. The defensive parser used lastIndexOf('{') which picks the INNERMOST nested brace — broken for the analyzer's 11-key nested schema ({"medical": {...}, "daubert": {...}, ...} — the inner-most { would be inside literatureHits[0], not the outer object).
Fix shipped in commits 8af2a89 and 13e5977: strip markdown fences first, then use indexOf('{') + lastIndexOf('}') for the OUTER brace span. Plus pinned Haiku to dated revision claude-haiku-4-5-20251001 for stability, and added a CI smoke test that fires a known-good chunk against /api/analyze-smoke on every main-branch deploy.
This is the kind of bug a smart competitor would never tell you about — a silent quality regression that affected real customer sessions. We're flagging it because reproducibility means showing the failures, not just the wins.
Sample output: pediatric neurologist depo, chunk 0
From the 160:06 mark of the pediatric neurologist depo. Witness is a treating physician facing wrongful-death cross-examination over a missed aneurysm in a teenage patient (Jonathan).
"Examination of Jonathan — did you actually read your report? The report? She dictated it. I'm sorry. Dave, I mean, I don't recall. I usually do it same day or the next day, but I don't remember what... There was that referral slip for the CT scan. I may have misheard you in this — but did you actually send a copy of that over to Children's ahead of time? I fax it the moment I write an order for imaging — we make a photocopy in our office and we fax it to Children's Hospital right away. Okay, do you have any confirmation of that fax being sent? I don't need it and I don't get it. Okay, so you don't keep anything like that in your office? Yeah. Okay, but it's your understanding it would've been sent that day, is that correct? All right, how long after you dictated your report did you send it to the pediatrician? I don't recall. Do you have a custom and practice to do within a certain time frame? Either within a reasonable time — could be the same day, it could be longer than that... I don't recall, I don't recall, I don't recall."
What the analyzer flagged in this single 120-second chunk:
- Daubert vulnerability: 75/100
- Admission: "I don't need it and I don't get it [fax confirmation]" — the witness concedes there is no documentary trail for the imaging referral, undercutting the standard-of-care defense.
- Evasion pattern: "I don't recall" repeated 6 times in 90 seconds. Pattern flagged for escalation.
- Breach element advanced: failure to maintain referral documentation — a documented standard-of-care violation in pediatric neurology.
- FRE foundation triggers: 803(6) Business Records, 613(a) Prior Inconsistent Statements, 702 Reliability of Witness Opinion.
- Top cross-exam question generated: "You stated you 'usually' read your dictated report 'same day or the next day,' but you 'don't remember' doing so in Jonathan's case — do you have any documentary evidence, such as a timestamp on your dictation system or an entry in your office EHR, that would confirm whether you actually reviewed your own report before sending it to International Pediatrics?"
5 cross-exam questions generated for this chunk alone. A senior plaintiff's attorney would generate similar questions in 15-20 minutes of prep work; the analyzer produced them in 7 seconds.
Reproducibility
Anyone with the four ingredients can rerun this test:
- Public YouTube video IDs:
rb1GiUAX6aw,9hjNW-eHvlg,Pq6o1fJFrmI,Emc1TaC3ceU,9g1V1RV8F2A,gUTPIOPkDHU,4dZ9p8nC24U - youtube-transcript-api (Python, MIT-licensed)
- Anthropic Claude Haiku 4.5 API access (any developer key)
- Our analyzer prompt — exposed via the public
/courtroom-ai/api/analyzeendpoint (or copy-pasted from our public methodology essay)
Re-run, compare your numbers to ours. If they're materially different, file an issue.
Product improvements queued from this test
- Auto-load YouTube CC transcript as
chartContext. When a user pastes a YouTube URL into the prep panel, fetch the transcript viayoutube-transcript-apiand feed it to the analyzer as chart context. Then "witness now says X but YouTube transcript says Y" cross-references fire automatically. (Backend already acceptschartContext; frontend wiring queued.) - Tune the non-clinical preamble heuristic to inspect a rolling window, not the chunk in isolation. A procedural sentence after 60 seconds of clinical Q+A shouldn't suppress analysis.
- Per-specialty Deepgram keyword boosts for vascular surgery (pulses, ischemia, fasciotomy, myoglobin) and pediatric neurology (saccular aneurysm, anterior communicating artery). Reduces ASR errors on medical jargon.
- Hybrid Daubert score — combine the regex pre-screen with model output so a low-content live chunk that follows 60 seconds of high-signal content still gets context-aware analysis.
Disclosure: I'm the founder of MedLegal AI. The reproducibility commitment described in our methodology essay applies to this test too. The dual-pass numbers above are unedited; the production bug we caught is documented because hiding it would violate the commitment. The point of publishing failures, not just wins, is so attorneys can evaluate vendor claims with their eyes open.