How We Test Clinical Dental AI Before It Touches a Patient Record: Eval Harnesses and Regression Gates
You probably think of testing clinical AI the way you'd test a billing integration — feed it a handful of sample cases, confirm the outputs look right, and ship. However, an agent that reads a periapical radiograph, drafts a clinical note, or suggests a CDT code fails in a way a billing script never will: quietly, plausibly, and one patient at a time.
At NexV we treat every clinical model the way an SRE treats a production service — guarded by a golden dataset, an automated eval harness, and a regression gate that can block a release on a single dropped percentage point. This is the QA layer sitting underneath everything else we've written about running clinical AI on Bedrock and keeping it inside a HIPAA boundary.
NexV tests clinical dental AI with three layers before it touches a patient record: an adjudicated golden dataset of clinician-verified cases, an automated eval harness that scores every candidate build against ground truth, and a regression gate that blocks any release where sensitivity, specificity, or calibration drops below the last shipped version.
Why Silent Degradation Is the Failure Mode That Should Scare You
A crashed service pages someone at 3 a.m. A clinical model that quietly gets worsemodel drift monitoring does not — it keeps returning confident, well-formatted answers that are now wrong slightly more often than they were last week.
That gap between "looks fine" and "is fine" is where clinical AI does its real damage. A caries-detection model that drifts from 92% to 86% sensitivity emits no error, throws no exception, and trips no uptime alarm — it just misses more lesions.
Silent degradation is when a clinical model's accuracy quietly drops without any error or alert — outputs stay well-formatted and confident while becoming wrong more often. It is the dominant failure mode for clinical AI because nothing crashes, so only a measurement against known-correct cases can catch it.
Degradation has many sources, and most of them are invisible from inside the application. A model-version bump, a changed system prompt, a re-indexed retrieval store, or schema drift in the upstream chart data can each move accuracy without changing a single line of your own code.
Keep in mind that the model is only one moving part. The prompt, the retrieval layer, the tool definitions, and the input distribution all drift independently — which is exactly why a one-time launch validation tells you almost nothing three months later.
What a Golden Dataset Is — and Why Ours Is Adjudicated, Not Scraped
A golden dataset is a fixed, version-controlled collection of real clinical inputs paired with known-correct outputs — the ground truth every candidate build is measured against. For NexV, that means de-identified radiographs, charts, and note transcripts where the right answer has already been established by a clinician.
The word that matters is adjudicated. A single dentist's read is an opinion, so each case in the set is labeled independently by multiple clinicians, and disagreements are resolved by a senior reviewer before the case earns its ground-truth label.
A golden dataset is a fixed, version-controlled set of real clinical inputs paired with clinician-verified correct outputs. NexV builds it from de-identified cases labeled independently by multiple dentists, with disagreements resolved by a senior reviewer — so the benchmark reflects adjudicated ground truth, not one practitioner's opinion.
Coverage is deliberate, not random. We stratify the set so that rare-but-dangerous cases — periapical lesions, vertical root fractures, interproximal caries hiding under existing restorations — are over-represented relative to how often they appear in a normal day's charts.
This matters because a model can score 95% on an unstratified sample and still fail catastrophically on the 3% of cases that carry the most clinical risk. The golden set is engineered to make those cases impossible to average away.
Every case is de-identified before it enters the set, and the whole corpus lives inside the same HIPAA boundary and BAA coverage as production PHI. A golden dataset is still patient data, and we treat it that way.
How the Eval Harness Runs Every Candidate Build
The eval harness is the automated runner that takes a candidate build, executes it against the entire golden dataset, and scores the outputs against ground truth with no human in the loop. It is the difference between "we spot-checked it" and "we measured it against 4,000 adjudicated cases in eleven minutes."
Every run is reproducible. The harness pins the model version, the prompt template, the retrieval index snapshot, and the dataset version, then records all four in the result — so a score is always traceable to the exact configuration that produced it.
An eval harness is an automated runner that executes a candidate AI build against an entire golden dataset and scores every output against ground truth — no human spot-checking. NexV's harness pins the model version, prompt, retrieval snapshot, and dataset version into each result, making every score reproducible and traceable.
Scoring is task-specific, not generic. A radiograph classifier is scored on per-tooth detection; a note-summarization agent is scored on whether every clinically material fact survived the summary; a coding agent is scored on exact CDT-code match plus tooth-number agreement.
For agents that retrieve before they answer, the harness also scores the retrieval step on its own — because a wrong final answer built on the wrong retrieved context is a different bug than a wrong answer built on the right context.
What Metrics a Clinical Eval Actually Tracks
Accuracy is the wrong headline metric for clinical work, because it hides the asymmetry between failure modes. Missing a lesion (a false negative) and flagging a healthy tooth (a false positive) carry very different clinical costs, and a single accuracy number smears them together.
So the harness tracks sensitivity and specificity separately, per task and per finding type. For a screening-grade tool we hold a sensitivity floor and accept some false positives; for a tool that triggers irreversible treatment suggestions, the balance shifts hard the other way.
Clinical evals track sensitivity (catching true positives) and specificity (not flagging healthy cases) separately, never a single accuracy score — because a missed lesion and a false alarm carry very different clinical costs. NexV also tracks calibration, so the model's stated confidence actually matches how often it is right.
We also track calibration — whether a model that says it is 90% confident is actually right about 90% of the time. A miscalibrated model is dangerous even when it is accurate, because downstream automation and clinicians both lean on confidence to decide when to trust it.
Every metric is logged per finding type, so a build that improves overall but quietly regresses on vertical root fractures still gets caught. Aggregate gains hiding a localized regression is one of the most common ways a "better" model is actually worse.
How a Regression Gate Blocks a Bad Build
A regression gate is an automated rule that compares a candidate build's eval scores against the currently shipped version and blocks the release if any guarded metric falls below threshold. It is the mechanism that turns "we have evals" into "a worse model cannot ship."
The gate runs in CI, not in someone's judgment. A pull request that bumps a model version or edits a prompt triggers the harness automatically, and the merge stays blocked until the scores clear every gate.
| One-time launch validation | Regression-gated pipeline |
|---|---|
| Validated once, before go-live | Re-validated on every build, prompt edit, and model upgrade |
| Pass/fail by human spot-check | Pass/fail by automated scoring against ground truth |
| Silent degradation goes unnoticed for months | A single dropped metric blocks the release in CI |
| No record of which config produced which result | Every score pinned to model, prompt, index, and dataset version |
| Regressions found by patients and clinicians | Regressions found by the gate, before deploy |
A regression gate is an automated CI rule that compares a candidate build against the currently shipped model and blocks the release if any guarded metric — sensitivity, specificity, or calibration — falls below threshold. It converts "we have evals" into "a model that scores worse than production cannot deploy."
Thresholds are not all-or-nothing. We set hard floors that block outright and softer bands that require a human sign-off, so a 0.3-point movement inside noise does not halt every release while a real regression still stops cold.
The gate is also where we encode clinical priorities directly. A build can improve five metrics and still be blocked if it regressed sensitivity on a high-risk finding, because that is the trade we have decided we will never make silently.
Why We Pin Model Versions and Re-Run the Suite on Every Upgrade
Model-version pinning means production points at a specific, frozen model build — never "latest" — so the behavior you validated is the behavior you ship. An unpinned model can change underneath you when the provider updates it, and your accuracy can move without any deploy on your side.
When a new model version does land — a Claude or Bedrock model upgrade — we treat it as a candidate build like any other. It does not reach production until it has run the full golden-dataset suite and cleared every regression gate.
Model-version pinning locks production to a specific frozen model build instead of "latest," so provider updates can't silently change clinical behavior. When a new version is available, NexV runs it through the full golden-dataset eval suite and regression gates before promoting it — an upgrade is treated as a candidate, not a default.
This is also how we catch the upgrades that are genuinely better in aggregate but worse on a specific clinical task. A newer general-purpose model frequently wins on note summarization while moving differently on fine-grained radiograph detection, and the suite is what tells us which.
How Shadow Mode Catches What Offline Evals Miss
Offline evals measure a model against cases you already have. Shadow mode measures it against the cases you don't — the live distribution of real charts flowing through the practice, which always contains inputs your golden set never anticipated.
In shadow mode, a candidate build runs on live traffic in parallel with production, but its outputs are logged rather than used. Clinicians never see the shadow output, no patient record is touched, and we compare the two streams for divergence.
Shadow mode runs a candidate clinical model on live traffic in parallel with production, logging its outputs without showing them to clinicians or writing to any patient record. It surfaces real-world divergence and input distributions a fixed golden dataset can't anticipate — catching drift before the build is ever promoted.
Divergence is the signal. Where the shadow build and production disagree, we route those cases to clinician review, and the genuinely hard ones get adjudicated and folded back into the golden dataset — so today's surprises become tomorrow's regression tests.
That feedback loop is the point. A golden dataset that never grows slowly stops resembling the real input distribution, and shadow mode is how we keep the benchmark honest.
How Do You Know Your Clinical AI Is Still Working in Production?
Pre-deploy gates protect the release. They do nothing about the slow drift that accumulates after deploy, as the patient population, the imaging hardware, or the upstream chart schema gradually shifts away from what you validated against.
So production gets its own continuous monitoring: confidence distributions, clinician override rates, and a steady trickle of cases sampled for review and scored against the same metrics as the offline suite. A rising override rate is often the first sign of drift, long before anyone files a complaint.
You know clinical AI is still working by monitoring production signals — clinician override rates, confidence distributions, and a continuous sample of cases scored against the same sensitivity and specificity metrics as the offline suite. A rising override rate is usually the earliest sign of drift, well before any patient-facing error surfaces.
Where This Leaves You
If you are deploying clinical AI without a golden dataset and a regression gate, you do not have a quality problem yet — you have a quality problem you cannot see. The model-choice and retrieval decisions matter, but they sit on top of a measurement layer most teams skip entirely, and the return on clinical AI only holds if the model stays as accurate as the day it shipped.
If you're scoping clinical dental AI and want a second set of eyes on the QA architecture, the team at NexV builds and operates HIPAA-grade clinical agents with eval harnesses and regression gates in production. Reach out for a working session — we'll map your clinical tasks, define the golden dataset they need, and leave you with a gate design that blocks a worse model before it ever touches a patient record.
Frequently Asked Questions
What is the difference between an eval harness and a regression gate?
An eval harness runs a build against the golden dataset and produces scores. A regression gate is the rule that reads those scores and blocks the release if any guarded metric falls below the currently shipped version — the harness measures, the gate decides.
How big does a clinical golden dataset need to be?
Size matters less than coverage and adjudication. A few thousand well-stratified, clinician-verified cases that over-represent rare high-risk findings beat tens of thousands of unlabeled or single-reader cases, because the gate has to be able to catch a localized regression.
Does running evals on patient data create a HIPAA problem?
Only if the data isn't handled correctly. NexV de-identifies every case before it enters the golden dataset and keeps the corpus inside the same HIPAA boundary and BAA coverage as production PHI — a golden dataset is still patient data.
Why not just trust the model provider's published benchmarks?
Provider benchmarks measure general capability, not your clinical tasks on your input distribution. A model that tops a general leaderboard can still regress on interproximal caries or CDT coding, which is exactly what your own golden-dataset suite exists to detect.
How often should the regression suite run?
On every change that can move behavior — every prompt edit, retrieval-index rebuild, and model-version bump — plus a scheduled run against fresh shadow-mode cases. The whole point is that no behavior-affecting change reaches production without clearing the gate.