Model Fallback Routing for Clinical Dental AI: Keeping the Agent Working When the Primary Model Is Throttled
What happens at your front desk when the model behind your AI phone agent returns a ThrottlingException at 7:58 on a Monday morning, just as the overnight voicemail backlog and the first same-day emergency calls arrive? If the agent has only one model wired into it, callers get dead air and your front-desk team inherits every call at the same moment.
Model fallback routing is the engineering practice that prevents that outcome. It treats the primary model as one tier in an ordered chain, uses evals to prove that each lower tier can handle the work it will be given, and decides in advance what the agent stops doing on a smaller model. The result is a practice that slows down during an outage instead of going dark.
Model fallback routing is an ordered chain of BAA-covered models plus a scripted floor. When the primary is throttled or down, the agent shifts to a tier proven by evals and keeps answering patients.
Why A Single-Model Agent Is A Single Point Of Failure
Most clinical AI deployments in dentistry start with one model ID hardcoded in the application, usually the frontier model that scored best in the pilot. That said, a production agent that answers phones, books hygiene visits, and drafts insurance verification notes depends on a provider quota, a region, and a model version, and the practice controls none of the three.
Keep in mind that throttling is how a shared service is expected to behave under load. On Amazon Bedrock, on-demand throughput is governed by per-account, per-region quotas on requests and tokens per minute, and a request that exceeds them is rejected with a ThrottlingException instead of being queued.
Sooner or later, a dental agent in production will run into failure modes that include but are not limited to:
- Quota throttling. A burst of concurrent calls pushes the account past its tokens-per-minute limit. The SDK's built-in retries absorb short spikes, but under sustained pressure those retries turn into a wall of latency.
- Regional degradation. A single AWS region shows elevated 5xx error rates or latency while the service stays technically available. To the caller on the line, that looks the same as an outage.
- Model retirement and version changes. Providers retire model versions on published schedules, and an agent pinned to a retired version fails outright. We cover the slower version of this problem in our guide to model drift monitoring for clinical AI.
- Upstream dependency pressure. Sometimes the model is healthy and the practice management system is the bottleneck. Router design has to account for PMS API rate limiting so that the model failover does not get blamed for a database constraint.
All of these end the same way when there is only one model in the path. The agent stops, and the stop shows up in the patient experience before it shows up on anyone's dashboard.
Compliance Comes Before Capability In The Fallback Chain
Before anyone debates whether a fallback model is capable enough, the first question is whether it is allowed to see the data. Every model in the chain receives the same appointment context, patient names, and insurance details as the primary, so every endpoint in the chain must be covered by a signed BAA and sit inside the same PHI handling boundary.
In practice, this rules out the most common shortcut: failing over from a BAA-covered endpoint to a consumer API key someone added during the pilot. Note that HIPAA eligibility attaches to a specific service and account configuration, so the fact that a model shows up in a console tells you nothing about whether it may receive PHI.
Before a tier enters the routing table, it has to pass compliance checks that include:
- BAA coverage. The provider and the specific service used by the tier are covered under the executed BAA. We walk through the full scope in HIPAA requirements for clinical AI in dental practices.
- Private transit and encryption. The tier is reached over the same private path as the primary, such as a VPC endpoint through PrivateLink, with KMS-encrypted logs. Cross-region inference stays within US regions.
- Audit parity. CloudTrail and the application audit trail record the tier name and pinned model version on every request. With that record, an auditor can reconstruct which model touched which patient record.
- No training retention. The provider terms for that tier prohibit using prompts and completions for model training. This needs a separate check for each tier, even when every tier comes from the same vendor family.
For most practices, the simplest compliant chain keeps every model tier on one BAA-covered service in one AWS account. Our breakdown of Amazon Bedrock for clinical AI explains why that setup cuts the review work for each new tier to a configuration check instead of a new vendor assessment.
Clear every tier for PHI before it enters the routing table. If a fallback has never been reviewed for BAA coverage, its first real use will come on the morning when nobody has time to check it.
How Tiered Model Routing Works
A tiered router sits between the agent's orchestration layer and the model endpoints and chooses a tier for each request, using health signals and the capability the task requires. Model IDs live in configuration instead of code, so promoting, demoting, or swapping a tier is a versioned config change that needs no redeploy.
A typical four-tier chain for a dental front-desk agent looks like this:
| Tier | Example endpoint | Role | Tasks permitted |
|---|---|---|---|
| 1 — Primary | Claude Sonnet on Bedrock, provisioned throughput, us-east-1 | Full capability | All tasks |
| 2 — Alternate capacity | Same model via a US cross-region inference profile | Absorbs throttling without changing behavior | All tasks |
| 3 — Smaller model | Claude Haiku on Bedrock, same account and BAA | Lower latency and cost, narrower capability | Only tasks it passed in parity evals |
| 4 — Deterministic floor | Scripted voice flow with message capture | No model dependency | Take messages, route emergencies, confirm callbacks |
The order of the tiers matters. Tier 2 adds capacity without changing behavior, so the router should use it up before dropping to a different model size or family.
Tier 4 deserves more attention than it usually gets. A scripted floor that collects the caller's name, callback number, and reason for calling, and flags anything that matches an emergency pattern for an immediate staff callback, changes the story from an AI outage to a slower, simpler agent for twenty minutes.
Exhaust the same model on alternate capacity first. Then drop to a smaller model limited to tasks it passed evals on, and finally to a scripted floor that captures messages and flags emergencies.
What Should Trigger A Failover?
Failover triggers should be measurable, scoped to each tier, and reversible. A clinical agent should watch these signals:
- Throttling responses. A ThrottlingException or HTTP 429 counts only after the bounded retry budget, with exponential backoff and jitter, is used up. Failing over on the first rejection wastes Tier 1 capacity that would have recovered within a second.
- Error rate. ServiceUnavailable and other 5xx responses above a set threshold, for instance 5% over a rolling 60-second window. A single error should never move traffic.
- Latency. P95 time-to-first-token above the budget your voice pipeline was designed around. We cover how to set that budget in chairside AI latency.
- Validation failure rate. A jump in structured outputs failing schema validation means the tier is degraded even while it returns 200s. This is why the router reads from the same checks described in structured output validation for clinical AI.
We wrap each tier in a circuit breaker with three states (closed, open, and half-open), so a tier that trips stops receiving traffic right away and is probed with a small share of requests before it is trusted again. In addition, every write carries an idempotency key, so a retried booking cannot land on the schedule twice when the first attempt actually succeeded upstream.
Fail over once the retry budget is spent on throttling, or when 5xx rates or P95 latency cross a set threshold. A spike in schema validation failures counts too, tracked per tier by a circuit breaker.
Output-Parity Evals: Proving The Fallback Can Do The Job
A fallback tier that has never been evaluated against the primary's workload is a guess. Output-parity evals answer one narrow question: on the tasks this tier is allowed to run, are its outputs equivalent to the primary's within a tolerance the practice has agreed to?
The suite comes from the agent's own traffic. We take representative, de-identified tasks (reschedule requests, new-patient intake, eligibility questions, post-op instruction lookups) and run them through every tier in shadow mode, scoring each tier against the primary and against labeled expected outcomes, as described in our framework for clinical AI evals.
Parity is measured task by task instead of as one blended score, and the dimensions we track include:
- Structural parity. Schema pass rate for booking payloads, CDT codes, and tooth references. A smaller model that writes tooth 3 in the wrong system fails here, which is why the suite encodes the conventions covered in tooth numbering standards.
- Decision parity. How often the tier agrees with the primary on whether to book, escalate to staff, or transfer as an emergency. A missed emergency triage is a hard fail no matter how well the tier scores elsewhere.
- Tool-call parity. Correct tool selection and correct arguments against the practice management system. For practices on Open Dental, that means the same endpoints and parameters described in our Open Dental API guide.
- Safety parity. Smaller models can respond differently to adversarial input, so the suite includes the cases from our prompt injection testing for clinical AI. A tier that follows an injected instruction the primary refused does not enter the routing table.
- Latency and cost. P50 and P95 latency and cost per completed task for each tier. These numbers determine whether Tier 3 is also worth running on a few low-risk tasks during normal operation.
What the suite produces is a per-task capability map. For example, a smaller tier might hold 99% schema parity on booking and rescheduling but fall into the high 80s on coordination-of-benefits questions across two dental plans, in which case it handles the first set during degraded mode and hands the second to a person.
Re-run the suite whenever the model version changes in any tier, and pin versions in configuration so eval results stay tied to the model actually serving traffic. After all, a capability map built against last quarter's model version says very little about this quarter's.
An output-parity eval runs real, de-identified agent tasks through each fallback tier in shadow mode. It scores schema, decisions, and tool calls against the primary and yields a per-task capability map.
Degraded-Mode Behavior At The Front Desk
Degraded mode is as much a product decision as an engineering one, and it should be written down before the first outage. The goal is that a patient calling during a throttling event still has a useful conversation, and the front-desk team knows exactly which work has come back to them.
A written degraded-mode policy maps each front-desk task to its behavior at each tier:
| Task | Normal (Tiers 1–2) | Degraded (Tier 3) | Floor (Tier 4) |
|---|---|---|---|
| Booking and rescheduling | Full booking with production-aware slot selection | Books from open slots without optimization | Captures the request for a staff callback |
| Insurance verification | Real-time eligibility and benefits summary | Queued for a batch run after recovery | Captures carrier and member ID |
| Emergency triage | Assesses and books same-day | Routes to on-call staff immediately | Routes to on-call staff immediately |
| Treatment-plan questions | Answers from the accepted plan | Defers to staff | Captures the question |
| Clinical note drafting | Full drafting | Paused and queued | Paused and queued |
Unoptimized booking in degraded mode gives up some of the production gains described in AI scheduling optimization, and that is an acceptable trade for twenty minutes. Likewise, pushing eligibility checks into a batch run delays the workflow covered in AI insurance verification without losing any of it.
Emergency routing should become more conservative as the agent drops through the tiers. When the agent is running on a smaller model, the threshold for handing a call to a person should go down, because missing a swollen jaw or a post-extraction bleed costs far more than one extra staff callback.
The front-desk experience during degraded mode should include:
- A staff-facing status banner. The dashboard shows the active tier and when it started, for example "Degraded: Tier 3 since 8:02 AM," along with the tasks that have been handed back. Staff should never have to guess why the agent declined something.
- An honest caller script. The agent tells the caller the system is running more slowly than usual and gives a firm callback window. That keeps patients' trust better than a smaller model attempting a task it failed in evals.
- A durable deferred-work queue. Every paused task, such as an eligibility check or a note draft, is written to SQS with a dead-letter queue and replayed after recovery. Idempotency keys keep the replay from creating duplicates.
These rules belong in the same runbook as the rest of your AI phone agent operations. That way the office manager and the on-call engineer are reading the same page when the banner turns amber.
Keep booking from open slots and capture callback details for everything else. Route emergencies to staff faster than normal, and queue paused work for replay after recovery.
Failing Back Without Flapping
Recovery needs as much design as failover. A router that returns to the primary as soon as one probe succeeds will bounce between tiers during a partial outage, and each bounce changes the agent's behavior mid-shift.
We use hysteresis: the primary has to pass a sustained window of healthy half-open probes, for instance five consecutive minutes under the error and latency thresholds, before it gets full traffic back. Furthermore, a conversation that is already in progress finishes on the tier where it started, so the caller never hears the agent's capability change between one sentence and the next.
Once the primary is back, the deferred-work queue drains at a controlled rate. Otherwise the replay itself can create a second throttling event, against the model or against the practice management system.
How Do You Know Your Fallback Routing Is Working?
A fallback path that only runs during real outages has, by definition, never been tested. Accordingly, we schedule game days in staging, and with the practice's agreement, controlled drills in production, that force each tier open and measure what happens.
A useful game-day checklist covers:
- Forced throttling. Inject 429 responses at the router and confirm that traffic moves to Tier 2 within the expected window. Alarms should fire before a person notices anything.
- Forced model loss. Disable Tiers 1 and 2 and confirm that Tier 3 sticks to its capability map. Any task outside the map should go to staff, with no attempt at it.
- Total model loss. Disable every model tier and confirm that the scripted floor answers, captures details, and routes emergency keywords to on-call staff. This is the drill most teams skip, and it is the one that protects patients.
- Replay and audit. Restore all tiers and confirm the deferred queue drains with no duplicate bookings. The audit trail should show the tier and model version for every request made during the drill.
Track tier distribution as a standing metric. If Tier 3 serves 0.5% of requests one week and 6% the next, that is an early quota or capacity signal to investigate before it turns into a Monday-morning event.
Cost belongs on the same dashboard. Sampling 5% of traffic through a smaller model for continuous parity monitoring on a workload that spends $2,000 a month on the primary adds roughly $10 to $35 a month, depending on the price gap between the two models.
Building For A Model You Cannot Always Get
Access to frontier models is a supply relationship like any other: quotas change, prices move, and model versions retire on the provider's schedule. An agent that degrades gracefully across tiers is protected against all three, and the same config-driven routing makes it simple to add a new tier once a better or cheaper model clears your evals.
Remember that the router, the parity suite, and the degraded-mode policy are three parts of one system. Each one is weak alone, and together they keep a practice's agent answering patients on the day the primary model is unavailable.
Frequently Asked Questions
Does a fallback model need its own BAA?
Coverage must extend to every endpoint that receives PHI. Two models on the same BAA-covered service and AWS account can share that coverage, but a model reached through a different provider or API key needs its own review before it enters the chain.
Should the fallback model come from a different provider?
A second provider protects against an outage across one provider, but it doubles the BAA, logging, and eval work. Most practices start with tiers from the same provider across regions and model sizes, then add a second provider once parity evals justify it.
How often should output-parity evals be re-run?
Re-run them on every model version change in any tier, on every prompt or tool-schema change, and on a fixed monthly cadence to catch silent drift on the provider's side. Pinning versions in config keeps results tied to what is serving traffic.
Will patients notice when the agent is in degraded mode?
Some will, and the agent should tell them plainly. A short script saying the system is running more slowly, with a firm callback window, keeps trust better than a smaller model attempting tasks it failed in evals.
How long should the agent retry before failing over on throttling?
Two or three attempts with exponential backoff and jitter, all inside the caller's latency budget. Past that point, another retry costs the caller more silence than a tier switch costs the practice.
Plan Your Fallback Chain Before The Next Monday Rush
If you are running a dental AI agent on a single model ID and want a second set of eyes on the failure path, the NexV team designs and operates HIPAA-grade clinical agents across phone, scheduling, PMS, and insurance workflows. We know what happens to a front desk when the model goes quiet, and we build so that it does not go quiet.
Reach out for a working session, and we will map your routing tiers, build the first output-parity suite from your own de-identified traffic, and leave you with a written degraded-mode runbook your front desk can follow. Your agent should slow down during an outage, not stop.