When the Primary AI Model Fails, What May the System Still Do?

A support assistant can retrieve the refund policy, but its primary model is timing out. Routing quietly sends the request to a smaller model, which writes a plausible reply and calls the refund tool. The dashboard stays green; an unapproved system made the decision.

Availability was preserved. The operating contract was not.

The useful question during an AI outage is not simply “Do we have a second model?” It is: when each dependency is slow, stale, unavailable, or untrusted, which functions may continue, which must shrink, which should wait or transfer to a person, and which must stop? That decision belongs in a tested degraded-mode contract before an incident.

A response is not proof that the service is healthy

Traditional availability checks often ask whether a process answers, an endpoint returns 200, or a dependency stays below a latency threshold. Those signals matter, but an AI capability can remain reachable while losing an essential condition for safe use. Retrieval may be stale. Citations may be missing. a policy engine may be unreachable. An audit sink may be dropping records. The model may accept prompts but fail a protected evaluation.

Losing generation does not always require losing the whole journey. A system may still authenticate the user, show verified sources, accept a case for later, or transfer evidence to an operator. The goal is the largest pre-authorized useful service whose evidence, permissions, and consequences remain approved.

This makes degraded mode different from disaster recovery. The ZharfAI guide to recovery consistency points asks how to restore one coherent system after disruption. This guide asks what the live system is allowed to do while one or more required conditions are missing.

What authoritative guidance establishes

The sources do not prescribe one AI failover architecture, but they establish important constraints.

The NIST AI Risk Management Framework Core calls for contingency processes for high-risk third-party AI failures, safe-failure evaluation, viable non-AI alternatives, and assigned mechanisms to deactivate systems that depart from intended use. It connects monitoring with override, incident response, recovery, and change management. AI RMF 1.0 is under revision; these are the published Core outcomes available on this article’s review date.

NIST SP 800-34 Rev. 1, published in 2010, remains a useful general contingency-planning foundation: use business-impact analysis to set requirements and priorities, document recovery procedures, and exercise the plan. It is not AI-specific and does not validate model substitution.

Cloud guidance adds engineering detail. Google’s graceful-degradation principle, last reviewed December 30, 2024, recommends throttling, early request shedding, partial-error handling, monitoring, and overload simulation. AWS’s current graceful-degradation guidance says failure paths should preserve critical functions, be simpler than primary paths, and be tested. Transaction consistency may outrank surface availability.

Those are verified positions. The contract below is ZharfAI’s synthesis for AI services, where a “backup” model can change behavior, tool selection, language quality, context limits, data routes, and risk.

Define the minimum useful function before the outage

Start with a business function, not a component. “The model is available” is not a service definition. “An authenticated customer can see the current, applicable refund rule with its source” is.

For each user journey, separate four layers:

• Essential invariant: what must never become false, such as “no refund without an eligible order and authorized approval.”
• Minimum useful function: the smallest worthwhile outcome, such as opening a case and showing verified policy passages.
• Optional enrichment: summaries, personalization, ranking, or recommendations that may disappear first.
• Forbidden degraded behavior: actions barred when evidence, logging, policy, identity, or review is missing.

This exposes false dependencies: conversation may need generation, while case intake does not. It also exposes hard ones. Without current policy, tenant identity, or the effect ledger, a consequential action may have no safe reduced form.

Map conditions, not just services

A dependency map should include more than vendor endpoints. Record the conditions that make each function admissible:

• identity, tenant and delegated authority;
• current source data, retrieval index and provenance;
• model, prompt, tool schema and policy versions;
• output validation, content controls and evidence requirements;
• approval capacity and operator competence;
• audit, idempotency, transaction and reconciliation stores;
• provider region, data-processing route, quota and latency budget;
• downstream systems of record and external effect channels.

For each edge, name the failure modes: unavailable, slow, stale, partial, inconsistent, overloaded, compromised, revoked, or outside the validated envelope. “Healthy/unhealthy” is too coarse: an index can be available but 18 hours behind, or a model can answer without required citations.

Set freshness and completeness requirements. A cached office address may be shown with a timestamp; a cached fraud block may require current confirmation or a closed gate.

Use function-level health, not one green light

Infrastructure probes answer narrow questions. Current Kubernetes probe documentation, last modified June 30, 2026, distinguishes startup, liveness, and readiness: a process may be alive yet not ready for traffic, and a failed readiness probe can remove it from service without forcing an immediate restart. The same separation is valuable above the container layer.

Create a capability health vector rather than one global status:

identity=ready
source_freshness=ready
retrieval=ready
generation=impaired
policy=ready
approval_capacity=constrained
audit_sink=ready
effect_executor=halted

Derive the allowed mode from policy over those signals. Do not let the model interpret its own health or decide whether missing controls are optional. Health evidence should have timestamps, source IDs, thresholds, and hysteresis so one transient timeout does not cause rapid switching between modes.

Give each failure one explicit disposition

The contract should choose among a small set of named responses.

“Serve stale,” “use a smaller model,” and “skip the check” are not dispositions by themselves. They are proposed mechanisms. Each still needs a bounded function, reason code, evidence, user message, expiry, and exit condition.

The abstention contract applies at the claim or action level. Degraded mode applies to the operating system around many claims and actions. They meet when a missing condition forces answer, seek-evidence, defer, or refuse behavior.

Treat an alternate model as a new release candidate

Models that accept the same API shape can still differ in instruction following, tool calls, language coverage, refusals, context, latency, safety policy, and data location. Even a same-provider upgrade can change behavior without an application-code change.

Therefore, automatic routing to an alternate model is safe only for task classes in which that exact alternate has already passed the relevant evaluations with the exact prompt, retrieval, tool, validator, and policy bundle. Qualify it for named functions, not for “all traffic.” A model approved to summarize a cited policy may still be unapproved to classify refund eligibility or construct tool arguments.

The AI system change-control guide explains why the release unit is often larger than the component that changed. Failover does not waive that coupling. If the alternate changes the system’s behavior, it is a release that happens under incident pressure—the worst time to discover an untested dependency.

Keep the degraded path simpler than the primary path

A failure design that adds a second provider, new credentials, cross-cloud networking, adapters, moderation, and caches creates a complex path used only during an incident. More parts create more unobserved states.

Prefer a reduced path that removes work:

• return verified records or cited source passages without synthesis;
• disable personalization, long-context processing, optional tools and writes;
• accept a durable request without claiming completion;
• use a deterministic template for a narrow status message;
• expose a qualified human handoff with the evidence already collected;
• shed low-priority traffic before it consumes scarce model or reviewer capacity.

Google and AWS use graceful degradation to preserve critical function while dropping noncritical work. AI adds one more requirement: the reduced output must not look indistinguishable from the full-quality, fully evidenced service. Users and downstream systems need a machine-readable mode and a visible explanation.

Work a refund assistant through the contract

Consider an assistant with three functions: answer refund-policy questions, recommend eligibility, and execute an approved refund.

The normal path authenticates the customer, retrieves the order and current regional policy, asks the primary model to assemble a cited explanation, validates required fields, and sends eligible cases to a separate approval and effect service.

Now the primary model’s timeout rate crosses the declared threshold. Identity, orders, policy retrieval, and audit remain healthy. The mode controller opens the generation circuit and selects the pretested evidence-only mode. The interface displays the current policy passages and order facts with timestamps. It may accept a request for review, but it does not state eligibility, create a persuasive explanation, or expose the refund button.

A smaller model can summarize short English passages but falls below the floor on Persian exceptions and tool arguments. It may produce only an optional, evidence-linked English summary. Persian users receive sources and human review; no locale silently receives weaker decision quality.

If the audit sink then becomes unavailable, the system stops accepting review requests because it cannot prove receipt and ownership. Read-only display may continue if policy permits. If current policy retrieval becomes stale beyond its limit, even evidence-only answers close; the system provides status and a case channel without quoting an expired rule.

Nothing here requires a model to recognize the incident. Deterministic policy maps measured conditions to allowed functions.

Make human transfer a capacity-backed service

“Send it to a human” fails when the queue lacks an owner, evidence, or capacity. A transfer contract needs eligibility, priority, maximum age, receipt, assigned role, hours, and a full-queue disposition. Degraded AI output must not become the reviewer’s unquestioned starting point.

The European Union’s AI Act, Regulation (EU) 2024/1689, adopted June 13, 2024 and published July 12, requires proportionate human-oversight measures for high-risk systems. Article 14 includes the ability, where appropriate, to disregard, override, or reverse output and to interrupt a system so it reaches a safe state. That legal duty applies to the Regulation’s defined scope; outside that scope, the design lesson still holds: an operator needs authority, information, time, and a real stop mechanism—not merely a queue label.

Control entry, operation, and recovery

Use an explicit state machine such as normal → impaired → degraded → halted → recovering → normal. Each transition needs:

• a measured trigger and observation window;
• the exact functions enabled and disabled;
• a policy version and responsible owner;
• a user-visible and machine-readable mode code;
• queue, time, cost and exposure limits;
• evidence recorded before and during the transition;
• an exit test, not only “the provider is back.”

Recovery deserves special caution. Close circuits gradually, drain or expire queued work, invalidate unsafe cached outputs, reconcile external effects, and verify that the restored bundle matches the approved release. A few successful requests prove reachability, not recovered behavior. Use a bounded canary and protected evaluations before full restoration.

The UK NCSC’s secure AI operation guidance, published November 27, 2023, recommends monitoring system inputs, outputs, performance, sudden and gradual behavior changes, and treating significant model, data, or prompt updates as new versions. Those observations should drive both incident entry and recovery evidence.

Test the missing condition, not a theatrical outage

An HTTP 500 exercise misses dangerous cases. Test late responses, partial streams, missing citations, stale retrieval, exhausted quota or reviewers, one unavailable locale, audit loss, compromised credentials, and a reachable model below its quality floor.

For each scenario, verify:

1. the correct mode activates within the allowed time;
2. prohibited tools and outputs become technically unreachable;
3. in-flight work reaches a known state without duplicate effects;
4. users see accurate limits and a usable next action;
5. queues, caches, retries and human workload stay within bounds;
6. recovery cannot reintroduce stale work or mixed versions.

Test with production-like identities, policies, languages, data age, traffic, and downstream failure. A fallback exercised only in a diagram is an undocumented system.

Measure preserved value and prevented harm

Ordinary uptime can reward unsafe continuity. Score degraded mode separately:

• correct mode selection: time to detect, false transitions and mode flapping;
• preserved function: share of essential journeys completed within the reduced contract;
• boundary integrity: forbidden-action attempts, missing receipts and policy bypasses;
• truthfulness: stale or uncited claims, incorrect mode labels and user misunderstanding;
• human sustainability: transfer acceptance, queue age, abandonment and reviewer load;
• recovery quality: reconciliation exceptions, duplicate effects, cache invalidation and time to validated normal service.

Also record the denominator: eligible requests in each mode. A 99% “success” rate is meaningless if the system quietly excluded Persian cases, high-value transactions, or requests requiring current evidence.

The pre-incident decision sheet

Before enabling automatic failover, require one signed decision sheet per important journey:

• What is the minimum useful function, and which invariant must survive?
• Which conditions authorize normal, reduced, transfer, and closed modes?
• Is the alternate truly equivalent, or is it qualified only for named tasks?
• Which data, tools, writes, locales and user groups disappear in each mode?
• How are limits shown to users and downstream systems?
• Can the effect boundary enforce the restriction without trusting the model?
• What happens to in-flight and queued work?
• Who owns the mode, the human queue, the incident, and the return to normal?
• Which exercise proves the path under realistic load and partial failure?
• Which evidence must exist before normal service resumes?

A backup endpoint is inventory. A degraded-mode contract is an operating decision. It accepts that partial failure will happen, preserves only the value that can still be defended, and makes reduced authority visible. If the organization cannot state what the AI may do without its primary model, it has not designed continuity; it has designed an improvised release during an outage.

Source notes — reviewed 1 September 2026

• NIST, AI RMF Core: contingency for third-party AI failures, safe failure, non-AI alternatives, monitoring, override, recovery, and deactivation.
• NIST, SP 800-34 Rev. 1: business-impact analysis, contingency requirements, recovery procedures, and exercises; published in 2010 and used here as general planning guidance.
• Google Cloud, Design for graceful degradation: throttling, early request shedding, partial errors, monitoring, and overload testing; last reviewed December 30, 2024.
• AWS, Implement graceful degradation: protect critical functions, simplify and test failure paths, and choose requirements that must survive.
• Kubernetes, Liveness, readiness, and startup probes: separate process life, readiness for traffic, and startup state; last modified June 30, 2026.
• UK NCSC, Secure operation and maintenance: monitor inputs and behavior and evaluate consequential updates; published November 27, 2023.
• European Union, Regulation (EU) 2024/1689: Article 14 human-oversight and safe-interruption requirements for high-risk AI systems within the Regulation’s scope.