The Recovery Cut: Rebuilding AI Systems Without Mixing Timelines

A support agent returns after an outage. Its customer database has been restored to 09:40, the object store to 09:55, the vector index from last night's snapshot, and the system prompt from a release deployed at 10:05. The service answers requests and every health check is green. It is also an impossible assembly: some documents do not yet exist in the database, some embeddings describe files that have since changed, and the prompt assumes a tool schema that the restored connector does not support.

The reader decision is therefore not merely “Do we have backups?” It is: for each part of this AI system, should we restore it, rebuild it, or deliberately abandon it—and what common recovery cut makes the resulting assembly safe to serve? A successful restore returns bytes. A successful recovery returns a coherent business capability with known loss, compatible components, reconciled external effects, and evidence that the system is fit to resume.

Define recovery as a business state

Traditional recovery objectives remain necessary. The AWS disaster-recovery guide defines recovery point objective, or RPO, around acceptable data loss and recovery time objective, or RTO, around acceptable downtime. The Google Cloud planning guide, last reviewed in July 2024, similarly starts with business impact and notes that tighter RPO and RTO usually cost more and add operational complexity.

Those objectives do not answer whether separately recovered components agree. An AI capability can meet a database RPO while its retrieval index, prompt, policy, tool registry, and conversation state refer to different histories. ZharfAI's analysis adds a consistency condition: the recovery point is valid only when every retained component can be traced to one compatible cut or safely reconstructed from it.

Write the recovery acceptance statement in business language. For example:

By 12:00, support staff can read all customer cases committed before 09:42, no post-09:42 action is silently repeated, every answer uses release 18's approved prompt and policy, and any uncertain ticket or message is held for reconciliation.

This statement makes loss, timing, version, and external effects reviewable. “Database restored” does not.

Map the state before choosing a backup product

The UK National Cyber Security Centre's secure AI development guidance, published in November 2023, tells teams to identify models, data, prompts, software, documentation, logs, and assessments; track and authenticate them; and retain the ability to restore a known-good state. That inventory is the starting point, but recovery needs relationships as well as a list.

Build a state graph with five classes:

For every node record its owner, tenant boundary, system of record, change rate, retention rule, recovery mode, latest proven restore point, dependencies, integrity evidence, and acceptable loss. Include encryption keys and identity configuration, but do not plan to restore compromised credentials as ordinary data. Reissue them from a clean trust root.

This graph should connect to the AI release passport: recovery needs the exact model, prompt, policy, tool, data, and infrastructure assembly, not just an application commit. It should also connect to the deletion graph, because retention for recovery cannot silently defeat an approved erasure schedule.

Choose a recovery cut, not the newest copy of everything

A recovery cut is the latest boundary at which authoritative state, versioned execution, and known external effects can be made mutually consistent. It may be a database transaction marker, named restore point, event-log offset, release manifest, object-version set, or a bundle of linked markers. Wall-clock time alone is fragile when clocks skew, writes arrive late, or systems record event time and processing time differently.

PostgreSQL's current point-in-time recovery documentation explains how a base backup plus archived write-ahead-log records can restore a database to a chosen target. It also makes the dependency chain explicit: incremental recovery needs earlier backups and the relevant log and history files. That is a database guarantee, not an AI-system guarantee. The recovery bundle still needs to say which object versions, event offsets, release digest, and index recipe correspond to that database target.

Use the temporal distinctions in our time-semantics guide:

• committed by the cut: authoritative state included in recovery;
• known after the cut: later information that may explain the incident but must not be blended into the recovered history without an explicit forward repair;
• effective at the cut: policy, entitlement, or fact whose valid-time interval covered the recovered decision;
• replayed after the cut: a new processing event that must retain the identity of the original business event.

If components cannot share one cut, document the skew and quarantine the affected capability. Do not hide a 09:40 database plus 10:05 index behind the phrase “latest available.”

Decide what to restore, rebuild, or abandon

The three modes solve different problems:

Vector indexes are usually rebuild candidates, not primary backup authorities. Preserve the authorized source objects, chunking and normalization rules, embedding model digest, index settings, access-control metadata, and source-to-vector identifiers. A fast index snapshot can shorten RTO, but it should be accepted only if its provenance matches the chosen cut. Otherwise rebuild and reconcile document counts, tenant partitions, deletion markers, and retrieval probes.

Conversation summaries deserve special scrutiny. If they drive entitlements, commitments, medical context, or financial actions, they are not harmless cache. Either preserve the underlying authoritative conversation under an approved retention rule or force a new confirmation. Do not rebuild a consequential commitment from a model-generated summary alone.

Design a recovery bundle

A useful recovery bundle is a small, integrity-protected manifest that points to the material needed for one recovery cut. It should include:

1. the incident-independent business service and its RTO, RPO, maximum component skew, and acceptable degraded mode;
2. database restore targets, event-log offsets, object-version manifests, and tenant partitions;
3. the approved release passport for code, model, prompt, policy, tool schemas, and infrastructure;
4. recipes and source snapshots for every derived index, feature set, or summary that will be rebuilt;
5. external-effect receipts, idempotency keys, and reconciliation queries;
6. key-reissuance and identity-bootstrap procedures from a clean administrative boundary;
7. validation datasets, invariant queries, adversarial tenant probes, and expected results; and
8. named decision owners for containment, cut selection, degraded service, write enablement, and failback.

Keep the bundle and runbook accessible when the primary control plane is unavailable. Google Cloud's infrastructure-outage guidance, last reviewed in May 2024, warns that business-critical recovery operations should not depend on management-plane changes such as creating a virtual machine or updating an IAM permission. The exact constraint differs by provider, but the design question is portable: can the recovery path run when the system used to configure that path is the system that failed?

Recover in a controlled order

NIST SP 800-34 Rev. 1, published in 2010, separates activation and notification, recovery, and reconstitution, where testing validates the restored system before normal operation resumes. NIST SP 800-184, published in December 2016, adds recovery playbooks, realistic testing, metrics, and continuous improvement after cyber events. Translate that discipline into this sequence:

1. Contain and preserve. Stop automated writes, exports, training ingestion, index refresh, and deletion jobs. Preserve logs and receipts without treating potentially compromised hosts as trusted evidence.
2. Select and approve the cut. Identify the last acceptable business point and the known uncertainty window. Record why a later copy was rejected.
3. Bootstrap clean control. Establish trusted operators, fresh credentials, recovery infrastructure, and immutable manifests. Do not import the incident's identity state blindly.
4. Restore authority first. Recover business systems of record and event histories, then verify counts, constraints, ownership, and tenant separation.
5. Deploy the compatible assembly. Recreate infrastructure and install the exact approved model, prompt, policy, code, and tool contracts for the cut.
6. Rebuild derivatives. Generate indexes, embeddings, summaries, and caches from verified inputs; mark them incomplete until reconciliation passes.
7. Reconcile external effects. Query remote systems by idempotency key or receipt. Classify each effect as committed, absent, reversed, or unknown before retrying.
8. Reconstitute gradually. Exercise the system in isolation, enable bounded reads, then selected writes, then normal traffic. Keep uncertain cases in a manual queue.

High availability does not replace this sequence. Replication may reduce outage time while faithfully copying a bad prompt, poisoned corpus, mistaken deletion, or ransomware encryption. AWS's recovery-strategy guidance distinguishes availability from one-time disaster recovery and notes that replication alone does not protect against corruption or destruction without point-in-time recovery.

Worked example: the support agent at 09:42

At 10:15, a team discovers that an importer began attaching documents to the wrong tenant at 09:43. The support agent subsequently embedded those files, summarized two conversations, opened three remote tickets, and sent one email. The database has point-in-time recovery; the vector service has an hourly snapshot; the ticketing vendor cannot join a database transaction.

The team selects 09:42 as the recovery cut:

The restored system first runs read-only. Cross-tenant canaries, known-case answers, source citations, policy-effective dates, and tool dry runs must pass. Only then can operators release writes. This is slower than attaching the latest snapshots, but it prevents a fast return to a state that never validly existed.

Microsoft's current Foundry Agent Service recovery guidance illustrates why capability claims must be checked per service: for the documented standard deployment, it says the service does not provide built-in replication, backups, point-in-time restore, active-active cross-region replication, or state merging. Its proposed recovery is often reconstruction, with agent definitions treated as code and some thread state potentially unrecoverable. That is one product's documented boundary, not a universal limitation—but it shows why “managed agent” cannot be translated into “managed recovery.”

Test business validity, not just restore completion

A drill should inject failures that force real decisions: corrupted primary data, a deleted object version, unavailable control plane, stale index snapshot, lost encryption key, changed model alias, partially committed tool call, cross-tenant contamination, and an operator who is unavailable. Run it outside the production failure domain and measure:

• achieved RTO and RPO by business capability, not storage service;
• maximum skew between recovered authoritative components;
• percentage of derived records with source and recipe provenance;
• orphan, duplicate, and missing-object rates after rebuild;
• unknown external effects remaining at write enablement;
• tenant-isolation and deletion-marker violations;
• time spent on undocumented manual steps; and
• percentage of acceptance checks whose expected result was defined before the drill.

Connect the evidence to the AI audit-evidence guide. A screenshot saying “restore succeeded” proves a job reported success. Stronger evidence joins the approved cut, artifact manifests, hashes, log offsets, test queries, effect receipts, operator decisions, deviations, and final authorization to resume.

The release gate

Do not call an AI service recoverable until its owner can answer yes to these questions:

1. Is the business capability, degraded mode, RTO, RPO, and maximum acceptable component skew explicit?
2. Are authoritative, versioned, derived, external-effect, and ephemeral states classified?
3. Can one recovery cut be named across databases, objects, events, releases, and indexes?
4. Does every derived store have a tested rebuild recipe and source manifest?
5. Can external effects be reconciled without blind retry?
6. Are recovery credentials and control paths independent of the failed boundary?
7. Do deletion, retention, sovereignty, and tenant rules still hold in backups and recovery environments?
8. Can the team prove application-level validity before enabling writes?
9. Has failback been rehearsed without discarding actions taken during degraded operation?
10. Has the whole procedure met its target under a realistic timed drill?

Revisit the plan after any model, prompt, policy, index, identity, region, vendor, retention, or tool change. Also revisit it when restore duration approaches RTO, provenance coverage falls, a drill exposes hidden control-plane dependencies, or the product starts treating previously ephemeral state as a promise to users. Recovery is not a storage feature. It is a maintained agreement about which history the organization can safely resume.

Source notes — reviewed 2026-08-18

• NIST SP 800-34 Rev. 1 — published May 2010 and updated November 2010; contingency planning, recovery phases, reconstitution, and business-impact priorities.
• NIST SP 800-184 — published December 22, 2016; recovery playbooks, realistic exercises, metrics, and improvement.
• NCSC Guidelines for Secure AI System Development — published and reviewed November 27, 2023; AI asset inventory, versioning, protection, failover, and known-good restoration.
• AWS Disaster Recovery of Workloads — RPO, RTO, availability, resilience, and business-continuity framing.
• Google Cloud disaster-recovery planning guide — last reviewed July 5, 2024; business-impact analysis and RPO/RTO tradeoffs.
• Google Cloud infrastructure-outage architecture — last reviewed May 10, 2024; end-to-end recovery, correctness, criticality tiers, and management-plane dependency limits.
• PostgreSQL continuous archiving and point-in-time recovery — current version 18 documentation reviewed August 18, 2026; base backups, WAL archives, incremental dependencies, restore targets, and timelines.
• Microsoft Foundry Agent Service disaster recovery — current official product guidance reviewed August 18, 2026; state boundaries, reconstruction, service limitations, and recovery drills.