The Long-Running Agent: AI Workflows That Survive the Real World

An agent that works only while one process, connection, and credential remain alive is a long API request, not a durable workflow. Real work waits for an approver overnight, loses a worker during deployment, receives the same event twice, encounters an expired token, and discovers that one of five external systems changed while it was paused.

Durability means the business process can reconstruct its state, decide what is still valid, and continue safely after interruption. It does not mean every attempt runs exactly once or that the model can resume from an unstructured chat transcript. The workflow engine must own state and time; the model supplies bounded decisions inside that system.

Put the state machine outside the model

Represent the workflow with explicit states such as:

received → validated → planned → awaiting_approval → authorized → executing → reconciling → completed

Also model terminal or exceptional states: rejected, cancelled, expired, compensating, and manual_review. Each transition should record:

• workflow and task identity;
• input or event that triggered the transition;
• previous and next state;
• code, model, prompt, policy, and schema versions when relevant;
• normalized decision or tool request;
• authorization and approval references;
• timestamp, retry count, and deadline;
• authoritative result or unresolved uncertainty.

Do not treat “the last assistant message” as workflow state. Chat history mixes user language, instructions, retrieved content, model output, and tool observations. It is useful context, not a transactional source of truth.

A durable runtime can use event sourcing, checkpoints, a database state machine, or a workflow platform. The key requirement is that committed transitions survive worker loss and can be replayed or reconstructed without repeating completed side effects.

Separate deterministic orchestration from nondeterministic work

Model calls, current time, randomness, network requests, and human decisions are nondeterministic. A replay-based workflow must record their outcomes and replay those recorded facts rather than call them again while rebuilding state.

Use two conceptual layers:

• orchestration: deterministic decisions about which step should happen next based on persisted history;
• activities: side-effecting or nondeterministic operations such as calling a model, writing a record, sending a message, or querying an external API.

Temporal is one vendor-specific, open-source example of this pattern: its documentation describes durable event history and replay. Other platforms use different execution models and guarantees. Do not copy a vendor claim into an architectural guarantee; document exactly which parts of your chosen system are durable, how history is stored, and what happens during regional loss or operator error.

Persist the useful output of an LLM activity: the model identifier, normalized input reference, structured result, validation status, and safety decision. Avoid storing raw secrets or unnecessary personal data in workflow history, because durable history is deliberately hard to erase or mutate.

Assume activities can execute more than once

Queues and distributed systems normally favor at-least-once delivery. A worker may complete a side effect and crash before acknowledging it. The scheduler then retries, even though the target system has already changed.

For every write, define an idempotency key derived from the stable business operation, not the retry attempt. A good key might be:

tenant + workflow_id + action_type + target_id + action_version

The target service should atomically store the key with the effect and return the original result on duplicate requests. If the external API does not support idempotency, add a gateway ledger, use a uniquely constrained business reference, or reconcile before retrying.

HTTP semantics help but do not solve the business problem. RFC 9110 defines an HTTP method as idempotent when multiple identical requests have the same intended effect as one. GET, PUT, and DELETE have idempotent semantics; POST is not generally idempotent. Even an idempotent method can produce repeated logging or fail around concurrent state changes, and a nominally non-idempotent endpoint can implement an application-level idempotency key.

There is no practical blanket “exactly once” guarantee across an agent, queue, payment processor, email provider, and database. State the narrower guarantee you can enforce: for example, “one ledger entry per business operation key, with reconciliation of unknown outcomes.”

Classify errors before retrying

Retry only when another attempt may be both safe and useful. A retry policy should distinguish:

Set attempt limits, total elapsed-time budgets, per-activity timeouts, and a next-review time. Infinite retry loops turn a temporary incident into uncontrolled load and a stale future action.

Record the server’s retry hint where trustworthy, but still apply local caps and jitter. Put exhausted work in a visible review queue with the last authoritative state and safe operator actions. “Dead letter” should not mean invisible abandonment.

Treat waiting as a persisted state

A durable agent may wait minutes or weeks for:

• human approval;
• a scheduled time;
• a document upload;
• an asynchronous provider callback;
• inventory or payment settlement;
• a dependency to recover.

Persist the wait condition, deadline, responsible party, reminder policy, and cancellation rule. Do not keep a thread sleeping or a browser tab open. Resume from a verified event, not from any message that happens to carry the workflow ID.

Use correlation plus authentication for callbacks. If you adopt the CloudEvents specification, its vendor-neutral event envelope provides useful id, source, type, and schema metadata, and permits consumers to treat the same source plus id as a duplicate. CloudEvents standardizes event description, not transport delivery or exactly-once processing, so the consumer still needs deduplication and authorization.

At resume, revalidate all time-sensitive assumptions: record version, price, inventory, identity, authorization, policy, and approval. Approval should bind to a preview or request hash. A changed request is a new decision, not a continuation.

Design cancellation and compensation

Cancellation is cooperative, not magical. It must propagate to queued activities, child workflows, tool requests, and user-facing status. An activity already committed in an external system may be impossible to cancel.

For multi-step business transactions, define compensating actions before release. The classic 1987 Sagas paper by Garcia-Molina and Salem describes long-lived work as a sequence of transactions with compensating transactions. Compensation is not rollback: a shipped package may need a return, a sent email cannot be unsent, and a refund creates another ledger event rather than erasing the charge.

For every side effect, document:

• whether it can be cancelled before execution;
• how to detect whether it committed;
• whether compensation exists;
• who may authorize compensation;
• what happens when compensation also fails.

Keep the original effect and the compensation in the audit trail. A clean final state must not erase the incident history.

Concrete example: approving and issuing a purchase order

Consider an agent that turns a department request into a purchase order:

1. Receive: create workflow PO-2026-481 and deduplicate the intake event.
2. Validate: confirm supplier, cost center, currency, requested items, and tenant.
3. Plan: the model classifies the request and proposes accounting codes; a deterministic validator checks them.
4. Quote: retrieve current price and availability as an activity; persist quote ID and expiry.
5. Approve: show a fixed preview containing supplier, total, terms, and quote version. Persist approver and request hash.
6. Wait: the workflow can stop for two days without holding a process.
7. Resume: recheck the quote, budget, supplier status, policy, and approver authority.
8. Commit: issue one purchase order using idempotency key PO-2026-481:issue:v1.
9. Unknown outcome: if the supplier gateway times out, query by the same business reference before retrying.
10. Notify: send the receipt as a separate idempotent activity and record delivery status.

If the quote expires, the workflow returns to quote_required; it does not silently buy at a new price under the old approval. If the PO was issued but notification failed, retry the notification without issuing another PO. If the user cancels after issuance, start the documented cancellation or compensation path.

Version workflows without corrupting live runs

Long-running executions outlive deployments. A code change that reorders activities or interprets an old event differently can break replay or produce a new effect.

Use explicit workflow and schema versions. Test new code by replaying production-like historical event histories. For an incompatible change, keep the old transition path for existing runs, migrate state with a reviewed procedure, or start a new workflow linked to the old one.

Model and prompt updates also need versioning. A paused workflow should not silently send an old approval into a materially different model plan. Decide which changes are safe for in-flight work and which force replanning and reapproval.

Bound history growth. Use snapshots or “continue as new” mechanisms while preserving lineage, evidence, and deduplication. Never truncate the only record needed to decide whether an external write already happened.

Make operations inspectable

Operators need more than logs. Provide a workflow view showing:

• current state, age, deadline, and owner;
• last successful transition and next eligible action;
• pending activity, attempt count, and backoff;
• approval, credential, and policy status;
• child workflows and external references;
• idempotency keys and unknown outcomes;
• safe controls for cancel, retry, reconcile, or escalate.

Instrument every activity with a trace ID, workflow ID, activity ID, and attempt number. OpenTelemetry supplies vendor-neutral trace, metric, and log specifications, including span links useful for asynchronous work. Telemetry does not replace the durable event record; sampled traces may be missing, while the workflow ledger must remain authoritative.

Review Observability for AI Agents for model- and tool-level instrumentation and Audit-Ready AI for evidence retention.

Test failure, not only the happy path

Before production, inject failures at every boundary:

• kill the worker before and after a tool commit;
• deliver the same event many times and out of order;
• expire or revoke credentials during a pause;
• change a record after approval but before commit;
• delay a callback beyond its deadline;
• deploy a new workflow version with old runs active;
• make compensation fail;
• lose telemetry while the workflow continues;
• cancel during queued, running, and unknown-outcome states.

Release criteria should include:

• no duplicate consequential effects in crash-and-retry tests;
• every write has an idempotency key and authoritative reconciliation path;
• every persisted state can resume on a different worker;
• stale approval, policy, identity, and record versions fail closed;
• retry budgets and backoff prevent runaway load;
• operators can find and resolve every unknown outcome;
• replay compatibility is tested for supported live versions;
• cancellation and compensation meet documented behavior;
• recovery time and oldest-stuck-workflow objectives are met under load;
• a named owner receives alerts for queue age, failure rate, and stalled waits.

Track completion time by state, retry and duplicate rate, unknown-outcome age, approval wait, compensation rate, replay failures, stuck workflows, and operator interventions. For the wider release discipline, use The AI Operational Readiness Checklist.

Frequently asked questions

Is checkpointing the conversation enough?

No. A checkpoint must include structured workflow state, completed effects, versions, authority, deadlines, and deduplication data. Restoring tokens or messages alone cannot prove what committed.

Should the model decide retry policy?

The model may classify an unfamiliar error for review, but deterministic policy should govern retryable codes, limits, backoff, and safety. Do not let generated text override authorization or idempotency.

Does durable execution guarantee exactly once?

Not across arbitrary external systems. Durable orchestration can record decisions and resume reliably; side effects still require idempotency, reconciliation, and sometimes compensation.

When should a workflow go to a human?

When the result is unknown, state conflicts after approval, compensation is risky, policy is ambiguous, or retry budgets are exhausted. Escalation is a designed state, not a catch-all exception.

Source notes

Sources reviewed and current as of July 30, 2026:

• RFC 9110: HTTP Semantics — formal definition and limits of idempotent HTTP methods.
• CloudEvents specification — CNCF vendor-neutral event envelope and duplicate identity semantics; it does not define delivery guarantees.
• Sagas, Garcia-Molina and Salem, ACM SIGMOD 1987 — primary paper on long-lived transactions and compensation.
• OpenTelemetry specification — vendor-neutral telemetry primitives; not a workflow system of record.
• Temporal documentation — a vendor-specific example of event-history replay and durable execution, not the only implementation pattern.