
One Writer, Many Agents: Concurrency Control That Survives Failure
A field guide to choosing serialization, version checks, transactions, leases, and fencing tokens when several AI workers can touch the same state.
Read MoreZharfAI Team

An operator tells a facilities agent to investigate a cooling alarm. The agent reads telemetry, asks a model for a diagnosis, opens a maintenance ticket, reserves a technician, and begins ordering a replacement part. Then a sensor correction arrives: the alarm was false. The operator presses Stop. The chat spinner disappears immediately.
What stopped? The browser may have stopped waiting while the orchestration continues. The orchestration may have stopped scheduling new steps while an RPC handler keeps computing. The worker may have received a cancellation request while the supplier order has already committed. A clean interface can hide an untidy reality: a stop signal changes intent; it does not prove that every descendant stopped or that earlier effects were undone.
This field guide addresses one operating decision: for each running AI task, should a stop request abandon observation, request cooperative cancellation, wait for confirmed quiescence, or trigger compensation and reconciliation? The answer depends on what is still running, what has crossed an external commit point, how much proof the consequence requires, and how long the system can safely wait.
Teams often compress three states into the word “cancelled”:
These states can occur seconds or hours apart. The first is a signal about demand. The second is a workflow decision. Only the third can support a strong claim such as “nothing more will happen.” A product must not render state one as if state three were proven.
This distinction complements the retry and idempotency guide. A timeout makes completion uncertain; a cancellation makes future intent clear. Neither one tells you whether an external action already happened.
The WHATWG DOM Standard defines AbortController as a way to store an abort reason and signal observers; dependent signals can combine sources such as a user action and a timeout. This is a useful local primitive. It is not a distributed rollback protocol. Each API that accepts the signal still needs an abort algorithm, and a remote system cannot observe a JavaScript object unless the application propagates the intent.
The gRPC cancellation guide is explicit about cooperative behavior: a server handler may already be busy, the library generally cannot interrupt application code, and long-running handlers must check cancellation and stop their own work. Propagation to upstream calls is automatic in some languages and an application responsibility in others. The gRPC core concepts add the crucial warning that changes made before cancellation are not rolled back. Client and server can also reach different local conclusions about the same RPC.
Infrastructure termination has similar boundaries. Kubernetes documents a graceful phase followed by forceful termination in its Pod lifecycle: a process receives a stop signal and a grace period, then may receive SIGKILL. A forced API deletion does not wait for proof that the process disappeared from the node. Killing a worker also says nothing by itself about a payment, message, reservation, or write the worker already sent elsewhere.
Orchestrators expose the gap directly. AWS Step Functions describes cancellation of integrated .sync jobs as a best-effort attempt that can fail because of permissions or an outage. Microsoft states in its Durable Task instance-management documentation that termination is queued, reaches Terminated eventually, and currently does not propagate to activities or sub-orchestrations; those may run to completion.
Temporal makes the choice visible in its official Java SDK activity cancellation modes: wait for confirmation, try to cancel and return immediately, or abandon without requesting cancellation. Waiting can block if an activity does not heartbeat or ignores the request. These documents establish mechanisms and limits. The control contract below is ZharfAI analysis built from them, not a standard published by those projects.
Do not give every operation the same red button. Assign a stop contract before production:
| Contract | What the caller may conclude | Appropriate use | Main risk |
|---|---|---|---|
| Abandon | “I will no longer wait for or display this result.” | Disposable reads or speculative alternatives whose continued work has no material effect | Wasted compute and hidden descendants |
| Request stop | “The system asked descendants to stop.” | Cooperative model generation, retrieval, transforms, and low-impact background work | Request may be delayed, ignored, or lost |
| Confirm quiescence | “Named descendants reached known terminal states.” | Expensive work, shared-resource operations, and actions where later completion would surprise the user | Slow or unavailable acknowledgements |
| Compensate and reconcile | “Committed effects were enumerated and brought to an approved state.” | Orders, messages, reservations, permissions, deployments, records, and physical actions | Compensation can fail or create a second effect |
“Hard kill” is an enforcement mechanism inside the second or third contract, not a universal fifth answer. It can stop a process after a grace period, but it cannot travel backward through a committed database transaction or recall an email already accepted by another server.
Cancellation should address one logical task across every attempt and descendant. Carry a trusted envelope outside model-generated text:
| Field | Purpose |
|---|---|
| Logical intent ID | Joins the original request, retries, fallbacks, and child work |
| Stop request ID | Makes repeated stop messages idempotent and auditable |
| Requested by and reason | Separates user withdrawal, supersession, deadline, policy, and incident control |
| Requested-at time and stop deadline | Bounds graceful cleanup before escalation |
| Propagation scope | Names which branches, tools, queues, and resources are covered |
| Required contract | Abandon, request, confirm, or compensate |
| Effect ledger reference | Points to external actions already attempted or committed |
| Policy and release version | Preserves the rules in force when the decision was made |
Derive identity, authority, consequence, and scope from authenticated product state. The model may suggest that a task is obsolete, but it must not forge a user withdrawal, widen a cancellation scope, or decide that reconciliation is unnecessary. That follows the boundary in the AI tool-permission guide: generated intent is not authority.
A boolean cancelled=true cannot represent the transition. Use explicit states such as:
RUNNING: work may start or commit;STOP_REQUESTED: intent changed; new descendants are forbidden;QUIESCING: known descendants are acknowledging, draining, or being force-stopped;STOPPED_CLEAN: required descendants confirmed no relevant effect remains in flight;STOPPED_WITH_EFFECTS: execution stopped, but committed effects remain valid and visible;COMPENSATING: approved inverse actions are running;RECONCILIATION_REQUIRED: the system cannot determine or restore the required business state automatically;UNKNOWN: a descendant or external system cannot be reached, so completion is unresolved.Only the controller may advance these states, using evidence from workers and external systems. A UI can say “Stop requested” immediately, then update to “Stopped,” “Stopped; two effects remain,” or “Needs review.” Honest intermediate state is better than a reassuring false terminal state.
Keep UNKNOWN durable rather than converting it to success after a timeout. Schedule reconciliation and preserve identifiers. The durable-agent workflow guide explains why reconstructible state belongs outside a transient model process.
An AI request can fan out into retrieval, several model calls, subagents, browser sessions, tool adapters, queue messages, database jobs, and vendor APIs. Build a task tree or directed execution graph with parent-child identifiers and a cancellation capability at each edge.
When STOP_REQUESTED arrives:
Reserve capacity for control traffic. A saturated service that has no room for cancellation, status queries, or cleanup has made overload irreversible. The admission-control field guide recommends a narrow reserve for exactly this kind of health and control work.
Every tool adapter should declare its effect class and commit evidence:
| Effect class | Example | Stop behavior |
|---|---|---|
| Pure or disposable read | Retrieve a document, score a candidate answer | Cancel or abandon; discard late result |
| Reserving but reversible | Hold inventory, reserve a slot, create a draft | Stop, then release or expire with confirmation |
| Idempotent write | Upsert a record by stable business key | Re-read authoritative state; do not assume transport failure means no write |
| Compensatable action | Create an order that can be cancelled | Record the original ID, apply the approved inverse, verify both states |
| Irreversible or externally governed | Send an email, disclose data, trigger physical work | Stop future steps, surface the effect, and route to human or domain procedure |
Record prepared, submitted, accepted, committed, and verified separately where the integration supports them. The most dangerous moment is a cancellation racing an external commit. The worker may see the stop just after the vendor accepted the request but before the response returned. Query by the logical intent or idempotency key; never infer “not committed” from a cancelled transport.
Compensation is a new consequential action, not an eraser. Give it independent authorization, idempotency, evidence, and verification; define the desired business state before choosing the least harmful path to it.
Return to the false cooling alarm. The workflow has five descendants:
| Descendant at stop time | Observed state | Decision |
|---|---|---|
| Telemetry retrieval | Complete, read-only | Retain as evidence |
| Diagnosis generation | Streaming | Request cooperative stop; discard late tokens |
| Maintenance ticket | Committed as a draft | Close as false alarm with a linked reason; do not delete the audit trail |
| Technician reservation | Accepted with a reservation ID | Cancel idempotently and verify release |
| Parts order | Request sent; response missing | Mark unknown, query supplier by intent key, then cancel only if the order exists |
The UI first shows “Stop requested.” New descendants are blocked. Generation acknowledges quickly. The ticket and technician adapter report their final states. The supplier API is unavailable, so the workflow becomes RECONCILIATION_REQUIRED, not STOPPED_CLEAN.
Ten minutes later, reconciliation finds that the part order committed before the stop. Policy allows cancellation below a value threshold, so a separate authorized action cancels it and records the cancellation receipt. The task finishes as STOPPED_WITH_EFFECTS: the historical ticket and order remain in the ledger, while no operational obligation remains open. That outcome is more truthful than pretending the task never ran.
For each logical intent, retain the minimum evidence needed to answer:
Link these events without logging unnecessary prompt contents or secrets. The audit-evidence guide provides the broader pattern: build evidence around assertions an investigator must prove, not around every field telemetry happens to expose.
Measure stop-request-to-admission-close, acknowledgement latency, quiescence latency, force-stop rate, descendants discovered after stop, compute completed after withdrawal, committed effects after stop request, compensation success, reconciliation age, and false-clean rate. A fast spinner disappearance is not a reliability metric.
Cancellation tests must target timing boundaries:
Assert business invariants, not only process exit: no new descendant after the stop barrier; no duplicate compensation; every committed effect has an authoritative terminal or unknown state; and the UI never claims clean stop while the ledger disagrees.
Before a tool-using AI workflow exposes a production stop control, answer yes:
A trustworthy stop button is not a UI event and not a process signal. It is a protocol that closes admission, propagates changed intent, gathers acknowledgements, discovers committed effects, and reconciles uncertainty. Design that protocol before an agent gains consequential tools; after the wrong action begins, the missing semantics become the incident.

A field guide to choosing serialization, version checks, transactions, leases, and fencing tokens when several AI workers can touch the same state.
Read More
An operating guide for deciding when an AI call may be retried, hedged, failed over, or reconciled—and for preventing timeout ambiguity from becoming duplicate business action.
Read More
A field guide to workload identity, token exchange, scoped credentials, connector proxies, and evidence when AI agents need temporary access to enterprise systems.
Read MoreIf this note maps to a real system in your organization, start with the services page or a shipped case study.