One Writer, Many Agents: Concurrency Control That Survives Failure

Two operations agents receive different alerts about the same production service. One decides to roll the service back to the last known-good release. The other validates a hotfix and decides to promote it. Both agents are competent. Both hold valid tool permissions. Both read a plausible version of the deployment state. If they act at nearly the same time, the later packet—not the better decision—may determine what runs.

This is a concurrency failure, not a reasoning failure. A model can explain its plan perfectly and still write against state that changed after the plan began. A lease can expire while a slow worker continues. A new leader can take over while a delayed command from the former leader is still in flight. A database row, object, ticket, branch, invoice, reservation, or physical controller can therefore receive two individually valid but jointly incompatible actions.

The operating decision is: for each shared resource, should workers partition ownership, serialize through one writer, compare and swap against a version, commit inside a transaction, hold a lease with a fencing token, or produce mergeable proposals instead of direct writes? The choice belongs in application architecture. It must not be improvised by a language model after contention begins.

Concurrency is not another name for retry

A retry repeats one logical intent because its outcome is uncertain. Concurrency means two or more live intents overlap on the same invariant. The retry and idempotency guide prevents one intent from becoming several effects. It does not decide which of two different intents should win.

Cancellation is also different. A stop request changes future intent, but an old worker may continue long enough to send a late write. The cancellation semantics guide explains why loss of interest or loss of a lease is not proof of quiescence. Concurrency control makes the target reject an action whose authority or expected state is no longer current.

Three claims must remain separate:

1. The worker once obtained ownership. A coordinator granted a lease or elected a leader.
2. The worker still believes it owns the resource. Its local timer, cache, or session has not told it otherwise.
3. The target will accept only current ownership. The database, service, or adapter validates a version or monotonically newer fencing token at commit time.

Only the third claim closes the stale-writer path. A distributed lock that the target never checks is an advisory story, not an enforcement boundary.

What established systems actually guarantee

HTTP already carries a useful optimistic-concurrency primitive. RFC 9110 defines If-Match for state-changing requests: the origin server evaluates a strong entity tag before performing the method and normally returns 412 Precondition Failed when the representation has changed. The specification describes this as protection against the lost-update problem. It does not decide how an AI workflow should resolve the conflict; it gives the application an honest refusal instead of a silent overwrite.

The Amazon DynamoDB concurrency guidance distinguishes optimistic version checks for low contention, transactions for atomic multi-item changes, and a lease-and-heartbeat lock client for long-running coordination. It also warns that global tables use last-writer-wins reconciliation, so a version pattern does not provide the same protection across Regions. The datastore's replication contract matters as much as the code around it.

At the database level, PostgreSQL 18 transaction-isolation documentation says successfully committed Serializable transactions have an effect consistent with running one at a time. When PostgreSQL detects a serialization anomaly, an application must be prepared to retry the entire transaction. The transaction must contain the integrity-critical read and write; reading, asking a model for a minute, and later opening a new transaction does not preserve the original snapshot.

Coordination systems address longer ownership. Kubernetes Leases are used for node heartbeats and component leader election so one active instance can perform control work while peers stand by. Apache ZooKeeper's lock recipe uses ephemeral sequential nodes, watches only the next predecessor to avoid a herd, and attaches a GUID so a client can discover whether a create succeeded when the response was lost.

Those mechanisms help select an owner. They do not, by themselves, stop a former owner from reaching an external target late. Google's original Chubby lock-service paper makes the missing step explicit: a lock holder can pass a sequencer containing the lock generation to the server it calls, and the server rejects an invalid or older sequencer. This is the fencing pattern. Ownership becomes a value the effect target can compare, not merely a belief held by the worker.

Object storage exposes the same principle without calling it a lock. Google Cloud Storage request preconditions let a write or delete require an exact object generation. A delayed delete aimed at an older generation then fails rather than removing a newly created object with the same name. Preconditions are useful because they bind the action to the object instance the caller actually observed.

These are verified protocol behaviors. The operating model below is ZharfAI analysis that composes them for tool-using AI systems.

Choose the smallest concurrency contract that protects the invariant

Do not put a distributed lock around everything. First name the resource, invariant, contention pattern, and consequence.

Partitioning is usually the cheapest safe answer. A worker processing account A should not contend with one processing account B. A single writer is appropriate when a deployment, ledger, schedule, or finite inventory has one meaningful order. Compare-and-swap fits short read-modify-write operations. Transactions fit invariants that live inside one transactional boundary. Leases with fencing fit work that cannot stay inside a database transaction. Proposal-and-merge is the safe default when automatic conflict resolution would erase meaning.

The admission-control field guide is relevant here: serialization moves contention into a queue, so the queue still needs limits, expiry, fairness, and a clear rejection state. Concurrency safety must not create an unbounded warehouse of obsolete work.

Put an ownership envelope outside the prompt

Every consequential write should carry machine-derived concurrency fields alongside the business arguments:

Generate these fields in trusted application code. The model may identify a likely record from natural language, but the product must resolve that suggestion to a canonical resource under authenticated scope. The model must not invent a version, widen a lock key, extend its own lease, or decide that two resources are “close enough” to share a token.

Bind the concurrency envelope to the release described in the AI release passport. A worker running old adapter code may format a fencing token incorrectly or omit a version precondition. Release identity therefore belongs in the same evidence chain as resource identity.

Enforce at the final effect boundary

The guard must sit where the irreversible or authoritative change occurs. If an orchestrator checks a token and then calls an unguarded vendor API, a delayed worker can bypass the guarantee through another path.

For a database row, enforce a version predicate in the update or use a transaction. For an object, require a generation or ETag precondition. For a deployment controller, store the highest accepted fencing token per service and reject a lower one. For an external API without concurrency support, place a single controlled adapter in front of it; serialize calls there, reconcile authoritative state, and never expose the vendor credential directly to workers.

Keep the model call outside a database transaction and usually outside the shortest lock. Take a snapshot, plan, validate, then reacquire or verify current state immediately before commit. If the state changed, the plan is stale even when its prose remains sensible. Re-read, re-evaluate policy, and either replan or surface a conflict.

Do not automatically retry a 409, 412, failed condition, or serialization error with the same old arguments. A concurrency refusal is new information. The correct response may be to recompute, merge, abstain, ask a human, or declare the intent superseded.

Worked example: a delayed deployment agent

Assume a deployment controller stores release=R12, version=41, and highest_fence=781.

1. Agent A receives an incident alert, obtains lease token 782, reads version 41, and prepares a rollback to R11.
2. A network partition delays Agent A before commit. Its local process continues and still believes the lease is usable.
3. The lease expires. Agent B obtains token 783, reads the current service, validates hotfix R13, and commits with expected_version=41 and fence=783.
4. The controller atomically records release=R13, version=42, and highest_fence=783.
5. Agent A's delayed rollback arrives with expected_version=41 and fence=782.

The controller rejects A for two independent reasons: the resource version changed and the fencing token is older than the highest accepted token. A records STALE_OWNER, fetches version 42, and routes the original incident intent for reassessment. It does not “helpfully” retry the rollback against R13.

If the controller checked only whether A once held the lease, the stale rollback could win. If it checked only the expected version, a different operation that does not change that version might still expose a gap. The target should validate the strongest invariants it owns, atomically with the effect.

Treat multi-resource work as a declared workflow

Many agent actions cross a database, a ticketing system, object storage, and a vendor API. No single lock makes those systems one transaction. Define a canonical order and a state machine such as:

• PROPOSED: plan exists but owns nothing;
• CLAIMED: a bounded owner and fencing token exist;
• PREPARED: current versions and policy checks pass;
• COMMITTING: the guarded target action is in flight;
• COMMITTED: authoritative result and new version are recorded;
• CONFLICTED: a precondition or serialization guard refused the action;
• STALE_OWNER: a newer fencing token exists;
• RECONCILIATION_REQUIRED: outcome or cross-system state is uncertain.

Use an effect ledger for every external commit. If step two fails after step one succeeds, do not hold a fake global lock and call the workflow atomic. Record the partial state, stop incompatible successors, and compensate or reconcile through the same discipline described in the cancellation guide.

Failure modes that pass ordinary tests

• Lock without fencing: an expired owner sends a delayed write after a new owner begins.
• Last writer wins: a clean replication policy silently discards the more informed action.
• Time-to-live as truth: clock skew, pauses, or partitions make a local expiry estimate disagree with the coordinator.
• Wrong resource key: two aliases for one customer, branch, asset, or object receive separate locks.
• Check then act: code validates a version, releases the guard, and performs the write later.
• Blind conflict retry: the worker resubmits stale arguments until the condition happens to pass.
• Long model call inside a transaction: locks, connections, or snapshots live far beyond the integrity-critical section.
• One token, several targets: the first system enforces fencing while a secondary adapter ignores it.
• Merge by fluent summary: a model combines incompatible edits without preserving either invariant.
• Success before verification: the UI reports completion from the worker response rather than the authoritative new state.

The remedy is not more chain-of-thought. It is smaller ownership scope, guarded commits, and explicit conflict states.

Test the race, not only both happy paths

Build deterministic tests that pause workers at every boundary: after read, after lease acquisition, after planning, before commit, after target acceptance, and before acknowledgement. Then inject duplicate delivery, response loss, worker crash, lease expiry, coordinator failover, delayed packets, message reordering, clock skew, and two aliases for the same resource.

Assert invariants:

• at most one incompatible commit is accepted for a resource version;
• a lower fencing token can never overwrite a higher accepted token;
• a failed precondition creates no partial effect;
• retries reread authoritative state and stay within a bounded budget;
• stale owners become terminal or reconcile; they do not spin;
• the UI distinguishes waiting, conflict, stale owner, unknown, and committed;
• every accepted effect links to intent, resource, version, token, release, and result evidence.

Monitor conflict rate, serialization retries, lease turnover, stale-writer rejection, lock wait, queue age, fencing enforcement coverage, manual-merge rate, reconciliation age, and commits without a recorded precondition. Slice by resource class and workflow. A low global conflict rate can hide one high-consequence resource whose lock key is wrong.

Preserve enough evidence to prove the claim. The audit-evidence guide shows the broader method: log the assertion an investigator must verify, not every token the model produced.

The one-writer gate

Before several AI workers can change shared state, answer yes:

1. Is the canonical resource and protected invariant explicitly named?
2. Is ownership partitioned wherever practical?
3. Is the selected contract—queue, compare-and-swap, transaction, lease with fencing, or merge—appropriate to contention and consequence?
4. Does the authoritative target enforce the version or fencing token atomically with the effect?
5. Can a former owner continue running without being able to commit?
6. Are model calls kept outside the shortest integrity-critical section?
7. Does a conflict trigger reread and reassessment rather than blind retry?
8. Are multi-resource partial effects represented and reconciled honestly?
9. Do all adapters enforce the same resource identity and token semantics?
10. Have delay, reordering, expiry, crash, failover, and alias races been tested?
11. Can operators see who owns the work, which version was read, and why a commit was refused?
12. Can evidence prove the exact writer and state the target accepted?

Parallel agents are useful because they reduce elapsed time and bring specialized judgment to a workflow. They are dangerous when concurrency is treated as a scheduling detail. The reliable design gives each shared invariant one enforceable commit order—and makes stale authority fail at the place where failure still prevents harm.

Source Notes — reviewed August 12, 2026

• RFC 9110: HTTP Semantics — June 2022 standard reviewed on the publication date; source for strong entity-tag preconditions, If-Match, If-None-Match, lost-update prevention, and 412 Precondition Failed behavior.
• PostgreSQL 18: Transaction Isolation — current documentation reviewed on the publication date; source for Serializable guarantees, serialization failures, and whole-transaction retry requirements.
• Amazon DynamoDB: Best practices for handling concurrent updates — current documentation reviewed on the publication date; source for optimistic versions, transactions, lease-based locking, and the global-tables limitation.
• Kubernetes: Leases — current documentation reviewed on the publication date; source for heartbeat and leader-election uses of Lease objects.
• Apache ZooKeeper: Recipes and Solutions — version 3.7.2 documentation reviewed on the publication date; source for ephemeral sequential lock nodes, predecessor watches, GUID recovery, and shared-lock mechanics.
• Google Research: The Chubby lock service — original 2006 OSDI paper; source for lock generations, sequencers, compare-and-swap, delayed-writer rejection, and lock-delay limitations.
• Google Cloud Storage: Request preconditions — current documentation reviewed on the publication date; source for generation-match guards and delayed-delete race prevention.