The Queue Is a Policy: Admission Control for AI Under Load

A customer-support assistant is serving short policy questions when an evaluation job submits thousands of long prompts. At the same time, one agent begins a document-heavy tool workflow, a provider slows down, and impatient clients retry. Every request is individually legitimate. Together they fill the queue, extend time to first token beyond the users’ deadlines, occupy memory with work that may never be consumed, and make the service spend its remaining capacity finishing obsolete attempts.

“Add more GPUs” is not a complete answer. Capacity can disappear faster than new instances warm, a downstream API can become the bottleneck, and a larger fleet still needs rules for scarcity. The operating decision is more precise: when capacity is constrained, which AI work should be admitted now, queued briefly, degraded safely, deferred durably, or rejected clearly?

This field guide develops an admission-control architecture for that decision. It treats the queue as a policy surface—not an accidental buffer—and optimizes for useful, on-time outcomes rather than raw request throughput.

Throughput is not useful work

Overload creates a feedback loop. Work takes longer, more requests remain in flight, clients reach their deadlines, retries add offered load, and the service completes responses nobody is waiting to use. Google’s SRE guidance on cascading failures recommends bounded queues, early load shedding, deadline propagation, and task-specific graceful degradation because queued and late work consumes resources without necessarily producing value.

Amazon’s current load-shedding guidance makes a useful distinction between throughput, all work offered to a service, and goodput, work completed successfully and soon enough for the client to use. An AI gateway can report rising requests per second while goodput falls: generations finish after disconnection, tool calls outlive approvals, and batch traffic delays urgent interactive work.

Autoscaling remains necessary, but it is not admission control. Scaling reacts to a signal and has a delay. Admission control decides what the system may accept at this instant, with the capacity and dependencies actually available. A resilient service needs both.

Separate established mechanisms from the AI policy

Several mechanisms are well established:

• bounded queues prevent arbitrary buffering;
• deadlines identify work that is too late to be useful;
• load shedding rejects excess work before it consumes scarce resources;
• backpressure lets a slower consumer limit a faster producer;
• quotas and fair queuing stop one flow from starving others;
• resource-pressure triggers can activate protective actions.

The Reactive Streams specification, for example, defines non-blocking backpressure so a subscriber controls demand and buffers can remain bounded. Kubernetes API Priority and Fairness classifies requests into priority levels, isolates concurrency, applies fair queuing and shuffle sharding, and can either queue briefly or reject excess traffic. Envoy’s overload manager turns monitored resource pressure into staged actions such as stopping new requests; its documentation distinguishes host overload protection from upstream circuit breaking.

The five-way disposition model in this article—admit, queue, degrade, defer, reject—is ZharfAI analysis built from those mechanisms. It is not a standard published by Google, Amazon, Kubernetes, Envoy, Reactive Streams, vLLM, or the IETF. The AI-specific work is to define request cost, consequence, fairness, and safe degradation without allowing the model to claim its own priority.

Give every request an admission envelope

Do not let a raw prompt enter a shared inference queue. Build a trusted envelope outside the model:

The model may estimate complexity as one signal, but it must not set priority, tenant, deadline, approved, or emergency. Derive trusted values from authenticated product state, the requested operation, and versioned policy. This extends the AI permission layer: access authority and resource priority are different decisions, but neither belongs to generated prose.

Cost must be multi-dimensional. A 200-token classification and a 200-token answer with five retrieval calls do not consume the same resources. An agent may hold an inference slot while waiting for a tool, then create fresh inference and database work. Estimate at least prompt tokens, maximum output tokens, expected generation time, retrieval fan-out, tool calls, and retained state. Reconcile the estimate against actual use so bad forecasts do not become permanent policy.

Current vLLM production metrics expose running and waiting requests, queue time, prompt and generation tokens, prefill and decode time, and KV-cache use. These are implementation signals, not a complete business admission policy, but they show why request count alone cannot describe model-serving pressure.

Choose one of five explicit dispositions

Do not hide rejection as a long spinner. RFC 6585 defines HTTP 429 for rate limiting and permits a Retry-After field; its specification deliberately leaves identification and counting policy to the server. Use 429 when a caller or quota is constrained, and choose other status semantics deliberately for service-wide unavailability. More important than the exact code is an honest reason: tenant_quota, class_capacity, dependency_open, deadline_impossible, or queue_full.

A retry time is a promise only if the server can justify it. When recovery is unknown, say so. Clients should not convert every rejection into synchronized retries. Carry the stable logical intent and retry budget described in the retry and idempotency field guide.

Build small queues with expiry, not warehouses of hope

Queue capacity needs three bounds:

1. count or weighted seats: how much concurrent and waiting work may exist;
2. age: how long work may wait before it becomes stale;
3. cost: how many tokens, tool calls, bytes, or downstream reservations the queue represents.

Check the absolute deadline when a request arrives, before every expensive stage, and again when it leaves a queue. Propagate the remaining deadline to retrieval, inference, and tools. Cancel on user disconnect where the product semantics allow it. A queue item without a live consumer, durable business purpose, or authoritative status route is usually waste.

FIFO is not automatically fair. A long document job at the head can block many short lookups; one tenant can fill every slot; repeated attempts can appear as independent demand. Classify by workload and then isolate flows within a class. The goal is not to make every request equally fast. It is to prevent one source or cost shape from removing everyone else’s chance of progress.

Kubernetes APF offers a useful analogy: requests can occupy different numbers of “seats,” priority levels have isolated concurrency, and fair queuing reduces starvation among flows. Copy the principle, not the configuration. AI cost estimates are uncertain and change with context length, cache state, model, batching, and tool behavior.

Make fairness stronger than a premium label

Amazon’s multi-tenant fairness guidance combines per-workload quotas, throttling, and isolation so an unplanned increase from one tenant does not spread failure across all tenants. A practical AI policy can combine:

• a guaranteed share per workload class;
• a per-tenant or per-team quota inside that class;
• borrowable burst capacity while the system is healthy;
• lower priority for above-quota work during contention;
• weighted cost, so one very large request cannot masquerade as one small request;
• a narrow reserve for health, cancellation, incident control, and recovery.

Avoid an unlimited “highest priority” lane. If every team can mark work urgent, priority becomes an attack on fairness. Require a controlled vocabulary, authenticated eligibility, a maximum share, and review of who used the override. Consequential writes are not necessarily the highest-throughput priority; sometimes the correct overload behavior is to stop new writes while preserving read-only diagnosis and cancellation.

Measure fairness at the outcome, not only the admission gate. Track whether tenants and workload classes receive their intended share of useful completions, whether low-volume flows make progress, and whether cost-estimation error systematically harms one language, document type, or customer segment.

Degrade the service, never the control

Graceful degradation must be defined per task before overload:

• a knowledge assistant may switch to permission-safe search results with citations instead of a generated synthesis;
• a drafting tool may return a shorter, explicitly incomplete draft;
• a background evaluation may defer to a durable queue;
• a low-risk classifier may use a tested smaller model;
• a consequential agent action may stop at a saved proposal and require later revalidation.

Do not degrade by dropping authorization, evidence, output validation, tenant isolation, or post-action verification. Do not silently use stale context, broaden a cache key, remove citations, or substitute a model that has not passed the workload’s gate. The model-routing guide explains why provider health and cost are only part of a routing decision; an overload fallback still needs capability and policy compatibility.

Treat every degraded mode as a separate product behavior with its own evaluation, user message, metrics, and rollback. Exercise it regularly. A rarely used emergency path is likely to be the least reliable code in the service.

Worked example: one gateway, four workloads

Consider an illustrative bilingual operations platform with a measured stable capacity of 40 interactive-equivalent seats. A short policy lookup costs one seat, a long-document synthesis three, and a tool-using work-order proposal four plus a separate tool budget. These numbers are local measurements, not universal ratios.

The controller reserves two seats for health, cancellation, and incident operations. It assigns 18 to interactive lookup, eight to work-order proposals, and 12 to batch evaluation. Unused batch seats may be borrowed, but the batch class cannot consume the interactive floor. Each tenant receives a bounded share inside each class, and requests carry absolute deadlines.

A provider slowdown doubles generation time while one tenant submits 500 evaluation cases. The controller:

• admits health and cancellation traffic from the reserve;
• admits interactive questions within tenant shares;
• queues only the interactive burst that can finish before its deadline;
• freezes new consequential dispatch but preserves already approved intent for revalidation;
• defers batch cases to durable storage and reports a status identifier;
• rejects excess above-quota attempts early instead of sending them to the provider;
• cancels generations whose clients disconnect;
• blocks automatic retries from consuming a second seat budget.

The service completes fewer total requests per minute, but more users receive useful on-time outcomes. That is the point. After recovery, batch capacity drains gradually so it does not create a second spike.

Test the cliff, not only the happy slope

Capacity tests should increase offered load beyond the point where useful throughput plateaus. Include:

• a sudden traffic spike and a gradual ramp;
• loss of half the inference capacity;
• provider latency without explicit errors;
• a shift from short prompts to long contexts and long outputs;
• tool and retrieval slowdown behind healthy inference;
• one noisy tenant and many quiet tenants;
• synchronized retries, hedges, and reconnects;
• cancellation during queueing, generation, and tool execution;
• queue-store failure and stale admission state;
• priority abuse and cost underestimation;
• failover into cold caches or cold model replicas;
• recovery while a large deferred backlog exists.

Verify that goodput plateaus rather than collapses, protected classes still progress, expired work is not executed, rejections are cheap, cancellations propagate, and degraded modes remain semantically safe. Connect the results to the operational readiness checklist, including named owners and a tested return to normal mode.

Operate from a decision-complete scorecard

The AI observability guide covers trace linkage across models and tools. Add admission-specific measures:

• offered, admitted, queued, deferred, degraded, rejected, cancelled, and completed work by class and tenant;
• useful completion before deadline, not only server success;
• queue depth, weighted cost, and p50/p95/max age;
• estimated versus actual tokens, duration, tool calls, and seats;
• rejection reason and false-shed rate;
• fairness share and starvation time;
• work completed after disconnect or deadline;
• retry amplification and duplicate logical intents;
• dependency saturation and circuit state;
• degraded-mode outcome quality;
• emergency-reserve consumption and recovery time.

Revisit policy when the request mix changes, a new model changes cost shape, context limits expand, tools are added, tenant size becomes more uneven, a degraded mode fails evaluation, or operators repeatedly use the emergency lane. Do not tune only to reduce rejection rate. A lower rejection rate can mean the system is accepting work it cannot finish.

The admission gate

Before a shared AI service accepts production traffic, answer yes:

1. Does every request carry a trusted class, tenant, deadline, consequence, cost estimate, and cancellation path?
2. Are inference, retrieval, tools, and other dependencies protected by separate budgets?
3. Are queues bounded by count, age, and weighted cost?
4. Can the controller distinguish admit, short queue, safe degradation, durable deferral, and explicit rejection?
5. Are priority and emergency eligibility derived outside the model and capped?
6. Can quiet tenants and small flows progress during a noisy-neighbor event?
7. Do retries share one logical-intent and cumulative budget?
8. Are degraded modes tested without weakening evidence, authorization, or validation?
9. Does overload testing extend beyond saturation and through recovery?
10. Do dashboards report useful on-time completion and fairness, not just throughput?

The queue determines who waits, who succeeds, whose deadline expires, and which work consumes the last healthy unit of capacity. That makes it a product, reliability, and governance policy. Design it explicitly before overload designs it for you.

Source Notes — reviewed August 10, 2026

• Google SRE Book: Addressing Cascading Failures — online chapter from the 2016 SRE book; source for bounded queues, load shedding, graceful degradation, deadlines, and retry-driven overload.
• AWS Builder Center: Using load shedding to avoid overload — published June 12 and modified June 15, 2026; source for goodput, early rejection, bounded work, queue age, prioritization, and overload testing.
• AWS Builder Center: Fairness in multi-tenant systems — published June 12 and modified June 15, 2026; source for workload quotas, throttling, burst treatment, and tenant isolation.
• Kubernetes: API Priority and Fairness — living documentation reviewed on the publication date; source for priority-level isolation, seats, fair queuing, shuffle sharding, bounded queues, and reject-versus-queue behavior.
• Envoy: Overload manager — current 2026 project documentation reviewed on the publication date; source for resource-pressure monitors, triggers, staged actions, and the boundary with circuit breaking.
• Reactive Streams for the JVM — specification release 1.0.4; source for subscriber-controlled demand, asynchronous backpressure, cancellation, and bounded buffering.
• vLLM: Production metrics — living serving documentation reviewed on the publication date; source for running, waiting, queue-time, token, prefill, decode, and KV-cache metrics.
• RFC 6585: Additional HTTP Status Codes — April 2012 standards-track RFC; source for HTTP 429 and optional Retry-After semantics.