A Shared Queue Can Distort Your AI Experiment

A team tests an AI triage assistant on half its incoming support requests. Those requests reach a specialist sooner, so the experiment looks successful. But both groups use the same specialists. If assisted requests move forward by pushing other requests back, the control group is no longer an untouched picture of the old workflow.

This is an illustrative problem, not a reported ZharfAI deployment. It asks a precise question: what should a team randomize when AI changes a resource that everyone shares? Assigning users fairly is not enough if their outcomes remain connected through a queue, inventory pool, reviewer or computing budget.

Decide which effect would justify the change

Two questions sound similar but require different evidence. The first is whether an assisted request does better than an unassisted request inside a partly assisted system. The second is whether the whole operation does better when the new policy replaces the old one. A comparison at 50% adoption does not automatically answer the full-adoption question.

Write the target effect in an operational sentence: “For eligible requests entering this service, does using the new triage policy throughout the queue reduce time to accepted resolution without worsening quality or abandonment?” Specify the population, resource pool, outcome window and deployment coverage. That sentence is the estimand—the effect the experiment is intended to estimate.

Also separate elapsed time from labor and output quality. A shorter wait may be valuable even when nobody works fewer hours. Our guide to measuring AI productivity after review and rework handles that accounting problem. Here, the concern comes earlier: whether the comparison itself represents the decision.

Map the connections before drawing the groups

Interference means one unit's treatment can change another unit's outcome. DoorDash's 2018 engineering account explains this through customers sharing a delivery fleet: changing conditions for one group also changes the capacity experienced by the other. It describes assigning algorithms by region and time rather than by individual customer. That is a historical design example, not a universal prescription. DoorDash on network effects.

For an AI service, draw the real resource map. Which requests share reviewers? Can one tenant consume the inference concurrency needed by another? Does a recommendation remove inventory that another customer might buy? Do employees copy useful answers into a common knowledge base? A separate feature flag does not separate those paths.

Distinguish beneficial spillover from competition. Faster assisted work might free specialists for everyone, hiding a useful system-wide gain in a small between-group difference. Alternatively, preferential routing might improve assisted work while delaying the control group, exaggerating the apparent gain. There is no dependable rule that interference always pushes the estimate toward zero.

Do not confuse statistical isolation with customer-data isolation. Two tenants can be perfectly separated for permissions and still compete for the same workers. The experiment needs the dependency map relevant to its outcome, not merely the access-control diagram.

A small example exposes the wrong comparison

Consider 100 hypothetical jobs, with 50 labeled assisted and 50 labeled control. The numbers below are stipulated outcomes in three different worlds, not observations, a fitted queueing model or an AI benchmark. Job mix and quality are held equal solely to make the arithmetic visible.

Inside the mixed system, assisted jobs appear 33.3% faster than controls: (24 - 16) / 24. Yet the whole queue has not improved relative to the all-old condition. Under the separately stipulated all-new condition, the policy gain is 10%: (20 - 18) / 20. The within-queue contrast is not that policy effect.

Now replace only the mixed row with 18 minutes for each group. Shared capacity relief gives the mixed queue a 10% improvement, but the assisted-versus-control contrast is zero. The arithmetic illustrates two possible directions of distortion; it does not say which occurs in your system. Collecting more jobs within the same unsuitable comparison cannot identify an unobserved full-adoption outcome without additional assumptions or a different design.

Choose an assignment boundary that can hold

The following decision table is ZharfAI's proposed design aid. It translates the dependency question into practical choices; it is not a protocol validated by the cited organizations.

Start with the smallest boundary that contains the important interactions, not the smallest identifier available in a logging table. A team is not a separate pool if its difficult cases flow to the same central reviewer. A region is not separate when workers continually cross its edge.

Larger clusters can reduce contamination but leave fewer independent assignments and less precision. Splitting a service into artificial dedicated capacity may make the test cleaner while changing the system being evaluated. State that limitation: evidence for isolated pools may not transport unchanged to a pooled production service.

Switch time blocks only when the system can forget

A switchback experiment changes the policy for an entire pool across randomized time blocks. It can avoid simultaneous competition between treatment groups inside that pool. It does not erase work, memory or behavior left by the previous policy.

Research by Bojinov, Simchi-Levi and Zhao formalizes switchback design using assumptions about how long past treatment affects outcomes and whether future assignments influence present behavior. These assumptions matter before selecting an estimator. The paper's results are not a guarantee for a workflow with persistent learning or unbounded backlog. Design and Analysis of Switchback Experiments.

List what persists: queued jobs, specialist fatigue, cache warmth, inventory depletion, recommendations already accepted and staff knowledge. Measure plausible persistence in operational traces before committing to block length. A five-minute model response does not imply a five-minute effect on the service.

If a transition exclusion period is used to target settled operation, choose it in advance, apply it consistently and report the discarded share. Do not lengthen it after seeing an inconvenient result. Record transition outcomes separately because a system that switches frequently in production must also pay their cost. If effects outlast the available blocks, a longer persistent-cluster design may be more credible than repeated switching.

Avoid predictable old/new alternation that always gives one policy the busy shift. Randomize an allowed schedule, balancing important calendar conditions. NIST's randomized-block guidance explains why known nuisance factors should be controlled through comparable blocks. In this application, a balancing stratum such as weekday and shift is distinct from the pool-time unit that receives treatment. NIST on randomized blocks.

Record assignment, exposure and unfinished work separately

An experiment flag says what should happen, not necessarily what happened. Uber's 2022 account describes exposure logging when a unit accesses a parameter that differs between experiment variants. That distinction is useful for AI fallbacks, unavailable models and workers who ignore a suggestion. Uber's experimentation platform.

A minimal operational record should retain an experiment version, pool identifier, scheduled block, assigned policy, assignment probability, actual policy exposure and relevant timestamps. Add model/configuration version, fallback reason, cross-pool transfer and final outcome status. Prefer protected record references over duplicating customer content in an experiment store.

Predefine which event assigns a job: arrival, first eligible decision or another defensible point. A job that finishes after a boundary should not silently move into the next policy's group. For long-running work, record both its original assignment and later exposures; choose cohort or block-level outcomes to match the intended effect and carryover model. No timestamp convention makes mixed exposure disappear.

Reconcile planned units, assigned units, exposed units and analyzed units. Microsoft documents how assignment, logging, joins and filtering can create sample-ratio mismatch. Diagnose unexplained discrepancies before trusting an effect. Microsoft's mismatch guidance.

Apply that check to the actual design. Equal numbers of randomized time blocks need not contain equal numbers of requests. Traffic differs by time, and treatment may itself change volume. Ticket-count inequality is therefore not automatically an assignment failure. Keep failures, abandoned requests and jobs still open at the cutoff visible rather than filtering to convenient completions.

Analyze the randomized units, not just the log rows

DoorDash's 2019 analysis article explains the nested structure of deliveries within region-time units and the danger of treating correlated observations as independent. Its comparison of methods belongs to that dataset; it does not establish one universally best model. DoorDash on switchback analysis.

A million requests across a handful of pools do not create a million independent policy assignments. Ask the analyst to match uncertainty calculations to clustering, serial dependence, the assignment schedule and the carryover assumptions. Aggregating rows is useful, but adjacent blocks may still be dependent. A standard ticket-level t-test is not rescued by a large export.

Choose weighting before reading outcomes. An equally weighted average across service pools answers a different question from the experience of an average arriving request. If treatment changes volume, even the latter denominator deserves scrutiny. Publish totals, denominators and the intended weighting alongside the estimate.

Test the analysis machinery with A/A exercises, where both labels serve the same policy, and with constructed effects on historical traces. These checks can reveal broken assignment or misleading uncertainty, but cannot prove that a future treatment has no spillovers. Power planning should use the proposed schedule and realistic dependence, with statistical review before launch—not a generic calculator fed the number of tickets.

Inspect the failures a winning average can hide

Before exposure, agree on quality and service limits: serious errors, abandonment, old backlog, high-percentile waiting time and cross-pool transfers. Keep existing safety and approval controls. Randomization is not permission to send unsafe answers or postpone urgent work.

Watch for a policy that finishes easy jobs now and leaves difficult ones for the next block. Report outcomes for eligible arrivals with an appropriate follow-up period, plus queue-wide backlog and completion measures. A shorter average among completed jobs can coexist with worse service for people who remain waiting.

Freeze or track other experiments that share the resource. A model upgrade, staffing change or new routing policy can alter the treatment actually delivered. The AI change-control guide explains what needs to move together in a release. A canary limits exposure while checking health; it does not by itself establish the business effect of full adoption.

Make the rollout claim no broader than the evidence

The decision record should name the tested population, assignment unit, adoption coverage, observation period, effect range, quality limits and unresolved interference. If the result only describes a mixed queue, say so. Do not multiply its percentage across every employee or request as though full adoption had been tested.

Expand only where the estimated benefit is useful, uncertainty is acceptable and operational constraints hold. Otherwise, change the assignment boundary, collect longer follow-up or report the result as inconclusive. If persistent learning makes switching implausible, acknowledge that design limit rather than declaring a clean control group by convention.

After rollout, monitor the deployed service for queue length, quality, arrival mix and capacity changes. Monitoring can challenge whether the experimental result remains relevant; an encouraging before/after trend is not a replacement randomized experiment.

The most useful question at the review meeting is simple: did AI improve this service, or did it change who waited? Answering it requires an experimental boundary that follows the work.

Source notes — reviewed September 19, 2026

• DoorDash, network effects and switchbacks, February 13, 2018. Historical marketplace implementation.
• DoorDash, switchback analysis, February 20, 2019. Nested data and analysis choices.
• Bojinov, Simchi-Levi and Zhao, switchback design, first submitted August 31, 2020; version 4 dated September 17, 2025. Original research with explicit assumptions.
• NIST/SEMATECH, randomized block designs, undated handbook page. General experimental-design guidance.
• Microsoft Research, sample-ratio mismatch, September 14, 2020. Experiment-integrity guidance.
• Uber, experimentation platform, July 21, 2022. Assignment and exposure engineering. The decision aid, AI applications and numerical example above are ZharfAI analysis, not validated customer results.