A lower p95 does not always mean a faster assistant. Learn how to pool latency measurements, expose traffic-mix changes and judge the evidence before a release.
An assistant's release dashboard shows a lower 95th-percentile response time. The product manager asks whether the new version is ready for everyone. Before answering, ask how that number was produced: from the requests themselves, or by averaging the p95 values reported by several servers?
Even a correctly calculated percentile can support the wrong conclusion when the workload changes. The constructed example below moves from a combined p95 of 1,000 milliseconds to 150 milliseconds while both request groups become slower. It is not a ZharfAI benchmark. It is a way to test whether the evidence behind “faster” means what the release team thinks it means.
Sort request durations from smallest to largest. A percentile describes a position in that ordered distribution, rather than total elapsed time divided by request count. For the examples here, we use the nearest-rank convention: round 0.95 × N upward to an integer and take the observation at that position.
The convention matters. NIST's percentile handbook section describes multiple estimation and interpolation methods. Small samples can produce different results under different conventions. A report should therefore identify its estimator and sample count, not merely attach the label p95 to a number.
A single percentile does not retain the distribution around it. It cannot tell you how many observations were extremely fast, how widely the durations were spread or how slow the slowest request was. When each server exports only a precomputed p95, information needed for the fleet-wide percentile has already disappeared. Weighting those values by request count cannot restore it.
This is a different issue from choosing a high percentile instead of an average. A team can correctly decide to measure tail latency and still calculate that metric incorrectly when combining servers, regions or time windows.
Consider one deliberately simple interval. Server A completes 100 requests, each taking 100 ms. Server B completes ten requests, each taking 1,000 ms. Constant durations make the arithmetic visible; they are not intended to simulate a real service.
The servers' individual p95 values are 100 and 1,000 ms. Their unweighted average is 550 ms. A request-count-weighted average is approximately 181.8 ms: (100 × 100 + 10 × 1,000) / 110. In this particular fixture, that also happens to be the mean request duration. Neither number is the combined p95.
There are 110 observations. The nearest-rank p95 is observation 105. The first 100 observations are 100 ms, and the remaining ten are 1,000 ms. The combined p95 is therefore 1,000 ms.
Prometheus's histogram guidance explains the aggregation limitation of precomputed summary quantiles. Preserve observations or a mergeable representation of their distribution, then calculate the percentile over the intended population. The same principle applies across time: averaging a day's five-minute p95 values does not produce the day's p95. A quiet interval and a busy interval do not supply interchangeable summaries.
Now treat the two populations as short-answer and long-task cohorts. In a second interval, short answers become much more common. Every request within each cohort has the duration shown below.
| Interval | Short-answer cohort | Long-task cohort | Combined p95 |
|---|---|---|---|
| First | 100 requests at 100 ms each | 10 requests at 1,000 ms each | 1,000 ms |
| Second | 990 requests at 150 ms each | 10 requests at 1,200 ms each | 150 ms |
In the second interval, observation 950 belongs to the short-answer cohort. The combined percentile really is 150 ms. The arithmetic is sound; the claim that the release made the work faster is not. Short answers slowed from 100 to 150 ms, and long tasks slowed from 1,000 to 1,200 ms.
Keep both views. The overall distribution describes actual traffic, while cohort comparisons help assess like-for-like performance. If a comparison uses a fixed reference mix, reweight the cohort distributions to those shares before calculating the combined percentile. Do not average their percentiles under a new set of weights.
Define cohorts before inspecting the result: task type, approximate input size or tool route are possible choices. Be careful with generated answer length. A release may itself change that length, so conditioning on the observed output afterward can remove part of the effect under investigation. Comparing equivalent tasks and reporting actual output lengths provide complementary evidence; they are not the same adjustment.
For a streaming assistant, time to the first response fragment differs from time to the complete answer. A quick “working on it” message is not necessarily useful answer content. If one version stops its timer at completion and another stops at its first output, the comparison has changed its meaning.
Write down the exact events. One possible server-side contract starts when the service accepts the request and ends when a complete, valid answer is ready. Separately record model time, queueing, tool calls and delivery. This is a proposed measurement contract, not a universal definition every product must adopt.
If the promise concerns the user's wait, server-only measurement is insufficient. Measure network and presentation separately or limit the claim to the server boundary. Our inference-latency engineering guide explains how to locate time within the path and choose an optimization. Here the prior question is whether the two reported numbers measure the same thing.
Retries need the same discipline. Three model attempts serving one user request are not three user experiences. Attempt duration is useful diagnostic information, but the user-level duration should include the whole wait. Nor can stage percentiles simply be added and called an end-to-end percentile: the slowest stages need not occur in the same requests.
Retaining every raw duration has storage and processing costs. A histogram instead preserves counts across duration ranges. Those ranges, the measurement meaning and the reporting period must be compatible before distributions are combined. Similar metric names are not sufficient evidence of compatibility.
The OpenTelemetry metrics data model specifies histogram counts, boundaries, time windows and aggregation temporality. Delta measurements for separate periods differ from cumulative measurements since a starting point. A collection pipeline must handle that distinction and counter resets correctly; adding successive cumulative snapshots can count the same requests repeatedly.
Our practical recommendation is a small measurement contract alongside the metric: duration unit, start and end events, cohort, outcome, reporting interval and instrumentation version. Use bounded, predefined labels. A prompt or customer identifier should not become an endlessly expanding metric label, creating both operational cost and unnecessary exposure of sensitive information.
Resolution matters near a decision boundary. Suppose the deadline is two seconds, but a histogram bucket spans one to five seconds. That bucket's count cannot reveal how many requests completed before two seconds. Displaying an estimate to three decimal places does not recover the missing detail. An appropriate boundary or a direct on-time-completion counter gives stronger evidence for that particular decision.
Keep the representation choice subordinate to the question. Fine-grained distributions improve resolution but cost more to retain. Coarse summaries are cheaper but may not settle a close release decision. If the available representation cannot distinguish pass from fail at the chosen threshold, report that uncertainty and collect suitable evidence.
A release could reject difficult tasks sooner and lower an overall latency figure. That is not automatically an improvement. Google SRE's monitoring chapter distinguishes successful-request latency from failed-request latency and explains how quick errors can mislead. Filtering failures out of the entire report creates another blind spot.
Show successful-answer percentiles alongside error counts and durations, timeouts and abandoned requests. An unfinished request has no known completion time. Do not record it as zero, or label its cutoff time as successful completion. If you record time to termination, name the outcome accurately.
For an operational decision, add a complementary ratio: valid answers completed by the deadline divided by all eligible requests. Fix eligibility in advance. For an arrival-based cohort, wait until its members have reached their deadlines or separately identify still-open requests. Comparing a fully observed cohort with a partly observed one is not a fair acceptance test.
Early rejection under overload might be an intentional protective policy. Explain it as a change in coverage, not faster service for everyone. The product owner can then judge the tradeoff between access and waiting time without having it concealed inside an attractive percentile.
A test in which each virtual user waits for completion before sending another request reduces its arrival rate when the service slows. Grafana k6's open-versus-closed model documentation explains this coupling and describes arrival-rate executors that separate arrivals from iteration completion.
A closed model is not inherently wrong: it can match users who actually wait before doing the next task. But if the decision concerns an independently arriving workload, a virtual-user count alone is inadequate. Report planned arrivals, actual requests sent and completed requests. A load generator failing to offer the intended traffic is not proof that the service handled it.
Preserve cache conditions, model warmness, input sizes and answer quality with the result. A shorter but incomplete answer may finish sooner; that does not make it acceptable. The quality floor must remain beside the speed requirement. None of this article's numerical fixtures is evidence from such a load test.
A measurement method can be correct while the release is genuinely slower. A release can also be better while the current report cannot establish that improvement. Treating these as different outcomes prevents a dispute about arithmetic from turning into an unsupported verdict about the product.
The following is ZharfAI's proposed review table, not a certification standard or a default policy imposed by the cited tools.
| Available evidence | Proposed decision |
|---|---|
| Only averaged server or interval percentiles | Do not accept the improvement claim; recompute from mergeable data |
| Better global p95, worse important cohort | Investigate mix and cohort impact; the global value is not rollout permission |
| Faster successful answers, more errors or timeouts | Review valid, on-time completion across eligible requests |
| Changed clock boundary or test workload | Establish a comparable baseline |
| Same boundary and load, complete accounting, acceptable cohorts and quality | Consider limited expansion with explicit stop conditions |
Attach counts, time windows and measurement versions to the report. A handful of requests is weak evidence about the tail. When a small apparent gain is below repeat-run variation or estimation uncertainty, avoid presenting it as established improvement. There is no single magical minimum sample size appropriate to every workload.
To understand a particular slowdown, inspect the path taken by that request. Our agent-observability guide addresses that diagnostic task. Selected traces do not replace accounting representative of the full traffic population. Then connect the release decision to the model, prompt, tool route and measurement configuration, as described in our change-control and progressive-rollout guide.
Revisit the comparison after changes to routing, instrumentation, bucket boundaries or cohort shares. A useful performance report does more than show a smaller number: it identifies which work became faster, for whom, at what success rate, and what remains unknown.
This is a standard practical Blog guide. The numerical fixtures, decision table and proposed measurement contract are ZharfAI analysis, not customer findings or a product-performance test. All five sources below were opened directly. Undated and living references are identified by the review date; no new-release claim is attached to them.
Separate signature checks, durable receipt and permission to act. A practical webhook admission design keeps repeated deliveries from becoming fresh AI workflow requests.
Read More
Repeated JSON keys can disappear before validation begins. Reject ambiguity at the raw-input boundary and keep reviewers and executing tools on the same accepted data.
Read More
Decide when streamed AI output may become a final answer by separating byte decoding, event boundaries, stop reasons and application acceptance.
Read MoreIf you want the agents and automation in this guide working for your engineering team or business processes, start with a small pilot you can measure.