Which AI Forecast Should Set Tomorrow’s Capacity?

A support manager asks an AI forecasting system how many requests will arrive tomorrow. It returns 125. The planning spreadsheet reserves capacity for 125 requests. But an unused slot and a request left waiting do not have the same consequence. Even a statistically sound average can be the wrong number to turn directly into a capacity plan.

This guide is for operations owners and data teams choosing that number. It uses a constructed service-planning example, not a customer result or a model benchmark. The decision is which forecast quantity to request, how to evaluate it, and when the evidence is insufficient to change the operating plan.

First decide what the forecast number means

“Tomorrow's volume” is not a complete specification. Define the service, arrival channel, time zone, cutoff, forecast horizon and unit. Count incoming requests rather than completed requests if arrivals are the target: completions also depend on capacity. Separate reopened work, duplicates and carried-over backlog so that the target does not silently change when the team changes its process.

Then distinguish three outputs. A mean is the probability-weighted average of possible outcomes. A median divides the distribution around its middle. An upper quantile, such as the 80th percentile, identifies a threshold at or above which the cumulative probability reaches 80%. They need not be the same number, particularly when occasional surges stretch the upper tail.

The statistical reason to name the target is well established. Tilmann Gneiting's research on evaluating point forecasts explains why the scoring rule and requested statistical quantity must match. Squared error targets the mean; absolute error targets a median. Neither automatically expresses a manager's asymmetric consequences.

This applies to classical time-series models, machine-learning regressors and AI forecasting services alike. An assistant saying it is “80% confident” in a number has not supplied a validated 80th-percentile forecast. Require a defined forecasting method and an evaluation history, not a confidence adjective generated in prose.

Translate unequal consequences into a percentile

Consider a deliberately restricted planning problem. Capacity is chosen once before arrivals, unused capacity has a constant per-unit cost, and unmet work has another constant per-unit cost. There are no shared constraints, fixed setup charges or carryover effects. Write the loss as:

loss = shortage_cost × max(actual − capacity, 0) + excess_cost × max(capacity − actual, 0)

For positive costs in this single-period model, an optimal capacity is the quantile at p = shortage_cost / (shortage_cost + excess_cost). If shortage costs four units and excess costs one, the target is the 80th percentile. With equal costs it is a median. The ratio chooses the percentile; the forecast distribution supplies its value.

This is a mathematical decision model, not a universal staffing rule. Its connection to asymmetric forecast evaluation also appears in Amazon Forecast's metric documentation, which distinguishes losses at specific quantiles from averages across several quantiles. We cite those definitions, not a recommendation to purchase that service.

Ask the process owner what the costs include. An idle slot might be usable for other work; delayed requests might accumulate rather than disappear. Estimates may be uncertain. Show how the selected capacity changes under plausible cost ratios instead of asking the language model to invent a monetary value for customer frustration.

Work the choice through four possible days

Suppose tomorrow's arrivals can be 80, 100, 120 or 200, each with probability one quarter. These four outcomes are invented to make the arithmetic inspectable. The mean is 125; any value from 100 to 120 is a median. Using the smallest value whose cumulative probability reaches the target, the 80th percentile is 200.

Under the four-to-one shortage/excess cost model, compare three capacity choices:

For capacity 125, the unused slots across the four days are 45, 25, 5 and zero. Their average is 18.75. Unmet requests are zero, zero, zero and 75, also averaging 18.75. Total expected loss is therefore 18.75 + 4 × 18.75 = 93.75. Reserving 200 has more unused capacity but lower loss under these particular assumptions.

That does not mean 200 is a more accurate estimate of the mean. The mean-squared error is 2,075 at 125 and 7,700 at 200. The two numbers solve different problems. This is a comparison of decisions from one distribution, not evidence that one forecasting model outperforms another.

Now change shortage cost from four to two while keeping excess cost one. The target percentile becomes two thirds, selecting 120 in this distribution. Its expected loss becomes 55, versus 75 at capacity 200. Nothing about the possible arrivals changed. The decision changed because the consequences did.

This discrete example also exposes a terminology trap. The 80th-percentile capacity covers all four listed outcomes, not exactly 80% of them. Probability mass at a threshold can make coverage exceed its nominal level. Do not turn a quantile label into a promise of an exact observed frequency.

A prediction interval is not a capacity instruction

An interval describes uncertainty about the future outcome; a capacity decision expresses a tradeoff. In a continuous distribution, a central 80% prediction interval runs from the 10th to the 90th percentile. Its upper endpoint is not the 80th percentile. Nor is a confidence interval for an estimated mean a prediction interval for tomorrow's arrivals. The FPP3 chapter on prediction intervals explains their role and dependence on forecast horizon.

Show the selected quantile and an uncertainty range separately. Label each with its horizon and target. Avoid a symmetric “plus or minus” band unless the forecasting method justifies it. Counts can be skewed and bounded below by zero; a convenient normal-shaped band may be poorly matched to them.

Also distinguish a band for each future day from a guarantee that an entire week stays inside its bands. Do not add daily 90th percentiles to obtain a weekly 90th percentile without modeling dependence. If two equally likely scenarios put 100 requests in either of two periods and zero in the other, each period's 90th percentile is 100, but the total is always 100—not 200. Aggregate scenario paths or fit the required total directly.

Grade the forecast at the percentile you will use

For a forecast quantile q at probability p, use the unscaled pinball loss:

pinball = p × max(actual − q, 0) + (1 − p) × max(q − actual, 0)

At p = 0.8, underprediction receives four times the weight of equal-sized overprediction. In the four-day example, average pinball loss is 19 at capacity 120, 18.75 at 125 and 15 at 200. Multiplying by five reproduces the corresponding decision losses. This identity holds because we deliberately made the operating loss linear with those costs.

The scikit-learn mean_pinball_loss reference exposes the target through alpha; set it explicitly rather than silently using the default median target. Its sample weights and output aggregation also need deliberate choices. A large service should not dominate every comparison merely because the evaluator pooled all rows without considering the decision.

Document the scoring convention. FPP3's distributional accuracy chapter presents a quantile score with a factor of two, while noting that it is often omitted. At the median, the unscaled formula above equals half the absolute error. Different constants preserve a ranking but change the reported number. Normalizing by observed volume changes interpretation again; a zero-volume denominator needs explicit handling. A dashboard number is not comparable until its formula, weights and evaluated cases are known.

Replay the decision at the time it was made

Create a forecast ledger with one row per service, issuance time, target interval, horizon and requested quantile. Preserve the predicted value, model version, input snapshot reference and eventual observed target. Keep issued forecasts immutable; store revisions as new issuances. The latest estimate is not a substitute for the forecast available when capacity was committed.

Use rolling-origin evaluation: move the forecast cutoff through history, fit with earlier information and evaluate the actual planning horizon. If the rota is fixed seven days ahead, excellent next-day performance is not the required evidence. Refit preprocessing and any interval calibration inside the permitted historical window too.

The same rule applies to business features. A planned campaign known on Monday can be an input to Monday's forecast; the campaign's final response count cannot. Our guide to decision-time snapshots explains how later corrections differ from information available at the decision. Preserve both when reconstructing a test.

Compare against the existing operating policy and a simple seasonal baseline on the same dates. Choose model settings on development windows, then retain a later untouched evaluation window. Do not repeatedly tune a percentile or buffer against the final report and describe the resulting score as independent evidence.

Inspect coverage, width and the difficult slices

Pinball loss is useful, but one aggregate cannot diagnose every failure. For each quantile, inspect how often observed outcomes fall below or at the prediction, accounting for ties in discrete counts. Separate weekday, service type and horizon. A plausible overall frequency can combine a model that underprepares one team and overprepares another.

For intervals, report coverage and width together. A very wide interval can cover almost everything while providing little guidance. The interval score described in the distributional-accuracy source combines width with penalties for observations outside the endpoints. Still inspect which side was missed and by how much; the operational consequences may be asymmetric even when the interval is central.

Small slices do not support confident reliability claims. Report observation counts and use uncertainty estimates that respect time dependence; a run of related days is not a collection of independent coin tosses. Predefine important slices instead of discovering a flattering subgroup after seeing the results. Separate unusual-event stress scenarios from measured historical performance.

Observation quality comes first. If a channel stopped accepting requests at its limit, recorded arrivals may omit unmet demand. Our stockout and censored-demand guide describes the analogous problem in sales. An impressive score against a capacity-limited target may simply reward repeating the old capacity limit.

Turn the forecast into a constrained operating decision

The forecasting service should return named quantities, not one unlabeled number: target, horizon, mean if requested, selected quantiles, interval definition and issuance identity. A separate planner applies capacity constraints and the approved cost assumptions. The assistant can explain the recommendation, but it should not silently change a percentile, round away the unit or label a scenario a measured result.

For actual staffing, request counts are only a start. Handling times, skill mix, breaks, shift sizes, response-time targets and queue carryover can invalidate the one-period linear model. Use a constrained scheduling or queueing model where needed. Do not “fix” infeasibility by relabeling a lower feasible capacity as the same protective percentile.

Before a trial, agree on allowable changes, who can override them and which outcomes count: backlog, waiting time, completed work and genuinely unused capacity. Evaluate the forecasting layer and the decision policy separately. A good distribution can feed a bad planner; a conservative planner can conceal a weak forecast.

Revisit the cost ratio and evaluation scope after channel changes, new service promises, automation or shifts in request mix. Track this alongside the signals in our post-deployment monitoring guide. The useful closing question is not “How accurate is the AI?” It is “Which quantity did we ask it to forecast, and does the plan built from that quantity still fit our consequences?”

Source notes — reviewed September 25, 2026

The numerical examples, planning separation and operating checklist are ZharfAI analysis. No forecasting service was benchmarked and no customer saving is claimed. This is an operational field guide, not a model-release report or financial advice.

• Gneiting, Making and Evaluating Point Forecasts: first submitted December 4, 2009; version two March 7, 2010. The cited claim concerns matching the target and scoring rule.
• Hyndman and Athanasopoulos, prediction intervals: Forecasting: Principles and Practice, third edition, 2021; online book updated September 16, 2026.
• The same authors, distributional forecast accuracy: quantile and interval scores, same online edition.
• The same authors, time-series cross-validation: rolling-origin and horizon-specific evaluation. These three chapters count as one organizational source.
• scikit-learn, mean_pinball_loss: version 1.9.1 documentation accessed on the review date; no fixed page-publication date stated.
• AWS, evaluating predictor accuracy: rolling, undated Amazon Forecast documentation, accessed on the review date; used for metric semantics, not service availability or procurement advice.