New Customers or New Orders? Split Your Model Test Accordingly

A sales team wants to use a model on customers it has never served. The evaluation report looks reassuring: training rows and test rows are different. But both sets contain orders from the same customers. The report has answered a narrower question than the sales team asked.

This is a constructed example, not a reported customer incident. It captures a decision for engineers and product owners: does the test need to represent another order from a known customer, a genuinely unseen customer, or both? Choose that promise before choosing a split percentage. Otherwise a precisely calculated score can support the wrong launch decision.

A new row does not establish a new customer

Order identifiers can be unique while the people, organizations, account histories and document templates behind them recur. Removing an exact duplicate is useful, but it does not make related observations independent. A model may exploit a customer's stable purchasing pattern without learning what transfers to another customer.

Kapoor and Narayanan's research taxonomy distinguishes duplicates, temporal leakage and dependence between training and test observations. Crucially, it relates dependence to the population about which a claim is made. Their work concerns ML-based science; it is not an estimate of failure rates in business software. Leakage and the Reproducibility Crisis in ML-based Science.

Customer overlap is therefore not automatically a defect. For a service predicting tomorrow's order from an established account, legitimately available account history may be essential. For a product promising performance on unfamiliar accounts, that overlap cannot supply the promised evidence. The problem is the mismatch, not the mere presence of a customer identifier.

This differs from the exposure problem in our benchmark-contamination guide. Keeping answers away from a model or its developers does not decide which related records belong together. Here the evaluation's unit of separation is the central design choice.

Write the deployment promise before selecting a splitter

Describe what arrives at prediction time, what the service already knows and which outcome it must predict. “Predict an order's eventual return using information available when the order is placed” is more useful than “predict returns accurately.” Then say whether the customer is already represented in development data.

The following is ZharfAI's proposed decision table, not a benchmark result or a universal rule for every dataset.

Define “new” carefully. New to the fitted model, new to the organization and placing a first-ever order are different conditions. A held-out longstanding customer can test unfamiliarity to the model, but may still have rich history that a newly acquired customer lacks. Report those differences instead of quietly treating the cohorts as equivalent.

For a mixed service, retain the separate scores even when management needs one summary. The summary should use a declared deployment mix, not whichever mix makes the model look best. Revisit that weighting when customer acquisition or repeat purchasing changes.

Four customers and eight orders expose the ambiguity

Imagine four customers, A through D, with two distinct orders each. These labels and allocations are deliberately artificial. No model is trained, and no accuracy improvement is claimed. They illustrate only what information crosses a boundary.

Both allocations have four training rows and four test rows. Neither repeats an order identifier across the boundary. In the first, however, all four test customers appear in training: customer overlap is 100%. In the second, neither of the two test customers appears there: overlap is 0%. The row counts alone hide that change.

This is one possible row allocation, not the inevitable outcome of every random split. Nor is the second allocation automatically sufficient: it says nothing about order dates, feature availability or the representativeness of C and D. A real development process also needs validation data; this tiny table omits that third partition solely to make the boundary visible.

Use a similarly small fixture in your own tests. Include an alias for A and an augmented copy of C1. A splitter that keeps different spellings or generated variants apart may still put the same underlying entity on both sides. A test of the split should fail before the training job starts.

Give related records a defensible group identity

Choose the group according to the generalization claim: customer, company, conversation, source document or another originating unit. For document classifiers, pages and synthetic rewrites from one source may belong to one family. For an account model, separate subsidiaries may or may not count as unfamiliar organizations; that is a product definition, not a convenient database default.

The GroupKFold documentation specifies nonoverlapping groups within each training/test fold and requires at least as many distinct groups as folds. Each group appears in a test fold once. This guarantees separation of the group labels supplied to it, not correctness of those labels or chronological separation.

Resolve identities using documented evidence and keep the mapping version with the split. Our customer-record matching guide explains why similar names do not prove identity. False separation can hide leakage; false merging can turn unrelated customers into one large group and distort coverage.

Do not convert every missing customer ID into a unique row ID and call the result a new-customer test. Equally, assigning all missing IDs to one giant customer invents a relationship. Investigate, use justified source-family evidence, or report an unresolved cohort separately. Publish how many rows and customers were excluded and why. Quietly removing difficult cases changes the population being tested.

Time has two boundaries, not one date column

The first boundary is the prediction moment: which features were available when the order arrived? The second is the model's training cutoff: which outcomes were already known when the model was fitted? An order can predate the cutoff while its return outcome becomes known afterward.

Consider an illustrative replay with training frozen on July 1. An order placed June 25 is assigned a label for whether it is returned within 30 days; that label is not finalized until July 25. A dataset exported in August contains it, but the July 1 model could not have learned from that finalized label. Exclude it from that training snapshot or apply a separately justified label-availability rule.

When scoring a July 10 prediction later, using its subsequently matured outcome as the test label is appropriate. Feeding that outcome, or a customer summary containing later events, into the July 10 input is not. Preserve feature versions, prediction time and label-availability time separately, as described in our temporal-truth guide.

The TimeSeriesSplit reference describes ordered splits with expanding training sets. Its gap counts samples, not days; equal spacing is needed for comparable test durations, and its groups argument is ignored. For irregular order arrivals, define calendar windows explicitly. A seven-row gap does not mean seven days, and a temporal splitter does not automatically isolate customers.

Combine customer exclusion with a chronological replay

For a future-unseen-customer claim, hold out the relevant customer groups from all development, including preprocessing, feature selection and threshold tuning. Then evaluate eligible later orders from those groups using features reconstructed at their prediction times. Rows outside the intended window do not become training rows merely because they are inconvenient to score.

This combination is our proposed operating method. Neither of the two splitters above, used alone, establishes both conditions. A practical split manifest should record source snapshot, customer-mapping version, group assignments, training cutoff, test interval, label maturity rule, exclusion reasons and random seed where relevant. Save the actual row assignments too: a seed does not identify a dataset that keeps changing.

If the promise concerns first-time buyers, select a cohort meeting that definition, not just customers whose identifiers happen to be absent from one table. If earlier history is unavailable, say that newness is unverified. For rolling replays, also declare whether the model is retrained at each cutoff; a fixed model and a continually updated model are different services.

Keep development and final assessment distinct. Google's dataset-splitting guidance separates training, validation and test roles, warns about duplicate examples and explains that repeated adaptive use wears out a test set. Once final-test feedback changes a feature or threshold, record that use and seek fresh assessment evidence rather than still calling the set untouched.

Fit transformations inside the same boundary

A customer-separated test can still leak information through a transformation learned from the whole dataset. Imputation, scaling and feature selection are part of model development, even when their code runs before the estimator. Scikit-learn's common-pitfalls guide recommends learning these transformations from training data and using a pipeline to keep fitting within the appropriate cross-validation subset.

A pipeline cannot repair an incorrect group definition or a future-derived feature already stored in the input table. Review the dataset builder as well as the model code. For each feature, identify its source, availability rule, aggregation window and whether any fitted state depends on held-out data.

Create training augmentations only from eligible training sources. Keep each generated example's parent identifier so a later export cannot unknowingly scatter a family across partitions. If a summarizer creates customer histories, make its input cutoff inspectable. A fluent summary of the future is still information the deployed model would not have had.

Count customers as well as rows when reporting results

A test containing many orders from a handful of customers is not evidence from the same number of independent customers. Report distinct customer counts, orders per customer, class counts and time coverage beside the chosen task metric. Avoid claiming precise population performance from a small or highly concentrated cohort.

The weighting also changes the question. In another arithmetic-only illustration, suppose one customer contributes 90 orders with a mean absolute prediction error of 1 day, while another contributes 10 orders with an error of 5 days. The order-weighted mean is 1.4 days. Giving each customer equal weight yields 3 days. Neither calculation is inherently dishonest; they describe different priorities.

Declare which priority the product serves before comparison, and show both when concentration matters. Keep known-customer and unseen-customer results separate. Compare candidate models on the same eligible rows and groups; a better score on an easier retained cohort is not evidence of improvement on the original population.

Rare outcomes create a real constraint. If too few customer groups contain positive cases, some folds may not support the intended metric. Do not split those customers across folds merely to make class proportions look tidy. Reduce the number of folds when justified, collect more independent groups, or state that the evidence is insufficient. Inspect large-group imbalance rather than promising that a group splitter will make every fold equally informative.

Make the split itself a release artifact

Before accepting a score, run explicit checks against the manifest. For a new-customer claim, require no customer overlap across development and final test. For every claim, detect unintended duplicate source examples, verify feature-time eligibility, confirm training-label availability and reconcile exclusions with the source snapshot. Check validation boundaries too, not just the last train/test comparison.

Test those checks with intentional fixture errors: an alias crossing partitions, a late label admitted into training, a future account aggregate and a copied document under a new filename. The expected result is a specific refusal or an explicitly classified exception. Record the failures rather than silently dropping rows until the test passes.

Reassess the split when identity resolution changes, a new channel arrives, label delays shift or the business expands from repeat buyers to acquisition. Those changes can invalidate the old evaluation question even if the model code stays identical. The durable acceptance condition is not “the test rows were different.” It is “this test represents the customers, information and period for which we are making this claim.”

Source notes — reviewed October 2, 2026

This is a standard practical Blog guide. The allocation tables, dates, arithmetic and operating checks are ZharfAI analysis and hypothetical examples, not measured customer results. The five sources represent scikit-learn, Google and Princeton researchers; three documentation pages from one project are not three independent organizations.

• Kapoor and Narayanan — Leakage and the Reproducibility Crisis in ML-based Science, arXiv version 1, July 14, 2022; taxonomy and the scope of generalization claims.
• Scikit-learn — GroupKFold, living stable documentation displaying version 1.9.1 at review; group separation.
• Scikit-learn — TimeSeriesSplit, same reviewed documentation version; ordered splits and sample-count gaps.
• Scikit-learn — Common pitfalls, living documentation reviewed on the date above; fitting preprocessing within training boundaries.
• Google — Dividing the original dataset, last updated December 3, 2025; development partitions, duplicates and repeated test use.