A model predicts product returns almost perfectly in testing and then fails on new customers. Somewhere, the evaluation let it use information it will not have once it is deployed. This lesson traces where such information comes from, through features built from the future, customers or events that land on both sides of a split, and preprocessing learned from held-out data, and shows how to choose a split that copies the way the model will be used.
Key terms
Data leakage
Definition Data leakage occurs when information that the intended prediction procedure would not have, at the prediction time and for the kind of cases it will face, influences the fitting, preprocessing, model selection or evaluation of a model. Common routes are features recorded after the prediction time or derived from the target, linked or copied records on both sides of a split, labels used before they would have been final, and preprocessing or selection that learns from held-out cases. Leakage usually makes a reported score better than the score the procedure will achieve in use. It is different from distribution shift, where a correctly held-out later period is harder because conditions changed. A surprisingly high score is a reason to investigate, but by itself it proves neither leakage nor its absence.
In plain words An evaluation is honest only if it imitates the real moment of prediction. Anything the model could not know at that moment, about the case itself, about other cases or about the future, has to stay out of every step the score is meant to judge.
Example A model predicts at checkout whether an order will be returned. If the table includes the date on which a return was received, that field is filled only for returned orders, so whether it is blank gives away the answer and the test accuracy can be nearly perfect. A subtler case keeps every feature legitimate but puts orders of the same customers in both training and test, while the model will be used for new customers. The group split below repairs that one.
Information boundary
Definition For a prediction made at a stated prediction time, the information boundary separates what the system can know at that moment from everything recorded later. A feature value is legitimate for a case only if every raw value used to compute it, including values brought in by joins and aggregates, was recorded and accessible before that case’s prediction time. Records therefore need more than one timestamp, the time an event happened and the time its value became available, and a label has a time at which it became final. A column’s name does not show which side of the boundary its values come from. Its construction recipe and its timestamps do.
In plain words Draw a line at the moment of prediction and ask, for every number the model sees, whether it existed on the near side of that line. A tidy column in the finished table can hide values from the far side.
Example A customer places orders on days 1, 5 and 9, and they arrive 2, 4 and 12 minutes late. The first order was delivered on day 2, before the second checkout. At checkout on day 5, the average delay of the customer’s earlier completed orders is 2 minutes, a legitimate feature. An average over the customer’s whole history is minutes, which uses the outcome of the day 5 order itself and of an order that has not been placed yet. In the finished table both versions appear as one plausible number called average customer delay.
Random row split and group split
Definition A random row split assigns rows to training, validation and test sets by chance, for example by shuffling the rows and cutting the list, ignoring any grouping or time order. Its test score estimates performance on future cases when the rows and those future cases can be treated as exchangeable, meaning that their joint distribution does not change when they are put in a different order, as is true of independent draws from one distribution. A group split first assigns each group, such as a customer, a patient or a machine, to one set, so that all rows of a group land together and no group appears in two sets. It matches deployment to new groups. When customers place repeated orders, a test order from a customer who is also in training typically resembles that customer’s training orders more than a new customer’s order would, so a random row split then estimates performance for familiar customers. Neither split is better in general. If the model will predict further orders of known customers from their legitimate earlier history, the evaluation should copy that setting, and when both known and new customers matter, results are reported for each separately.
In plain words First ask whom the model will meet. If it will meet strangers, keep every person, customer or machine entirely on one side of the split, because otherwise the test quietly lets it recognize old acquaintances.
Example A constructed table has customers A, B and C with two orders each, which arrived 2 and 2 minutes late for A, 8 and 8 for B, and 14 and 14 for C. A lookup rule predicts a known customer’s training outcome and, as a fallback stated in advance, the training mean for an unknown customer. Training on the first order of each customer and testing on the second, the lookup is exact and the test MAE is 0 minutes. A random split of the six rows into three and three would also put at least one customer on both sides every time, since three rows cannot consist of whole pairs. A group split trains on A and B and tests on C. The fallback is minutes, both C orders are 14 minutes late, and the test MAE is 9 minutes. Both numbers are correct. The 0 describes familiar customers and the 9 describes a new one.
Temporal split
Definition A temporal split trains on cases whose prediction times come before a cutoff and evaluates on cases after it, so that the evaluation imitates using the model forward in time. A faithful split checks two clocks. Each case’s features use only information available before its own prediction time, and each training label must have been final by the time the model would have been trained. When features or targets are computed over windows of time, cases near the cutoff can share raw events across it, so a gap is left between the last training case and the first evaluation case, long enough that no raw event contributes to both sides. Group and time boundaries can be needed together, for example when the model will be used on future orders from new customers.
In plain words Train on the past and test on the future, the way the model will be used. The past includes only answers that were already known when the model would have been trained, not answers that arrived later.
Example A return model will be retrained on day 40 and used for checkouts from day 41 on, and a return becomes final 30 days after checkout. A historical rehearsal of that retraining may train only on checkouts on or before day 10, whose labels were final by day 40. Checkouts from days 11 to 40 already have all their features, but their labels were still open, so training on them would pretend the answers arrived early. The rehearsal then scores checkouts from day 41 on, once their own labels have become final.
Duplicate leakage
Definition Duplicate leakage occurs when copies of the same event, or near copies such as one order entered by two systems, end up on both sides of a split. The model can then score well on a copy by recalling the training record instead of predicting. The repair is to settle a defined event identity, such as an order identifier backed by source or ingestion records, and to deduplicate or keep all copies of an event together before rows are assigned to sets. Equal values are not evidence of duplication. Two independent orders with the same price on the same day are two events, and deleting one would remove a legitimate common case.
In plain words One event must not be both studied and tested. Decide what counts as the same event before splitting, and decide it from evidence about identity, not from matching numbers.
Example Suppose a model reproduces the outcome of any event it was trained on and has an MAE of 6 minutes on genuinely new events. If 20 of 100 test rows are copies of training events, the reported MAE is
minutes, although the error on new events is 6 minutes. The copies cut the reported error by 20 percent without any gain in real predictive skill.
Preprocessing leakage
Definition Preprocessing leakage occurs when a preprocessing step, such as centering, scaling, imputation, building a category vocabulary or selecting features, estimates its quantities from data that include validation or test cases. The cure is a train-only fit. The step’s quantities are estimated from the training data alone, stored, and applied unchanged to validation, test and future cases. In cross-validation every training fold fits its own preprocessing before predicting its held-out fold, and a pipeline that bundles the preprocessing with the model makes this boundary explicit. This leakage can happen without any held-out target being used, as with centering or imputation. Its numerical effect ranges from none to large, depending on the step and the model, but in every case the score no longer evaluates the procedure that will run in deployment. A pipeline cannot repair a raw feature that was already computed from future events.
In plain words Every number a preprocessing step learns is part of the model, so it is learned from training data only. Held-out data are then processed the way new data will be, with the stored numbers.
Example A feature has training values 2, 4 and 6 and a held-out value 20. A train-only fit centers with the training mean 4, giving , and for training and for the held-out case, whose centered values need not average zero. Centering with the pooled mean 8 gives , , and , so the held-out case has changed the training data. For a least-squares linear model with an intercept this particular change is harmless, because shifting a feature by a constant only moves the fitted intercept and leaves every prediction the same. Imputation shows a real effect. With observed training values 3, 7 and 11, a blank training entry and a held-out value 100, the train-only median fills the blank with 7, while the pooled median of 3, 7, 11 and 100 is , so the held-out case alters a training example the model learns from.
Check yourself
- A feature has training values 4, 6 and 10 and one blank, and validation values 30 and one blank. With median imputation fitted on the training data only, which value fills each blank? Which value would a fit on all observed values use, and what is wrong with it?
Show the answer
The training median of 4, 6 and 10 is 6. The train-only fit stores 6 and fills both blanks with it, the one in training and the one in validation.
A fit on all observed values uses 4, 6, 10 and 30, whose median is . That puts 8 into the training blank, so a validation value has shaped the data the model learns from. The validation score then evaluates a procedure that peeked at validation data, not the one that will run on new cases, where only the training data are available for fitting.
- A checkout model for returns includes the customer’s total number of returns over the following year. After noticing a suspiciously good score, the team switches from a random row split to a split by customer. Is the problem fixed? What should be done?
Show the answer
No. The feature itself is computed from events after the prediction time, and the total can even include the return of the order being predicted. Under any split, each case still carries information from its own future, so changing the split does not touch the problem.
The repair is in the feature definition. Rebuild it from history known at checkout, for example the number of the customer’s returns that were final before the checkout time, or remove it if that history cannot be reconstructed. Training examples whose own return label would still have been open at the simulated training time are left out of training. The choice between a row split and a customer split is a separate question, decided by whether the model will be used for known or new customers.
- A lab measures 600 parts once each, and a model will predict a quality score for future parts. The parts were measured in 30 sessions of 20, each with its own instrument calibration, and future parts will be measured in new sessions. Is a random row split reasonable? What would change the answer?
Show the answer
Not for this task. Parts from one session share a calibration, so a test part whose session also appears in training resembles its training neighbors in a way that a part from a new session will not. The rows are not exchangeable with the future cases, and a random row split would estimate performance for sessions the model already knows, which is likely optimistic.
A group split by session matches the use. Assign whole sessions, for example 24 sessions to development and 6 to testing, and use whole sessions in any validation folds as well. If calibration drifts over time, the test sessions should also be later than the development sessions. A random row split would fit if the model were meant to predict parts within sessions it has already seen, or if there were good evidence that sessions share no effects, but the separate calibrations are exactly the reason to doubt that.