Validation, Overfitting and Honest Prediction

Definition In a prediction problem, training data are the observations used to build the model. Their outcomes and predictor values determine the fitted coefficients and anything else learned in fitting, such as imputation values or scaling constants. Test data are observations set aside before any of these decisions and used to assess the final procedure. Neither their outcomes nor summaries of their predictor values play any part in fitting, preprocessing or model choice. When a model has to be chosen, the choice uses validation data or cross-validation within the development data, which is everything except the test set. A split assigns each observation, or each linked group of observations, to one role before any outcome is used to choose a model, and it must match the prediction task. To predict for new people, all records of a person go to the same side. To forecast a later period, the test period comes after the training period. A random split of individual rows suits cases that behave like independent draws from the population of interest. Once test results have influenced the choice of model, those cases have served as validation data, and an untouched final assessment needs fresh test data.

Definition A loss function assigns to an observed outcome $y$ and a prediction $\hat{y}$ a number $L(y, \hat{y}) \ge 0$ that measures the cost of the miss. A set of $m$ predictions is scored by its average loss over the evaluated cases. With prediction error $e_i = y_i - \hat{y}_i$, the common scores for a numerical outcome are

The RMSE and the MAE are in the units of the outcome, and the MSE is in squared units. For a binary outcome coded $y_i = 0$ or $1$ and a forecast probability $p_i$ of the event $y_i = 1$, the Brier score is the average of $(p_i - y_i)^2$. The divisor is $m$, the number of evaluated predictions, because the score is an average loss and not an estimate of an error variance. This average over cases is also different from the mean squared error of an estimator, which is an expectation. Competing models should be scored with the same loss on the same evaluation cases, next to a simple baseline such as predicting the training mean.

Definition A fitting or model-selection procedure overfits when it adapts to features of the training sample that do not hold in the population where predictions will be made, such as chance patterns in the noise, so that it predicts new cases worse than a less flexible procedure would. Training error alone is not enough to detect it. For least squares, adding a predictor never increases the training residual sum of squares, and the least-squares polynomial of degree $n - 1$ passes exactly through $n$ points with distinct $x$ values. So the fit with the smallest training error tends to look better on the training data than it will on new data. In a single split the held-out error can still come out larger or smaller than the training error, since it depends on which cases land in each set.

Definition In $k$-fold cross-validation, the $n$ development observations are divided into $k \ge 2$ disjoint folds. For each $j = 1, \dots, k$, the whole procedure, including preprocessing and any data-driven variable selection or tuning it contains, is fitted to the other $k - 1$ folds and used to predict the cases in fold $j$. If fold $j$ has $m_j$ cases with average loss $\bar{L}_j$, the cross-validation estimate is the average loss over all held-out predictions,

which equals the plain mean of the fold scores when the folds have equal size. Leave-one-out cross-validation is the case $k = n$. Folds follow the same rules as a test split, so linked records stay in one fold. Time-ordered data need forward splits in place of this rotation, with each validation period coming after all the data used to fit its model. The estimate describes the procedure trained on about $(k - 1)/k$ of the data, not one fixed fitted model. If cross-validation is used to choose among models, the winner’s score tends to be optimistic, so the chosen model is assessed on an untouched test set, or the whole selection procedure is assessed with nested cross-validation. The fold scores share training data and are not independent, so their standard deviation divided by $\sqrt{k}$ is not automatically a valid standard error.

Definition Data leakage occurs when information that would not be available at the time and place of a real prediction influences the fitting, preprocessing or selection of a model, or the cases used to evaluate it. Common routes are predictors recorded after the outcome or derived from it, duplicate or linked records on both sides of a split, information from the future in time-ordered data, preprocessing such as imputation, scaling or variable selection learned from validation or test cases, and repeated changes to the model after looking at test results. Leakage typically makes the reported error smaller than the error the procedure will achieve in use. Every quantity learned from data belongs inside the training portion of each split or fold, and it is then applied unchanged to the held-out cases.

Definition Distribution shift occurs when the joint distribution of predictors and outcome in the setting where a model is used differs from the distribution that produced its training and test data. The predictor distribution can change, the frequency of the outcome can change, or the relationship between the outcome and the predictors can change. A test score is evidence about the population, period and measurement process that the test cases represent, and under a shift it need not carry over. Predicting at predictor values outside the range of the training and test data is extrapolation, which no held-out score from inside that range checks.