Training, Validation, and Test Sets

Definition The training set is the part of the labeled data that the learning algorithm reads to set the model parameters. For a model $f$ and a loss $L$, the training error is the average loss over the $n$ training examples,

also called the empirical risk on the training set. Every quantity a procedure learns from data, including preprocessing constants such as a mean used to fill missing values or a scale used to standardize a feature, is computed from the training examples only and then applied unchanged to all other data. Because the parameters were chosen to make the training error small, the training error of a fitted model tends to understate its error on new cases, and on its own it cannot rank models of different flexibility.

Definition The validation set is labeled data held out from fitting and used to compare candidate procedures, such as different model families, feature sets or hyperparameter values, and to make other development decisions. Each candidate is fitted on the training set and scored on the validation set with a metric chosen in advance, and the candidate with the best validation score is selected. Because the validation data take part in that choice, the winning validation score tends to understate the selected model’s error on new cases, more so when many candidates are compared. It is therefore not the final performance estimate. When data are scarce, cross-validation rotates the validation role through several splits of the development data, which is everything except the test set.

Definition The test set is labeled data set aside before any modeling decision and used once, after the whole procedure is fixed, to estimate how the final model performs. Suppose the final model $\hat{f}$ was fixed without using the test cases and the $m$ test cases $(\mathbf{x}_j, y_j)$ are independent draws from the deployment population. Then the test score

is an unbiased estimate of the population risk of $\hat{f}$, its expected loss on a new case from that population, and with few test cases it can still be far from it. If test results influence any choice, such as revising features after a disappointing score or reporting the best of several models by test score, the test set has served as validation data and its score is no longer unbiased. The smallest of several test errors is on average no larger than the population risk of the model it picks, and usually smaller. A planned refit of the chosen procedure on training and validation data together, done before the test set is opened, keeps the test set’s role. A later revision of the model needs evidence that has not been used yet.

Definition A model parameter is a quantity inside a model whose value the training algorithm sets from the training data, such as a coefficient or intercept of a linear model, or the single value a constant baseline predicts. The value training produces is an estimate, written with a hat as in $\hat{b}$, and a different training sample would usually give a different estimate. A hyperparameter is a setting fixed before the training algorithm runs that selects the model family or controls how training works, such as the number of neighbors $k$ in k-nearest neighbors, a penalty weight $\lambda \ge 0$, a polynomial degree or a step size. Parameters are fitted on the training set. Hyperparameters are chosen by comparing the validation scores of candidates that were each fitted on the training set. Choosing them by training error tends to favor the most flexible setting. For example, with $k = 1$ every training point is its own nearest neighbor, so when no two training points share a feature vector the training error is zero.

Definition A constant baseline predicts the same value $c$ for every case, with $c$ computed from the training targets alone. The best constant depends on the loss. Under squared error the training mean $\bar{y}$ is the unique best constant, because for every $c$

where the cross term has dropped out since $\sum_{i} (\bar{y} - y_i) = 0$, and the last term is positive unless $c = \bar{y}$. Under absolute error the best constants are the medians of the training targets, the middle value when $n$ is odd and every value between the two middle values, inclusive, when $n$ is even. For classification scored by accuracy, the constant with the highest training accuracy is the most frequent training class, and a tie between classes needs a rule stated in advance. A learned model adds something only if it beats an appropriate baseline on held-out data under the same metric, and the baseline’s constant is never recomputed from validation or test outcomes.

Definition The deployment population is the set of cases on which a model’s predictions will actually be used, described by which units they concern, where and when they arise, and how their features are measured at prediction time. Write $(\mathbf{X}, Y)$ for a new case drawn from it. The population risk of a fixed model $f$ is its expected loss on such a case,

Generalization is the performance of a fitted model $\hat{f}$ on new cases from the deployment population that played no part in fitting or selection, measured by $R(\hat{f})$ with $\hat{f}$ held fixed. The training error is an average over the training data and is a different quantity from this population risk. A test score estimates $R(\hat{f})$ only for the population the test cases represent, so the split has to copy the intended use, as data leakage and honest splitting works out. When the deployment population changes, an old score need not carry over.