A colleague standardizes every column of a table and then runs cross-validation on the standardized data. The scaler’s means and spreads were computed with the very rows each fold later holds out, so every fold score grades a procedure that has already seen part of its exam. This lesson defines k-fold cross-validation as an evaluation of a whole training procedure, preprocessing included, shows how to choose ordinary, stratified, grouped or temporal folds to match the data and the intended use, and puts every learned step inside a pipeline that is refitted in each fold.
Key terms
K-fold cross-validation
Definition Let the development data be the rows that remain after a test set has been set aside, and fix a training procedure, meaning every learned preprocessing step together with the model and its settings. Choose an integer with and partition the rows into disjoint nonempty folds . For each , fit the whole procedure on the training rows , all rows outside , and use the fitted result to predict the rows of . If is the score on fold , the cross-validation estimate is the mean of the fold scores,
When the score is the average over the fold’s rows of a per-row quantity, such as squared error or correct classification, the size-weighted mean with weights equals the average over all held-out predictions, and it agrees with the plain mean when the folds have equal sizes. For a metric that is not such an average, such as F1 or ROC AUC, the mean of fold scores and the metric computed once on the pooled held-out predictions can differ, and the pooled version mixes scores from different fitted models. Every row is predicted exactly once, by a model fitted without it. The estimate describes the procedure trained on about of the development rows, not one fitted model, and after a procedure is chosen it is refitted on all development rows. The training sets overlap, so the fold scores are dependent, and their standard deviation divided by is not automatically a valid standard error. Cross-validation compares and chooses procedures within the development data, and the test set stays outside it.
In plain words Every row takes one turn as unseen data, and the procedure is rebuilt from scratch for each turn, so no row ever grades a model that learned from it. The rotation uses all development rows for checking, but the result describes the procedure rebuilt on somewhat smaller training sets, not any single model.
Example Six constructed rows have inputs and labels . Three folds hold out the inputs , and , so each training set has rows, two thirds of the data. Two procedures run on these same folds. One standardizes the input and then classifies by the label of the single nearest training input. The other standardizes and then fits a logistic regression with an L2 penalty, with the details under preprocessing pipeline below. Both classify all six held-out rows correctly, so both have cross-validated accuracy . Six rows of well-separated constructed data show the mechanics only. They cannot show that the two procedures will do equally well on new data.
Stratified folds
Definition For a classification target, stratified folds are folds in which each class makes up about the same share as in the whole development set, as closely as whole rows allow. A common construction splits the rows of each class into nearly equal parts and joins one part of every class into each fold. Stratification reduces the chance variation of class counts between folds, and when a class has at least rows it keeps that class from vanishing from a validation fold, where a metric such as recall could be undefined. For a numerical target the same idea can be applied to bins of the target. Stratification controls only the label proportions. It does not keep linked rows together, it does not respect time order, and it does not make dependent rows independent.
In plain words Deal the rows of each class around the folds like cards, so that every fold has roughly the class mix of the whole data set.
Example In the six-row example each fold holds one row of each class, the overall proportion . For a larger case, take rows with positives and five folds of . If the folds are a random partition with all partitions equally likely, a given fold contains no positive with probability
and inclusion and exclusion over the five folds shows that at least one fold has no positive with probability about . Stratified folds put exactly positives in every fold.
Grouped folds
Definition Suppose each row belongs to a group, such as a customer, a patient, a machine or a measurement session, and the model will be used for groups it has not seen. Grouped folds assign whole groups to folds, so all rows of a group sit in the same fold and no group appears among both the training rows and the held-out rows of any split. Rows of one group are often more alike than rows of different groups, so a held-out row whose group also appears in training is usually easier to predict than a row from a new group, and ordinary row folds then tend to overstate performance for new groups. The number of groups, more than the number of rows, limits how much independent evidence the folds hold, and fold sizes can differ when groups differ in size. If the model will instead predict further rows of known groups, the folds should copy that use. Which units form the groups follows from how the model will be used and is settled before any scores are seen.
In plain words If the model will meet strangers, every person’s rows go into one fold together. Otherwise the evaluation quietly lets the model recognize people it has already met.
Example A table has customers with orders each, rows in all, split into ordinary random folds of rows with all partitions equally likely. For a held-out order, the probability that the customer’s other orders all land in the same fold is
about , so with probability about at least one other order of the same customer is among the training rows. With grouped folds of customers each, that probability is .
Temporal folds
Definition Temporal folds evaluate a procedure the way it will be used, forward in time. The rows are ordered by prediction time, and each split trains on rows before a cutoff and validates on a later block, so every training row precedes every validation row. In an expanding window each later split also trains on the earlier validation blocks, and in a sliding window the training period keeps a fixed length. Every feature may use only information available at its own row’s prediction time, and a training label must already have been final when the model would have been trained. When targets or features are built over windows of time, a gap between the last training rows and the validation block keeps the same raw events from feeding both sides. Shuffled or stratified folds put later rows into the training data for earlier rows, and stratifying the labels does not repair that. Temporal fold scores describe how the procedure would have done at past cutoffs, and conditions can still change after the last one.
In plain words Train on the past and validate on the future, one cutoff at a time, using only what was known before each cutoff.
Example With months of data and an expanding window, three splits can train on months to and validate on months and , train on months to and validate on months and , and train on months to and validate on months and . Suppose the target counts a customer’s purchases in the days after the prediction time. A training row dated in the last days before a cutoff then has a label that depends on purchases after the cutoff, inside the validation period, and that label was not yet final at the cutoff. Leaving out training rows from those days, a gap before each validation block, removes the overlap.
Preprocessing pipeline
Definition A preprocessing pipeline is an ordered chain of steps, such as an imputer, a scaler, an encoder or a feature selector, ending in an estimator. Each preprocessing step has a fit operation, which learns values from the rows it receives, and an apply operation, which maps rows using the stored values. Fitting the pipeline on rows fits the first step on , applies it to , fits the second step on that output, and so on, and finally fits the estimator. Predicting for new rows applies the stored steps in order and then the fitted estimator, without refitting anything. Cross-validating a pipeline fits a fresh copy of the whole chain on each set of training rows , so every learned value comes from alone. A step that learns nothing from data, such as the logarithm of a positive feature, is harmless outside the folds, and keeping it inside the pipeline costs nothing.
In plain words A pipeline treats preprocessing as part of the model. Whatever a step learns from data it learns from the training rows of the current fold, and the held-out rows are then processed the way new data will be.
Example In the first fold of the six-row example the training inputs are . Their mean is , the squared deviations add to , and dividing by the rows gives the variance . The scaler stores the mean and the standard deviation , and it maps the held-out inputs and to about and . The second and third folds store the means and with variances and . In each fold the logistic regression minimizes the mean log loss on the scaled training inputs plus , with the intercept unpenalized. The fitted slopes are about , and , and the fitted intercept is in every fold, because each scaled training input with label is matched by its negative with label . The held-out probabilities of class are about and in the first fold, and in the second and and in the third, so the threshold classifies all six rows correctly. In scikit-learn the comparison can be written as follows.
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
X = np.array([[0], [1], [2], [8], [9], [10]])
y = np.array([0, 0, 0, 1, 1, 1])
# (training rows, held-out rows) for each fold
folds = [
([1, 2, 4, 5], [0, 3]),
([0, 2, 3, 5], [1, 4]),
([0, 1, 3, 4], [2, 5]),
]
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=1))
# C = 1 / (4 rows * 0.1) gives the penalty 0.1 * w**2 / 2
# on the mean log loss; the intercept is not penalized
logit = make_pipeline(StandardScaler(), LogisticRegression(C=2.5))
for model in (knn, logit):
print(cross_val_score(model, X, y, cv=folds).mean())Fold leakage
Definition Fold leakage occurs when information from a fold’s held-out rows reaches the fitting of the model that is scored on those rows. Common routes are a preprocessing step fitted once on all development rows before the folds are formed, such as imputation values, scaling constants, category encodings, selected features or resampled rows, rows of one group or of a later period on both sides of a split, and hyperparameters chosen with the same folds whose scores are then reported. Under fold leakage the cross-validation score no longer evaluates the procedure that will run on new data, and it is usually optimistic. The size of the effect ranges from nothing to very large, depending on the step, the model and the data, so a small observed change does not excuse a leak. The repair is to place every learned step inside the pipeline, to choose folds that match the data, and to keep any tuning inside the training rows, as nested evaluation does.
In plain words Each fold is a rehearsal of meeting new data. Anything learned from the held-out rows before they are scored, even a mean or a list of chosen columns, spoils the rehearsal.
Example If the six inputs are standardized once with all rows, the scaler uses the mean and the standard deviation , so in the first fold the input becomes about instead of , because the held-out inputs and helped set the scale. With a single feature this rescaling keeps the order of the inputs, so the nearest neighbors and the accuracies here do not change. That is a property of this example and not a reason to accept the leak. A sharper case is an identifier column in which every value occurs in exactly one row, encoded by the mean label of the rows that share the value. Computed on all rows, every row’s code equals its own label, so a threshold of classifies every held-out row correctly whatever the labels are. Fitted inside each fold, every held-out identifier is unseen, receives the same fallback value, and carries no information.
Check yourself
- In the second fold of the six-row example the training inputs are with labels , and the held-out inputs are and . Find the stored mean and standard deviation, dividing by the number of training rows, the scaled held-out inputs, and the labels predicted by the one-nearest-neighbor model.
Show the answer
The mean is . The deviations have squares that add to , so the variance is and the standard deviation is .
The scaled training inputs are about , , and . The held-out input maps to and the input maps to .
The scaled input lies about from both and , so the inputs and tie as nearest neighbors. Both have label , so the prediction is under any tie rule, which is correct. In the same way ties between the inputs and , both with label , and the prediction is correct. Standardizing a single feature shifts it and multiplies it by a positive number, which keeps every distance comparison, so these are the same ties as on the original scale.
- A colleague standardizes all development rows and then runs six-fold cross-validation on the standardized table, with rows in each validation fold. Which rows influenced the scaling used in each fold, and how should the workflow be repaired? How many rows do two different training sets share, and what does that mean for the six fold scores?
Show the answer
The global mean and standard deviation came from all rows, so in every fold the rows being scored helped set the scale applied to them. That is fold leakage.
The repair fits a fresh scaler in each fold on that fold’s training rows only and applies the stored mean and standard deviation to the held-out rows. The same holds for every other learned step, such as imputation, encoding or feature selection, which is easiest to guarantee by cross-validating a pipeline that contains them all.
Two different training sets share rows, and every row belongs to the training sets of five of the six folds. The fitted models are therefore similar and their scores are correlated, so the six scores are not six independent experiments. Report the fold design, the mean score and the spread between folds as a description, not as a confidence interval. After the procedure is chosen, refit it on all development rows and evaluate it once on the untouched test set.
- A colleague proposes stratified five-fold cross-validation for each of three projects. In the first, loan applications from different applicants, of which percent defaulted, can be treated as independent draws from the population the model will serve. In the second, clinic visits come from patients and the model will be used for new patients. In the third, three years of daily sales are used to forecast next week’s sales. Is the proposal right for each, and what should replace it where it is not?
Show the answer
For the loan applications it fits. With rows per fold and defaults in all, stratified folds hold exactly defaults each, so no fold is left with too few defaults to score recall or precision.
For the clinic visits it is not enough. Stratification balances the labels but leaves visits of one patient spread over several folds, so the model is scored on patients it has partly seen. Use grouped folds that keep each patient’s visits together. If the classes are unbalanced, the folds can also be balanced as far as whole patients allow, but the grouping comes first.
For the sales forecast it is wrong. Stratified or shuffled folds would train on later days to predict earlier ones. Use temporal folds that train on the days before each cutoff and validate on the following week, build every lagged feature only from sales before the prediction time, and leave a gap whenever a feature or target window would otherwise cross the cutoff. In all three cases the prediction time and the unit of future use are defined first, and the folds are chosen to copy them.