Time series validation and overlapping labels

Definition Time is counted in integer observation intervals, and increment $k$ is the change observed at time $k$, for example the log return $r_k$ from time $k - 1$ to time $k$. A label with horizon $h \ge 1$ that starts at time $t$ is a function of the increments in its information interval

for example the $h$-period log return $y_t = r_{t+1} + \dots + r_{t+h}$, and the features $x_t$ of the row use only information available at time $t$. The label is complete at time $t + h$. Take the convention that a decision at time $T$ is made after increment $T$ has been observed. A chronological split at decision time $T$ trains only on rows whose labels are complete at $T$ and evaluates only forecasts made at $T$ or later,

Starts with $T - h < t < T$ belong to neither set at that decision, because their features are known but their labels are not. If the decision is made before increment $T$ is observed, the training condition becomes $t + h \le T - 1$. A random split assigns rows to training and validation without regard to these times, so in general it trains on outcomes that were not known when the validation forecasts were made.

Definition Walk-forward validation repeats the chronological split at increasing decision times $T_1 < T_2 < \dots < T_J$. At decision $T_j$ the model, together with every preprocessing step that learns from data, is fitted on rows with $t + h \le T_j$, either all of them (an expanding window) or only the latest $W$ of them (a rolling window). The fitted predictor $\hat f_j$ then forecasts the starts of the block $B_j = \{T_j, \dots, T_{j+1} - 1\}$, where $T_{J+1}$ marks the end of the sample, and each forecast is scored with a loss $\ell$ once its label is complete. With $n$ forecasts in all, the walk-forward score is

Every forecast uses only information available when it is made, because its training labels end by $T_j \le t$ and its features are known at $t$. The score is therefore the historical loss of the procedure run the way it would have been run. It estimates future loss only under further assumptions on the stability and dependence of the losses, which the chronological order does not supply.

Definition Two labels overlap when their information intervals share an increment, $\mathcal{I}_s \cap \mathcal{I}_t \ne \varnothing$. For two labels with the same horizon $h$ this happens exactly when $\lvert t - s \rvert < h$, and they then share $h - \lvert t - s \rvert$ increments. Suppose in addition that each label is the sum of its increments and that the increments are uncorrelated with a common variance $\sigma^2$. Then

so the correlation of the two labels is $(1 - \lvert t - s \rvert / h)^{+}$. For the mean $\bar y$ of the $n$ consecutive labels starting at $1, \dots, n$,

and $n \operatorname{Var}(\bar y)$ tends to $\sigma^2 h^2$ as $n$ grows, because the weights $h - \lvert k \rvert$ over the lags $\lvert k \rvert < h$ add up to $h^2$. The formula $\operatorname{Var}(y_t)/n = \sigma^2 h/n$ for independent observations therefore understates this variance by a factor that approaches $h$.

Definition Let $S$ be the set of validation starts and

the union of their information intervals. Purging removes from the training candidates every start $t \notin S$ whose interval meets $V$, so that every retained training label satisfies $\mathcal{I}_t \cap V = \varnothing$. For a contiguous validation block $S = \{a, \dots, b\}$ and a common horizon $h$, the union is $V = \{a + 1, \dots, b + h\}$, and a start outside the block is retained exactly when $t + h \le a$ or $t \ge b + h$. This removes the $h - 1$ starts $a - h + 1, \dots, a - 1$ before the block and the $h - 1$ starts $b + 1, \dots, b + h - 1$ after it, whenever they exist. In continuous event time the same test compares intervals such as $(t, t + h]$, and the endpoint convention must be stated, since an observation exactly on a boundary may or may not be shared. Purging removes only the specified overlap of label intervals. It does not make the retained rows independent of the validation rows, which can still depend on them through persistent volatility, slowly moving features or a common market regime.

Definition An embargo is an additional exclusion of training starts in a stated window after a validation block, applied on top of purging. Its width follows from the design, for example from the feature windows, and no single width removes all dependence. Suppose the features of start $t$ use the $L \ge 1$ most recent increments $t - L + 1, \dots, t$. Among the starts after the block that survive purging, those with $t \ge b + h$, the feature window meets $V = \{a + 1, \dots, b + h\}$ exactly when $t - L + 1 \le b + h$. Excluding them removes the $L$ starts $b + h, \dots, b + h + L - 1$, whose inputs were built partly from the validation outcomes. An embargo matters only in blocked designs that train on rows after the validation block. Walk-forward validation never does.

Definition Nested chronological validation uses two layers of splits. Outer fold $j = 1, \dots, J$ has an outer training window $D_j$, whose labels are complete at the outer decision $T_j$, and a later outer block $B_j$. Each candidate tuning parameter $\lambda$, for example a ridge penalty, is scored by chronological, purged inner splits that use only rows of $D_j$, and the inner winner is

The procedure is refitted on $D_j$ with $\hat\lambda_j$ and scored once on $B_j$, and the outer estimate is

Every step that learns from data, including imputation, normalization, feature selection and probability calibration, is fitted inside the same windows. The outer estimate scores the whole procedure, tuning included, rather than one fixed model. The minimized inner score is not an assessment of the selected model, because it took part in the choice. Equal fold weights $1/J$ and weights proportional to the number of predictions in each block give different averages when the blocks differ in size, so a report states which expected loss it estimates. If several procedures are then compared by their outer scores and the best is kept, those scores have become selection information as well, and assessing the kept procedure takes a later block that played no part in any choice.