Multiple testing and backtest overfitting

Definition A multiple testing problem has null hypotheses $H_1, \dots, H_m$ about one data set, p-values $p_1, \dots, p_m$, and a procedure that decides from the p-values which hypotheses to reject. Let $\mathcal{H}_0$ be the unknown set of indices of the true nulls, $m_0 = \lvert \mathcal{H}_0 \rvert$, $R$ the number of rejections and $V$ the number of rejected true nulls. The p-value of a true null is valid, or super-uniform, if

for every $u \in [0, 1]$, where $\mathbb{P}$ is the law that generates the data. An exact p-value, such as one computed from a continuous test statistic with a correctly specified null distribution, satisfies this with equality and is uniform on $[0, 1]$. If each hypothesis is rejected when $p_j \le \alpha$, then $\mathbb{E}[V] \le m_0\,\alpha$ under any dependence, by linearity of expectation. If all $m$ nulls are true and the p-values are independent and exact, then

because all $m$ p-values exceed $\alpha$ with probability $(1 - \alpha)^m$, a product that needs independence.

Definition Selection bias is the error made when a statistic is reported because it was the best of several and is then judged as if it had been chosen in advance. If the null scores $Z_1, \dots, Z_m$ are independent with a common distribution function $F$, then

since the maximum is at most $z$ exactly when every score is. Under these assumptions and for a continuous $F$, a winner whose single-test p-value is $u = 1 - F(z)$ has the selection-adjusted p-value $1 - (1 - u)^m$. If each score has distribution function $F$ but the dependence between them is arbitrary, only the bounds

hold in general, the lower one because the maximum is at least $Z_1$ and the upper one by the union bound. The count $m$ in these formulas is the number of candidates whose results could have influenced the choice, not the number shown. Trial accounting is the record of every configuration evaluated, including changed lookbacks, universes and entry thresholds and the variants that were discarded, together with the data and the score used for each, so that the selection experiment can be stated. When attempts are missing from the record, an adjustment that uses the recorded count corrects for a smaller search than the one that took place.

Definition The family-wise error rate of a procedure is $\operatorname{FWER} = \mathbb{P}(V \ge 1)$, the probability of at least one false rejection, computed under the true configuration of nulls and alternatives. The Bonferroni correction rejects $H_j$ when $p_j \le \alpha/m$. If every true-null p-value is valid, then whatever the dependence between the p-values and whichever hypotheses are true, the union bound gives

The bound is attained when $m_0 = m$, the p-values are exact and the rejection events are disjoint. Under strong positive dependence it is conservative, and if all the true-null p-values equal one exact p-value, the family-wise error rate is only $\alpha/m$.

Definition The false discovery proportion is $V/\max(R, 1)$, the fraction of rejections that are false, which is $0$ when nothing is rejected. The false discovery rate is its expectation,

Since $V \le R$, the proportion is at most the indicator $\mathbf{1}_{\{V \ge 1\}}$, so $\operatorname{FDR} \le \operatorname{FWER}$ for every procedure. Under the global null, $m_0 = m$, every rejection is false, the proportion equals $\mathbf{1}_{\{R \ge 1\}}$, and the two rates coincide. When some alternatives are true, controlling the false discovery rate at a level $q$ bounds the expected fraction of false rejections among the rejections. That is a weaker requirement than control of the family-wise error rate, and it allows several false rejections among many rejections. The false discovery rate is the expectation of a ratio, not the ratio $\mathbb{E}[V]/\mathbb{E}[R]$ of expectations.

Definition Fix a level $q \in (0, 1)$ and order the p-values as $p_{(1)} \le \dots \le p_{(m)}$. Let $k^*$ be the largest $k$ in $\{1, \dots, m\}$ with

and $k^* = 0$ if there is none. The procedure rejects the hypotheses with the $k^*$ smallest p-values. It is a step-up rule, which looks for the largest qualifying position, so a failed comparison at a smaller position does not end the search. Equivalently, $R = k^*$ and the rejected set is $\{j : p_j \le qR/m\}$, because a p-value at a position beyond $k^*$ that is at most $k^* q/m$ would make that position qualify as well.

The theorem is that if the p-values are mutually independent and every true-null p-value is valid, then

For the proof, fix a true null $i$ and let $R_i^0$ be the number of rejections when $p_i$ is replaced by $0$ and the other p-values are kept. Then $R_i^0 \ge 1$, because the zero p-value makes position $1$ qualify. The count $R_i^0$ depends only on $(p_j)_{j \ne i}$, and hypothesis $i$ is rejected exactly when $p_i \le qR_i^0/m$, in which case $R = R_i^0$. To see this, write $N(c)$ for the number of p-values at most $c$, so that a position $k$ qualifies exactly when $N(kq/m) \ge k$. Lowering $p_i$ can only raise $N$, so $R \le R_i^0$. If $p_i \le qR_i^0/m$, which holds in particular when $i$ is rejected, the original and modified counts agree at $c = qR_i^0/m$, so the position $R_i^0$ also qualifies for the original p-values. Then $R = R_i^0$ and $p_i \le qR/m$, so $i$ is rejected. Hence

Conditioning on the other p-values fixes $R_i^0$, and by independence $p_i$ stays valid, so the conditional expectation is at most $(qR_i^0/m)/R_i^0 = q/m$. Summing over the $m_0$ true nulls, whose rejection indicators add up to $V$, gives $\operatorname{FDR} \le m_0 q/m$. With exact true-null p-values each step is an equality and $\operatorname{FDR} = m_0 q/m$. Other dependence structures need their own conditions, such as a form of positive dependence, or a stricter threshold, and those results are not derived here.

Definition Backtest overfitting is present when a configuration is chosen from a candidate family because of its backtest on a sample and its score on that sample overstates its performance on data that played no part in the choice, because the choice rewarded noise along with any real effect. A diagnostic studies the selection step itself. Combinatorially symmetric cross-validation takes $N$ candidates whose performance series are cut into $S$ blocks of equal length, $S$ even. For each of the $\binom{S}{S/2}$ ways of choosing half the blocks as the selection sample, it picks the candidate with the best score on that half and records the relative rank $\omega = r/(N + 1)$ of that candidate’s score on the other half, where $r = 1$ is the worst of the $N$ and $r = N$ the best. The fraction of splits with $\omega \le \tfrac12$, the splits in which the selected candidate lands in the bottom half out of sample, is reported as an estimate of the probability of backtest overfitting. That number is conditional on the candidate family, the score, the data and the partition design. The splits also select on later blocks and evaluate on earlier ones, so the diagnostic does not follow the time order of a forecast. The number is not a probability that a future trade or strategy will lose, and trials missing from the family limit what it can say.