Risk backtesting and statistical tests

Definition Let $L_t$ be the loss of a fixed portfolio over period $t$, positive for an adverse outcome, so $L_t = -\Delta V_t$ when $\Delta V_t$ is the PnL of the period. Let $\mathcal{F}_{t-1}$ be the information available when the forecast for period $t$ is issued, and let

be the conditional distribution function of the loss under the physical measure $\mathbb{P}$, the right measure here because forecasts are compared with realized losses. A VaR forecast at confidence level $\alpha \in (0, 1)$ is an $\mathcal{F}_{t-1}$-measurable number $\operatorname{VaR}_{\alpha,t}$ that is meant to equal the lower conditional quantile

A VaR backtest uses the forecasts and the realized losses of periods $t = 1, \dots, n$ to test a stated hypothesis about the exceptions, the periods in which the loss exceeds its forecast. The hypothesis usually tested is correct conditional coverage at a stated rate $p_0$,

where $p_0 = 1 - \alpha$ for forecasts that equal the conditional quantile of continuous conditional loss laws. The loss must be measured over the horizon of the forecast and on the portfolio it was made for, and the forecast must be the one issued at time $t - 1$, not a version recomputed later with data that include $L_t$. A rejection is evidence against the tested hypothesis. A nonrejection in a short sample does not show that the forecasts are correct.

Definition The exception indicator of period $t$ is

which is $1$ when the realized loss strictly exceeds the forecast and $0$ otherwise. If the forecast equals the lower conditional quantile, then $F_t(\operatorname{VaR}_{\alpha,t}) \ge \alpha$ by the right continuity of $F_t$, so

with equality exactly when $F_t(\operatorname{VaR}_{\alpha,t}) = \alpha$, in particular whenever $F_t$ is continuous. An atom of the loss law at the quantile makes the exception probability smaller than $1 - \alpha$, and a test must then use the actual probability. Now suppose that $\mathbb{P}(I_t = 1 \mid \mathcal{F}_{t-1}) = p_0$ for every $t$, and that every earlier indicator $I_s$ with $s < t$ is known when the forecast for period $t$ is issued, that is, $\mathcal{F}_{t-1}$-measurable. Fix $i_1, \dots, i_n \in \{0, 1\}$ and write $A_j$ for the event $\{I_1 = i_1, \dots, I_j = i_j\}$. The event $A_{n-1}$ belongs to $\mathcal{F}_{n-1}$, so conditioning on $\mathcal{F}_{n-1}$ gives

and induction on $n$ turns the right side into a product of $n$ Bernoulli probabilities. So correct conditional coverage makes the indicators independent Bernoulli variables with parameter $p_0$, which is the null hypothesis of the tests below. The overlapping windows of the previous card break the measurability condition, which is why their exceptions can be dependent.

Definition The exception count over $n$ periods is $K = \sum_{t=1}^{n} I_t$. If $I_1, \dots, I_n$ are independent Bernoulli variables with parameter $p_0$, then $K$ has the distribution $\operatorname{Binomial}(n, p_0)$,

for $j = 0, 1, \dots, n$, with $\mathbb{E}[K] = n p_0$ and $\operatorname{Var}(K) = n p_0 (1 - p_0)$. The exact upper tail

measures how surprising $k$ or more exceptions are under this null. A one-sided exact test at a significance level $a$ for too many exceptions rejects when $\mathbb{P}(K \ge k) \le a$ for the observed count $k$. Because $K$ is discrete, the actual size of this test is the upper tail at the smallest rejected count, which is at most $a$ and usually below it. Without independence the count need not be binomial, since in general

and positively correlated exceptions spread the count more widely than the binomial law does. The count also discards the order of the exceptions, so a count near $n p_0$ says nothing about their timing, and a surprising count does not identify the cause of the miscalibration.

Definition The unconditional coverage test compares the null hypothesis that $I_1, \dots, I_n$ are independent Bernoulli variables with the stated parameter $p_0$ against the alternative that they are independent Bernoulli variables with an unrestricted parameter $p \in [0, 1]$. For an observed sequence with $k$ exceptions, the likelihood and the log-likelihood, with log the natural logarithm here and below, are

For $0 < k < n$ the derivative

vanishes only at $\hat p = k/n$, and $\ell''(p) = -k/p^2 - (n - k)/(1 - p)^2 < 0$, so $\hat p$ is the unique maximizer. For $k = 0$ or $k = n$ the likelihood is monotone in $p$ and is again largest at $\hat p = k/n$. The likelihood ratio statistic is

where a term of the form $0 \log 0$ at $k = 0$ or $k = n$ is read as its limit $0$. The binomial coefficient is the same under both hypotheses and cancels. The statistic is nonnegative, and it is $0$ only when $k = n p_0$. As a standard asymptotic result, under the null with a fixed $p_0 \in (0, 1)$ the statistic converges in distribution to $\chi^2_1$ as $n \to \infty$, so rejecting when $\operatorname{LR}_{\text{uc}} > 3.8415$, the $0.95$ quantile of $\chi^2_1$, gives a test whose size tends to $0.05$. The test is two-sided, so it rejects too few exceptions as well as too many. With a small expected count $n p_0$, or with $\hat p$ on the boundary, exact binomial probabilities are more reliable than the $\chi^2_1$ reference.

Definition For $t = 2, \dots, n$, the transition count $n_{ij}$ is the number of periods with $I_{t-1} = i$ and $I_t = j$, for $i, j \in \{0, 1\}$, so that $n_{00} + n_{01} + n_{10} + n_{11} = n - 1$. The first-order Markov alternative lets the exception probability depend on the previous indicator through $p_{01} = \mathbb{P}(I_t = 1 \mid I_{t-1} = 0)$ and $p_{11} = \mathbb{P}(I_t = 1 \mid I_{t-1} = 1)$. Conditional on the first indicator $I_1$, the likelihood of the observed sequence is

and its two factors are maximized separately at

Independence means $p_{01} = p_{11} = \pi$. With $m = n - 1$ and $k' = n_{01} + n_{11}$ the likelihood becomes $\pi^{k'}(1 - \pi)^{m - k'}$, which is maximized at $\hat\pi = k'/m$. Writing $\hat\ell_M$ and $\hat\ell_{\text{iid}}$ for the two maximized log-likelihoods, with $0 \log 0 = 0$, the independence statistic is

If no transition starts from one of the two states, for example when no exception occurs before period $n$, that factor of the likelihood is constant, its estimate is undefined, and the statistic is $0$. As a standard asymptotic result, under independent Bernoulli indicators with a fixed parameter in $(0, 1)$, the statistic converges in distribution to $\chi^2_1$ as $n \to \infty$. In a finite sample this reference needs all four transition counts to be reasonably large. The alternative is first-order dependence only, so the test cannot be expected to detect dependence that leaves the two adjacent transition probabilities equal.

Definition The conditional coverage test takes as its null hypothesis $p_{01} = p_{11} = p_0$, which is independence together with the stated rate, against the first-order Markov alternative. With the same conditioning on $I_1$, the null likelihood of the $m = n - 1$ transitions is

and the statistic is $\operatorname{LR}_{\text{cc}} = 2(\hat\ell_M - \ell_0)$ with $\ell_0 = \log\mathcal{L}_0$. Inserting the maximized independent log-likelihood between the two terms gives the exact identity

where $\operatorname{LR}_{\text{uc}}^{(m)}$ is the unconditional coverage statistic computed from the $m$ indicators $I_2, \dots, I_n$, which contain $k'$ exceptions. The coverage statistic computed from all $n$ indicators uses different data and in general does not satisfy this identity. As a standard asymptotic result, under the null the statistic converges in distribution to $\chi^2_2$, whose $0.95$ quantile is $5.9915$, and the same caution about sparse transition counts applies. A first-order Markov alternative is also narrower than the full hypothesis of correct coverage given all of $\mathcal{F}_{t-1}$, so a nonrejection does not confirm that hypothesis.