Model validation reproducibility and monitoring

Definition A financial model is the whole specification behind a number, namely the contract and its payoff, the probability measure and information set, the data and conventions, the calibration, the numerical approximation, the implementation and the intended use. Model validation assesses whether that specification is fit for the intended use, and it asks different questions. Conceptual soundness asks whether the defined mathematical object is the right one for the contract and the decision. Numerical verification asks whether the implementation computes that object to a stated accuracy, for example through convergence tests against independent benchmarks and through invariants. Empirical validation asks whether the inputs and assumptions are supported by evidence for the use, and monitoring asks whether that evidence still applies after deployment. Passing one of these checks answers none of the others. An analytic benchmark can confirm that code solves its equation without showing that the equation describes the market, and an exact solution for a different contract is the wrong value for this one. Supervisory model-risk guidance for banks frames expectations of this kind for the institutions within its scope. Such guidance has a stated scope and is not a theorem about any calculation, and the checks on this page certify neither a real model nor regulatory compliance.

Definition A financial invariant is a relation that holds exactly for every input in a stated class whenever stated hypotheses hold, as a consequence of the model’s definitions or of the law of one price. Examples are state probabilities that are nonnegative and sum to $1$, a covariance matrix $\Sigma$ with $w^{\top}\Sigma w \ge 0$ for every weight vector $w$, cash flows that are conserved when they are split and recombined, and put-call parity. Take a European call and a European put on the same stock with no dividends before $T$, with the same strike $K$, maturity $T$ and settlement, in a frictionless market with deterministic rates and discount factor $D(0,T)$. A long call with a short put pays $S_T - K$ at $T$, and so does one share held with a short position in $K$ zero-coupon bonds paying $1$ at $T$, so the law of one price gives

With stochastic rates the same argument holds with the bond price $P(0,T)$ in place of $D(0,T)$. An invariant test computes the residual

and flags $\lvert \epsilon \rvert$ above a tolerance chosen for the units and the use. A failure means that the calculation is wrong or that the hypotheses do not match the instrument, for example because of dividends, early exercise, settlement differences or transaction costs. A pass is necessary and not sufficient, because an error that adds the same amount to both option prices leaves parity intact, and every model that prices both options as discounted expectations under a pricing measure consistent with the stock and bond prices satisfies it, whatever volatility it uses.

Definition Let $A_h$ approximate an exact value $A$, where $h > 0$ is a resolution parameter such as a step size, and suppose

as $h \to 0$, with $c \ne 0$ and $p > 0$. Then

so the base-2 logarithm of the ratio estimates the order $p$. Without the exact value, the ratio of successive differences $(A_h - A_{h/2})/(A_{h/2} - A_{h/4})$ has the same limit, and the Richardson combination $(2^p A_{h/2} - A_h)/(2^p - 1)$ equals $A + o(h^p)$. A convergence test computes such ratios over a refinement sequence and compares them with the order the method should have. The limit describes the asymptotic regime, and coarse resolutions need not show it yet. Agreement between two implementations that share an assumption or a boundary treatment is weaker evidence than convergence to an independent benchmark. For the composite trapezoidal rule $T_n$ with $n$ intervals of width $h = (b - a)/n$ and an integrand $f$ with a continuous second derivative on $[a, b]$,

so $p = 2$ whenever $f'(b) \ne f'(a)$. When $f'(b) = f'(a)$ the leading term vanishes and the observed order can be higher. Monte Carlo error is a separate component. For $n$ independent draws with finite variance, the standard error of their mean is the standard deviation of one draw divided by $\sqrt{n}$, estimated by the sample standard deviation over $\sqrt{n}$. More draws shrink this sampling error but leave the bias from a time step or a truncated domain unchanged, and refining the step does not reduce sampling error. A convergence table and a sampling interval should each say which error they measure.

Definition Drift monitoring compares quantities observed after deployment, such as inputs, calibration residuals or realized outcomes, with a reference law, using a statistic, a threshold and an action that are all fixed before the new data are seen. For a mean monitor with a known-variance normal reference, assume that under the reference the observations $X_1, \dots, X_n$ are independent and identically distributed $N(\mu_0, \sigma_0^2)$ with $\mu_0$ and $\sigma_0$ known. Then

is exactly standard normal under the reference, and the rule that alerts when $\lvert Z \rvert > z_{*}$ raises a false alarm with probability $2(1 - \Phi(z_{*}))$ per check, about $0.0027$ for $z_{*} = 3$. If the observations are dependent, the variance of $\overline{X}$ is not $\sigma_0^2/n$. If the variance is estimated, the statistic with the sample standard deviation in place of $\sigma_0$ has a Student t distribution with $n - 1$ degrees of freedom under a normal reference. In both cases the reference calculation has to change. Repeated checks and many monitored quantities raise the chance of at least one false alarm. For $K$ checks at level $\alpha$ each, that chance is at most $K\alpha$ by the union bound, and it equals $1 - (1 - \alpha)^K$ when the checks are independent. An alert says that the new data are unusual under the reference, not which part of the model is wrong, and the absence of an alert does not show that nothing changed.

Definition A calculation is reproducible when its record is enough to regenerate the reported result within a stated tolerance. The record covers the input data with their definitions, timestamps, units and conventions, the parameters, the model and code version, the numerical settings such as grids, tolerances and sample counts, the software environment, and the random number generator with its seeds. A seed fixes the output only together with the same generator, the same assignment of random numbers to paths and the same order of floating-point operations, and parallel execution or a software change can alter the last two. Reproducibility is not correctness. Rerunning a calculation exactly reproduces its mistakes exactly, so it is paired with independent challenge, a review of whether the contract, the data, the derivation and the numerical method fit the intended use.

Definition The model limitations are the documented boundaries of the evidence for a model’s use. They state the hypotheses under which the outputs hold, the range of inputs and instruments the validation covered, the numerical accuracy achieved and the conditions under which it was shown, known weaknesses of the data, known failure modes, and the monitoring signals and actions that apply near those boundaries. Each check supports the use only where its hypotheses hold, so each hypothesis of a passed check is a candidate limitation.