Machine learning losses regularization and calibration

Definition Let $f_\theta$ be a prediction rule with parameter $\theta \in \Theta$ and $\ell(y, \hat{y})$ a loss. The population risk under the law $\mathbb{P}$ of the pairs $(X, Y)$ that the rule is meant to predict is

when the expectation exists. For a training sample $(x_1, y_1), \dots, (x_n, y_n)$ the empirical risk is the average loss

Regularized empirical risk minimization chooses a minimizer of $\widehat{R}_n(\theta) + \lambda\, \Omega(\theta)$, with a penalty $\Omega \ge 0$ and a strength $\lambda \ge 0$. If each training pair has the law $\mathbb{P}$, then $\mathbb{E}[\widehat{R}_n(\theta)] = R(\theta)$ for every fixed $\theta$. A parameter chosen from the sample is not fixed, though. Under the same hypothesis and with $\lambda = 0$, a minimizer $\hat\theta$ of the empirical risk and a minimizer $\theta^{*}$ of the population risk satisfy

provided these expectations exist, because $\widehat{R}_n(\hat\theta) \le \widehat{R}_n(\theta^{*})$ and $R(\theta^{*}) \le R(\hat\theta)$ hold on every sample. Here $R(\hat\theta)$ is the expected loss of the fitted rule on a new pair drawn from $\mathbb{P}$ independently of the sample.

Definition Let $X$ be an $n \times p$ design matrix, here a matrix of observed feature values, and $y \in \mathbb{R}^n$ the target vector, with no intercept and a strength $\lambda > 0$. The ridge estimator minimizes

Its gradient is $\nabla J(\theta) = \tfrac{2}{n} X^{\top}(X\theta - y) + 2\lambda\theta$. Setting it to zero and multiplying by $n/2$ gives the normal equations

For every nonzero $v \in \mathbb{R}^p$,

so the matrix is positive definite and invertible even when the columns of $X$ are linearly dependent. The Hessian $\nabla^2 J = \tfrac{2}{n} X^{\top}X + 2\lambda I$ is positive definite as well, so $J$ is strictly convex and its unique minimizer is

With $\lambda = 0$ the problem is ordinary least squares, whose inverse formula needs full column rank. Without full rank its minimizers form an affine set, and the pseudoinverse solution $X^{+}y$ is the one with the smallest Euclidean norm. An unpenalized intercept in the first column changes the penalty to $\lambda\, \theta^{\top} D \theta$ with $D = \operatorname{diag}(0, 1, \dots, 1)$ and the matrix to $X^{\top}X + n\lambda D$, which is still positive definite when that column is the vector of ones. The factor $n$ appears because the fitting error is averaged and the penalty is not. An objective written without the factor $1/n$ has $\lambda$ in its place.

Definition For a binary outcome $Y \in \{0, 1\}$ and a reported probability $q \in [0, 1]$, the Brier loss is $(Y - q)^2$, and the Brier score of $n$ forecasts is the average $\frac{1}{n} \sum_{i} (y_i - q_i)^2$. Let $p = \mathbb{P}(Y = 1 \mid X)$ be the true conditional probability given the features, and let the report $q$ be a function of $X$. Weighting the two possible errors by their conditional probabilities and adding and subtracting $p^2$ gives

The first term is the conditional variance of the event indicator and does not depend on the report. The second is zero only at $q = p$. The loss is dimensionless and lies in $[0, 1]$.

Definition A loss $\ell(y, q)$ for probability reports of a binary event is proper if, for every $p \in [0, 1]$, the expected loss

is minimized over $q$ at $q = p$, and strictly proper if $q = p$ is its only minimizer. The Brier loss is strictly proper, because $L(p, q) = p(1 - p) + (p - q)^2$, including the degenerate cases $p = 0$ and $p = 1$. The log loss $\ell(y, q) = -y \log q - (1 - y) \log(1 - q)$, with the natural logarithm, is strictly proper too. For $p$ and $q$ in $(0, 1)$,

which is negative for $q < p$ and positive for $q > p$, so $q = p$ is the unique minimizer, while the reports $q = 0$ and $q = 1$ give infinite expected loss. For $p \in \{0, 1\}$ the minimum is at the matching endpoint, with the convention $0 \log 0 = 0$. The absolute loss $\lvert y - q \rvert$ is not proper, since $L(p, q) = p + (1 - 2p)q$ is linear in $q$ and is minimized only at $q = 1$ whenever $p > 1/2$.

Definition Let $Y \in \{0, 1\}$ and let $Q = Q(X)$ be a reported probability with values in $[0, 1]$. The forecast is calibrated under $\mathbb{P}$ if

almost surely. When $Q$ takes finitely many values, this means $\mathbb{P}(Y = 1 \mid Q = q) = q$ for every value $q$ with $\mathbb{P}(Q = q) > 0$. The true conditional probability $\mathbb{P}(Y = 1 \mid X)$ is calibrated by the tower property, and so is the constant forecast equal to the base rate $\mathbb{P}(Y = 1)$. Write $c(Q) = \mathbb{E}[Y \mid Q]$. The squared-error identity of Financial prediction and feature construction, applied with the information generated by $Q$, and the law of total variance give

The last term, the reliability term, is zero exactly when the forecast is calibrated. The middle term, the resolution, measures how far the forecast separates cases with different event rates, which calibration alone does not provide.

Definition Let $\mathbb{P}_{\text{train}}$ be the law of the training pairs $(X, Y)$ and $\mathbb{P}_{\text{deploy}}$ the law of the pairs on which the rule will be used, both physical laws of observations. A distribution shift is any difference between the two laws. Covariate shift is the case in which the law of $X$ changes while the conditional law of $Y$ given $X = x$ stays the same, at least for deployment-almost every $x$. Suppose in addition that the deployment law of $X$ gives probability zero to every set of training probability zero, which is absolute continuity, so that it has a density ratio $w(x)$ with respect to the training law of $X$. Where both laws have densities, $w(x) = p_{\text{deploy}}(x)/p_{\text{train}}(x)$ at points with $p_{\text{train}}(x) > 0$. If also $\mathbb{E}_{\text{train}}[w(X)\, \lvert \ell(Y, f(X)) \rvert] < \infty$, then

The proof writes the deployment risk as an integral of the loss against the common conditional law of $Y$ and the deployment law of $X$, and then multiplies and divides by $p_{\text{train}}(x)$ where it is positive. If the conditional law changes as well, the weights alone no longer give the deployment risk. If deployment reaches a region the training law never visits, no weight exists, and the training sample carries no information about the loss there.