Loss Functions and Empirical Risk, machine learning lesson

A model that predicts delivery times is off by about one minute on most orders and by ten minutes on a few. A second model is off by about three minutes on nearly every order. Which model is better has no answer until we say what a mistake costs. This lesson defines that cost for a single prediction, averages it over a data set, separates the average on training cases from the expected cost on new cases, and shows which prediction each common loss rewards.

Key terms

Loss function

Definition A loss function $L$ assigns a number $L(y, \hat{y})$ to each pair of an observed target $y$ and a prediction $\hat{y}$ . The number is the cost charged for predicting $\hat{y}$ when the outcome turns out to be $y$ . For a numerical target, the losses on this page are nonnegative and equal zero exactly when $\hat{y} = y$ . Each of them depends only on the residual

e = y - \hat{y},

which on this page means observed minus predicted, so a positive residual says the prediction was too low. Some texts and software use $\hat{y} - y$ instead. A loss that depends only on $\lvert e \rvert$ is unchanged by that switch, but an asymmetric loss and any statement about the direction of an error are not.

In plain words A loss is a price list for mistakes. Choosing it is part of stating the prediction problem, because it decides which errors count as serious before any model is compared.

Example Three constructed deliveries took 3, 5 and 9 minutes, and a model predicted 2, 6 and 5 minutes. The residuals are $1$ , $-1$ and $4$ minutes. Suppose a planner charges 3 cost units for each minute of underprediction and 1 unit for each minute of overprediction, because a delivery that arrives later than predicted does more harm than an early one. The asymmetric loss

\begin{aligned} L(y, \hat{y}) &= 3\max(e, 0) \\[0.4em] &\quad + \max(-e, 0) \end{aligned}

charges 3, 1 and 12 units for the three deliveries. A residual of $2$ costs 6 units and a residual of $-2$ only 2 units, although both are two-minute errors. Under this loss the sign convention matters, so it has to be stated with the loss.

Squared error and absolute error

Definition The squared error loss is $L(y, \hat{y}) = (y - \hat{y})^2$ , measured in the square of the target’s unit. The absolute error loss is $L(y, \hat{y}) = \lvert y - \hat{y} \rvert$ , measured in the target’s unit. Averaged over $n$ cases with residuals $e_i = y_i - \hat{y}_i$ , they give the mean squared error and the mean absolute error

\begin{aligned} \operatorname{MSE} &= \frac{1}{n}\sum_{i=1}^{n} e_i^2, \\[0.4em] \operatorname{MAE} &= \frac{1}{n}\sum_{i=1}^{n} \lvert e_i \rvert . \end{aligned}

The root mean squared error $\operatorname{RMSE} = \sqrt{\operatorname{MSE}}$ is back in the target’s unit. It is never smaller than the MAE, and the two are equal exactly when all the $\lvert e_i \rvert$ are equal. Both losses are symmetric, so an error of a given size costs the same whether the prediction was too high or too low.

In plain words Absolute error charges every minute of error the same price. Squared error charges more for each additional minute, so one large miss can outweigh several small ones. Neither is right in general. Squared error suits a task in which the cost of a miss grows faster than its size, and absolute error suits one in which the cost grows in proportion to the size.

Example For the residuals $1$ , $-1$ and $4$ minutes, the squared errors are 1, 1 and 16, so the MSE is $18/3 = 6$ squared minutes. The absolute errors are 1, 1 and 4, so the MAE is $6/3 = 2$ minutes. The four-minute miss supplies 16 of the 18 squared units but only 4 of the 6 absolute units. A second model with residuals $2$ , $2$ and $2$ has the same MAE of 2 minutes but an MSE of only 4 squared minutes. Absolute error calls the two models equal, and squared error prefers the second. The first model’s RMSE is $\sqrt{6} \approx 2.45$ minutes, above its MAE because its errors have unequal sizes.

Empirical risk

Definition Let $(\mathbf{x}_1, y_1), \dots, (\mathbf{x}_n, y_n)$ be observed cases, $f$ a prediction function and $L$ a loss. The empirical risk of $f$ is its average loss on these cases,

\hat{R}_n(f) = \frac{1}{n}\sum_{i=1}^{n} L\bigl(y_i, f(\mathbf{x}_i)\bigr).

A hypothesis class $\mathcal{H}$ is the set of prediction functions a method may choose from, for example all functions $f(x) = b + wx$ of one feature, where the parameters $\boldsymbol{\theta} = (b, w)$ pick out one member. Empirical risk minimization on training cases chooses

\hat{f} \in \operatorname{argmin}_{f \in \mathcal{H}} \hat{R}_n(f),

a member of the class with the smallest training average. Such a minimizer need not be unique, and for some classes and losses none exists. For a fixed $n$ , the sum of the losses has the same minimizers as their average, but it is $n$ times larger, which starts to matter once another term such as a penalty is added.

In plain words Empirical risk is a score computed from the data at hand. Training a model means searching the allowed functions for one with a low score on the training cases. Changing the parameters moves within the class, while changing the class changes which prediction rules are possible at all.

Example Take the class of constant predictions $f(\mathbf{x}) = a$ , which ignore the features, and training targets of 2, 3 and 10 minutes. Under squared error

\begin{aligned} \hat{R}_3(a) &= \tfrac13\bigl[(2 - a)^2 + (3 - a)^2 \\[0.4em] &\qquad + (10 - a)^2\bigr], \end{aligned}

whose derivative $2(a - 5)$ vanishes only at $a = 5$ . The second derivative is $2 > 0$ , so the sample mean 5 is the unique minimizer, with training MSE $38/3 \approx 12.67$ squared minutes. Under absolute error the minimizer is the sample median 3, with training MAE $8/3 \approx 2.67$ minutes. Both numbers are averages over the cases used to choose $a$ , so they describe the fit, not how well the constant will predict new deliveries.

Population risk

Definition Let $P$ be the distribution of a new case $(\mathbf{X}, Y)$ in the setting where the predictions will be used. The population risk of a fixed prediction function $f$ is its expected loss on such a case,

R(f) = E\bigl[L\bigl(Y, f(\mathbf{X})\bigr)\bigr].

If $f$ is fixed before evaluation cases are drawn independently from $P$ , and $R(f)$ is finite, then the empirical risk of $f$ on those cases has expected value $R(f)$ , so it is an unbiased estimate. The argument fails for a function chosen with the help of the same cases. If $\hat{f}$ minimizes the empirical risk over $\mathcal{H}$ on independent training cases from $P$ , then

\begin{aligned} E\bigl[\hat{R}_n(\hat{f})\bigr] &\le \inf_{f \in \mathcal{H}} R(f) \\[0.4em] &\le E\bigl[R(\hat{f})\bigr], \end{aligned}

where the expectations are over the training sample and $R(\hat{f})$ is the population risk of the chosen function on a new case independent of that sample. On average, the training risk of the chosen function is therefore no larger than its population risk. A test set drawn from $P$ and never used for any choice gives an unbiased estimate of $R(\hat{f})$ . Reusing it to choose features, models or settings removes that guarantee. If the setting of use differs from $P$ , even an exact value of $R(\hat{f})$ answers a different question.

In plain words Population risk is the average cost the model would incur over all the cases it will meet. It cannot be computed from data, only estimated. When a model is chosen by minimizing its training risk, that training risk is optimistic on average, because the model was picked to do well on those very cases. More data shrink the random scatter of an average, but they cannot repair an average taken over the wrong kind of cases.

Example Suppose deliveries under one route condition take 1, 3 or 9 minutes with probabilities $1/4$ , $1/2$ and $1/4$ , and the constant prediction 4 minutes is fixed in advance. Its population squared risk is

\begin{aligned} R(4) &= \tfrac14 \cdot 9 + \tfrac12 \cdot 1 + \tfrac14 \cdot 25 \\[0.4em] &= 9 . \end{aligned}

A test sample 3, 3, 9, 1 gives empirical risk 9 and a sample 3, 3, 3, 1 gives 3. Single samples scatter, but for four independent draws from this distribution the expected empirical risk is exactly 9. Now let the constant be fitted instead, as the mean of four training deliveries. For the training sample 1, 1, 3, 3 the fitted constant is 2, its training MSE is 1 squared minute and its population risk is $9 + (4 - 2)^2 = 13$ . Over all training samples of four independent draws, the fitted constant has expected training MSE 6.75 and expected population risk 11.25, on the two sides of the best possible value 9.

Conditional mean

Definition For a feature value $\mathbf{x}$ , the conditional mean of the target is

m(\mathbf{x}) = E[Y \mid \mathbf{X} = \mathbf{x}].

Write $\mid \mathbf{x}$ for conditioning on $\mathbf{X} = \mathbf{x}$ . If $E[Y^2 \mid \mathbf{x}] < \infty$ , then every real prediction $a$ satisfies

\begin{aligned} &E\bigl[(Y - a)^2 \mid \mathbf{x}\bigr] \\[0.4em] &\quad = \operatorname{Var}(Y \mid \mathbf{x}) \\[0.4em] &\qquad + \bigl(m(\mathbf{x}) - a\bigr)^2 . \end{aligned}

This follows from writing $Y - a = (Y - m) + (m - a)$ and expanding, since the cross term has conditional expectation $2(m - a)E[Y - m \mid \mathbf{x}] = 0$ . So $a = m(\mathbf{x})$ is the unique minimizer of the expected squared loss at $\mathbf{x}$ . If $E[Y^2] < \infty$ , averaging over $\mathbf{X}$ shows that $m$ has the smallest population squared risk of all prediction functions, and any $f$ with $P\bigl(f(\mathbf{X}) \ne m(\mathbf{X})\bigr) > 0$ has a larger one. For absolute error the answer is a median instead. If $E[\lvert Y \rvert \mid \mathbf{x}] < \infty$ , the minimizers of $E[\lvert Y - a \rvert \mid \mathbf{x}]$ are exactly the conditional medians, the values $a$ with $P(Y \le a \mid \mathbf{x}) \ge 1/2$ and $P(Y \ge a \mid \mathbf{x}) \ge 1/2$ . They form a closed interval, which may be a single point.

In plain words Under squared error, the best single guess for a given situation is the average outcome in that situation, and the expected loss left over is the variance that no prediction based on $\mathbf{x}$ can remove. Under absolute error, the best guess is a middle value. A restricted class, such as straight lines, may be unable to represent a curved conditional mean, and a finite data set only estimates it.

Example Return to the route condition with outcomes 1, 3 and 9 minutes and probabilities $1/4$ , $1/2$ and $1/4$ . The conditional mean is $\tfrac14 + \tfrac32 + \tfrac94 = 4$ minutes and the conditional variance is 9 squared minutes. Predicting 4 has expected squared loss 9, and predicting 3 has $9 + (3 - 4)^2 = 10$ . Under absolute error the order reverses. Predicting 3 costs $\tfrac14 \cdot 2 + \tfrac12 \cdot 0 + \tfrac14 \cdot 6 = 2$ minutes on average and predicting 4 costs $\tfrac14 \cdot 3 + \tfrac12 \cdot 1 + \tfrac14 \cdot 5 = 2.5$ , because 3 is the conditional median, with $P(Y \le 3) = P(Y \ge 3) = 3/4$ . The mean 4 is not one of the possible outcomes, and a mean need not be. If the outcomes were instead 2 and 8 minutes with probability $1/2$ each, every prediction from 2 to 8 would have expected absolute loss 3, so the minimizer would not be unique.

Training objective and evaluation metric

Definition The training objective is the complete function that an optimizer minimizes over the parameters. It may be the empirical risk alone or contain further terms, for example

\begin{aligned} J(\boldsymbol{\theta}) &= \hat{R}_n(f_{\boldsymbol{\theta}}) \\[0.4em] &\quad + \lambda \sum_{j=1}^{p} w_j^2 , \end{aligned}

where $\lambda \ge 0$ sets the strength of a penalty on the slope coefficients $w_1, \dots, w_p$ and the intercept $b$ is not penalized. With $\lambda = 0$ the objective is the empirical risk again. Sample weights $\omega_i \ge 0$ with a positive total change the objective as well, replacing the plain average of the case losses $L_i$ by

\hat{R}_{\omega} = \frac{\sum_{i=1}^{n} \omega_i L_i}{\sum_{i=1}^{n} \omega_i}.

An evaluation metric is a number reported to judge predictions, such as the MAE, the RMSE or a decision cost, and it is reported together with the data it was computed on. Validation data serve while choices are still being made, and a test set kept apart from all choices gives the final estimate. The metric need not be the training objective.

In plain words The loss prices one case, the objective is what training actually minimizes, and the metric is what gets reported. Keeping them apart prevents two mistakes, reading a low training objective as good performance on new data, and expecting the optimizer to have improved a quantity it never saw.

Example Two candidate models have training MSEs of 6 and 4 and squared slope norms $\lVert \mathbf{w} \rVert_2^2$ of 1 and 9. With $\lambda = 0.5$ their objectives are $6 + 0.5 \cdot 1 = 6.5$ and $4 + 0.5 \cdot 9 = 8.5$ , so the penalized objective prefers the first model although its MSE is larger. Sample weights change the training average too. Giving the squared losses 1, 1 and 16 of the delivery example the weights 1, 1 and 2 produces $(1 + 1 + 32)/4 = 8.5$ in place of the unweighted 6. Weights like these need a reason from the application, such as a target population in which the third kind of delivery is twice as common, not merely a better score.

Check yourself

Two models are evaluated on the same four validation deliveries. Model A has residuals, observed minus predicted, of 1, $-2$ , 0 and 5 minutes, and model B has residuals of 3, $-3$ , 2 and $-2$ minutes. Compute the MAE, MSE and RMSE of each model. Which model does each loss prefer, and which would you choose for a task in which one large miss is much more costly than several small ones?

Show the answer

For model A the absolute errors 1, 2, 0 and 5 add to 8, so the MAE is 2 minutes. The squared errors 1, 4, 0 and 25 add to 30, so the MSE is 7.5 squared minutes and the RMSE is $\sqrt{7.5} \approx 2.74$ minutes.

For model B the absolute errors 3, 3, 2 and 2 add to 10, so the MAE is 2.5 minutes. The squared errors 9, 9, 4 and 4 add to 26, so the MSE is 6.5 squared minutes and the RMSE is $\sqrt{6.5} \approx 2.55$ minutes.

Absolute error prefers A, while squared error and RMSE prefer B. When one large miss is much more costly than several small ones, squared error matches the task better, and it favors B, whose largest miss is 3 minutes against the 5-minute miss of A. Four validation cases are very few, so this choice should be confirmed on more data.

A model predicts the same constant $a$ for every case, and its training targets are 1, 2, 4 and 13 minutes. Find the constant that minimizes the training MSE and the minimum value. Then find every constant that minimizes the training MAE. A classmate says that the median 3 is the only MAE minimizer. Is that right?

Show the answer

The training MSE is minimized by the sample mean $a = 20/4 = 5$ . The squared errors are 16, 9, 1 and 64, so the minimum MSE is $90/4 = 22.5$ squared minutes.

For any $a$ from 2 to 4, the distances to 1 and 13 add to $(a - 1) + (13 - a) = 12$ and the distances to 2 and 4 add to 2, so the MAE is $14/4 = 3.5$ minutes. Below 2, raising $a$ brings it closer to at least three targets and farther from at most one, so the MAE falls. Above 4, lowering $a$ does the same. Every constant in the interval from 2 to 4 is therefore a minimizer, with MAE 3.5.

The classmate is wrong. The median 3 is one minimizer, but with an even number of targets and two different middle values, every point between them minimizes the MAE. The mean 5 lies outside that interval, pulled up by the large target 13, and its MAE is 4.

A team trains with the objective $J = \text{MSE} + \lambda \lVert \mathbf{w} \rVert_2^2$ , where the MSE is computed on the training cases. Model A has training MSE 3 and $\lVert \mathbf{w} \rVert_2^2 = 4$ , and model B has training MSE 2 and $\lVert \mathbf{w} \rVert_2^2 = 10$ . Which model has the smaller objective when $\lambda = 0.25$ , and which when $\lambda = 0.05$ ? A colleague then reports the training MSE of 2 as the error to expect from model B on new deliveries. What is wrong with that, and what should be reported instead?

Show the answer

With $\lambda = 0.25$ the objectives are $3 + 0.25 \cdot 4 = 4$ for A and $2 + 0.25 \cdot 10 = 4.5$ for B, so A wins. With $\lambda = 0.05$ they are $3 + 0.2 = 3.2$ and $2 + 0.5 = 2.5$ , so B wins. The penalty weight changes the choice, which is why it is itself chosen on validation data rather than on the test set.

The training MSE was computed on the cases the model was fitted to, so it is not an unbiased estimate of the error on new deliveries and is usually too optimistic. It also measures a different quantity from the objective that training minimized. The report should give a stated metric, such as the MSE or the MAE, computed on a test set that played no part in any choice and was drawn from the deliveries the model will serve.

Loss Functions and Empirical Risk

Key terms

Loss function

Squared error and absolute error

Empirical risk

Population risk

Conditional mean

Training objective and evaluation metric

Check yourself

How this lesson connects

Builds on

Used later in