Regression Metrics and Baseline Comparisons, machine learning lesson

Two demand models were fitted earlier and frozen, and both are now scored on the same six held-out cases. One standard metric ranks model A first and another ranks model B first. Both rankings are correct, because each metric decides how much one large miss counts against several small ones. This lesson defines the common regression metrics with their units, shows which reference R-squared compares with, fits mean and median baselines on training data for an honest comparison, and explains when a percentage error stops making sense.

Key terms

Mean absolute error

Definition Suppose a fitted model, frozen before evaluation, makes predictions $\hat{y}_1, \dots, \hat{y}_n$ for $n \ge 1$ held-out cases with observed targets $y_1, \dots, y_n$ . This lesson writes the error of case $i$ as $e_i = \hat{y}_i - y_i$ , prediction minus observation, so a positive error is an overprediction. The mean absolute error is

\operatorname{MAE} = \frac{1}{n} \sum_{i=1}^{n} \lvert e_i \rvert ,

the average absolute loss over the evaluation cases. It is measured in the units of the target, it is zero exactly when every prediction is exact, and it does not depend on the sign convention for errors. Each case gets weight $1/n$ , and two models are compared fairly only when both are scored on the same cases with the same weights.

In plain words The MAE is the average size of a miss, in target units. A miss twice as large counts twice as much, and an overprediction and an underprediction of the same size count the same. If one model is scored on easier cases, or loses different rows to missing labels, a lower MAE may describe a changed evaluation set rather than a better model. An average over all cases can also hide a group where the model fails, such as inputs outside the range seen in training, so such groups deserve their own scores and case counts.

Example Six constructed held-out cases record demand in units.

Case	1	2	3	4	5	6
Observed $y_i$	0	2	4	6	8	10
Model A	1	3	5	7	9	16
Model B	2	4	6	8	10	12

Model A has errors 1, 1, 1, 1, 1 and 6 units, five small misses and one large one, while model B misses every case by 2 units. So the MAE of model A is $11/6 \approx 1.833$ units and that of model B is $12/6 = 2$ units. Under MAE, model A ranks first.

Mean squared error and root mean squared error

Definition With the same predictions, targets and errors, the mean squared error and the root mean squared error are

\begin{gathered} \operatorname{MSE} = \frac{1}{n} \sum_{i=1}^{n} e_i^2 , \\[0.4em] \operatorname{RMSE} = \sqrt{\operatorname{MSE}} . \end{gathered}

The MSE is the average squared loss, measured in squared target units, and the RMSE is measured in target units. The square root is taken after averaging. Because the square root is strictly increasing, MSE and RMSE rank models in the same order whenever they are computed on the same cases with the same weights. The RMSE is never smaller than the MAE, because

\begin{aligned} &\operatorname{MSE} - \operatorname{MAE}^2 \\[0.4em] &\quad = \frac{1}{n} \sum_{i=1}^{n} \bigl(\lvert e_i \rvert - \operatorname{MAE}\bigr)^2 , \end{aligned}

and the two are equal exactly when all the absolute errors are equal.

In plain words Squaring makes a large miss count far more than a small one. A miss of 6 units adds 36 times as much as a miss of 1 unit, not 6 times as much. Taking the root brings the score back to target units but keeps that emphasis, so the RMSE is not the average size of a miss. A large gap between RMSE and MAE says that some misses are much larger than others.

Example Model A has squared errors 1, 1, 1, 1, 1 and 36, so its MSE is $41/6 \approx 6.833$ squared units and its RMSE is $\sqrt{41/6} \approx 2.614$ units. Every squared error of model B is 4, so its MSE is 4 squared units and its RMSE is 2 units, equal to its MAE because all its misses have the same size. Model B now ranks first. Both rankings are correct for their metrics. If one large miss is especially costly, for example because a big shortage loses a customer, the squared metrics match the decision better, and if the cost of a miss grows in proportion to its size, the MAE does.

R-squared

Definition Let $\bar{y}$ be the mean of the evaluation targets $y_1, \dots, y_n$ , and let

\begin{aligned} \text{SSE} &= \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 , \\[0.4em] \text{SST} &= \sum_{i=1}^{n} (y_i - \bar{y})^2 . \end{aligned}

When $\text{SST} > 0$ , the R-squared of the predictions on these cases is

R^2 = 1 - \frac{\text{SSE}}{\text{SST}} .

It is at most 1, with equality exactly when every prediction is exact. It is 0 exactly when $\text{SSE} = \text{SST}$ , and it is negative, with no lower limit, when $\text{SSE} > \text{SST}$ . If all evaluation targets are equal, then $\text{SST} = 0$ and the formula is undefined, even for perfect predictions, where it would read $0/0$ . Some software returns a conventional finite value in that case, and a report should say which. On one fixed evaluation set $R^2 = 1 - n \operatorname{MSE}/\text{SST}$ , so R-squared ranks models exactly as MSE does.

In plain words R-squared compares the squared error of the predictions with that of a yardstick that predicts the evaluation mean for every case. The yardstick uses the very answers being scored, so it is a reference calculation, not a prediction anyone could have made in advance. A negative value means more squared error than the yardstick, which is entirely possible on held-out data. R-squared is not the share of predictions that are correct, and a high value does not show that a cause has been found. Because SST depends on how spread out the evaluation targets are, the same MSE gives different R-squared values on different evaluation sets.

Example The six observed demands have mean $\bar{y} = 5$ and deviations $-5, -3, -1, 1, 3, 5$ , so $\text{SST} = 70$ . Model A has $\text{SSE} = 41$ and model B has $\text{SSE} = 24$ , so

\begin{aligned} R^2_A &= 1 - \tfrac{41}{70} = \tfrac{29}{70} \approx 0.414 , \\[0.4em] R^2_B &= 1 - \tfrac{24}{70} = \tfrac{23}{35} \approx 0.657 . \end{aligned}

The order is the same as under MSE and RMSE.

Mean baseline

Definition A baseline is a simple predictor that ignores the features, is fitted on the training data only and is then frozen. Let $t_1, \dots, t_m$ be the training targets, with mean $\bar{t}$ . The mean baseline predicts $\bar{t}$ for every case. For every real number $c$ ,

\begin{aligned} \sum_{j=1}^{m} (t_j - c)^2 &= \sum_{j=1}^{m} (t_j - \bar{t})^2 \\[0.4em] &\quad + m(\bar{t} - c)^2 , \end{aligned}

so the training mean is the unique constant with the smallest training squared error. On held-out cases with mean $\bar{y}$ and $\text{SST} > 0$ , the same identity gives $\text{SSE} = \text{SST} + n(\bar{y} - c)^2$ for a constant prediction $c$ , so

R^2 = -\frac{n(\bar{y} - c)^2}{\text{SST}} \le 0 .

The held-out R-squared of the mean baseline is therefore never positive, and it is zero only when the training mean equals the evaluation mean.

In plain words The mean baseline asks how well one can predict knowing only the training targets. A useful model should beat it on the same held-out cases under the same metric. Refitting the baseline on the evaluation targets would use answers that are not available at prediction time, and it would give R-squared exactly 0 by construction.

Example Take the constructed training targets 1, 2, 3, 4 and 20. Their mean is 6, and the training squared error total is 250 at 6, compared with 295 at the median 3. On the six held-out cases the constant 6 has errors 6, 4, 2, 0, $-2$ and $-4$ , so $\text{SSE} = 76 = 70 + 6 \cdot 1^2$ and

R^2 = 1 - \frac{76}{70} = -\frac{3}{35} ,

about $-0.086$ . Its MAE is 3 units and its RMSE is $\sqrt{38/3} \approx 3.559$ units, so both models beat it on every metric here.

Median baseline

Definition A number $c$ is a median of $t_1, \dots, t_m$ if at least half of the values are at most $c$ and at least half are at least $c$ . The median baseline predicts a training median for every case. A constant $c$ minimizes the training absolute error total

\sum_{j=1}^{m} \lvert t_j - c \rvert

if and only if $c$ is a median. For odd $m$ the median is the middle ordered value and is unique. For even $m$ every number from the lower to the upper of the two middle ordered values minimizes the total, and a convention such as their midpoint picks one.

In plain words Move a constant prediction a small step to the right without passing any training value. The step adds to the distance from every value at or below the prediction and subtracts from the distance to every value above it. So the total falls while more values lie above than at or below, rises once fewer do, and stays level when the counts are equal, which is why an even number of values can have a whole interval of best constants. Since the median depends only on the order of the values, one extreme value pulls it much less than it pulls the mean.

Example For the training targets 1, 2, 3, 4 and 20, the median is 3. The absolute error total is $2 + 1 + 0 + 1 + 17 = 21$ at 3 and 28 at the mean 6. The value 20 lifts the mean to 6 but leaves the median at 3. On the six held-out cases the constant 3 has errors 3, 1, $-1$ , $-3$ , $-5$ and $-7$ , an MAE of $10/3 \approx 3.333$ units, larger than the held-out MAE of 3 units of the mean baseline. Each baseline is optimal for its loss on the sample it was fitted to, not on new cases.

Mean absolute percentage error

Definition If every evaluation target is nonzero, the mean absolute percentage error is

\operatorname{MAPE} = \frac{100}{n} \sum_{i=1}^{n} \frac{\lvert \hat{y}_i - y_i \rvert}{\lvert y_i \rvert}

percent. Some definitions leave out the factor 100 and report a fraction. If any target is zero, the formula is undefined. Replacing $\lvert y_i \rvert$ by $\max(\lvert y_i \rvert, \varepsilon)$ for a small $\varepsilon > 0$ , or adding $\varepsilon$ to it, defines a different metric, in which one row with a target near zero can dominate the average. The MAPE has no units and does not change when targets and predictions are multiplied by the same nonzero number, but it does change when the zero point of the scale moves. For positive targets and nonnegative predictions, an underprediction contributes at most 100 percent, while an overprediction has no upper limit.

In plain words The MAPE measures each miss relative to the size of the target. That suits quantities with a true zero, such as counts and amounts, when a miss of 5 matters more on a target of 10 than on a target of 1000. It does not suit targets near zero, or temperatures in degrees Celsius, whose zero point is a convention. Absolute and relative errors answer different questions, and neither is wrong in general.

Example A target of 0.01 predicted as 0.02 has absolute error 0.01 and percentage error 100 percent. A target of 100 predicted as 100.01 has the same absolute error and a percentage error of 0.01 percent. In the demand example the first observed demand is 0, so the MAPE of both models is undefined. Dropping that case without saying so would change the evaluation set, so a report keeps the case and names whatever metric it uses instead.

Check yourself

A frozen model predicts 3 units for each of three new held-out cases whose observed demands are 2, 4 and 6 units. Compute the MAE, the MSE and the RMSE, with units.

Show the answer

The errors, prediction minus observation, are 1, $-1$ and $-3$ units. The absolute errors add to 5, so the MAE is $5/3 \approx 1.667$ units. The squared errors are 1, 1 and 9, which add to 11, so the MSE is $11/3 \approx 3.667$ squared units and

\operatorname{RMSE} = \sqrt{11/3} \approx 1.915

units. The RMSE is larger than the MAE because the absolute errors are not all equal.

For the same three cases, compute R-squared. A teammate says the result must be an arithmetic slip, because R-squared always lies between 0 and 1. Respond, and then say what happens to R-squared, the MAE and the RMSE if all three observed demands had been 4 units.

Show the answer

The evaluation mean is 4, so $\text{SST} = 4 + 0 + 4 = 8$ , while $\text{SSE} = 11$ . Hence

R^2 = 1 - \frac{11}{8} = -\frac{3}{8} ,

or $-0.375$ . The value matches the formula for a constant prediction, $-3(4 - 3)^2/8 = -3/8$ . The teammate’s rule holds for a least-squares fit with an intercept, scored on the data it was fitted to. Held-out predictions have no such guarantee, and the negative value only says that predicting 3 for every case leaves more squared error, 11, than the reference that predicts the evaluation mean 4 for every case, 8.

If all three demands were 4, then $\text{SST} = 0$ and R-squared would be undefined, since the formula divides by zero. Each error would be $-1$ , so the MAE and the RMSE would both be 1 unit, and they remain perfectly well defined.

Three held-out cases have observed values 0, 5 and 50 units and predictions 1, 6 and 49 units. Find the MAE, and say whether the MAPE can be reported. One teammate drops the first case and reports a MAPE of 11 percent. Another replaces each denominator $\lvert y_i \rvert$ by $\max(\lvert y_i \rvert, 0.1)$ and reports about 341 percent. What does each number describe?

Show the answer

Every error has size 1, so the MAE is 1 unit. The MAPE of these three cases cannot be reported, because the first target is 0 and the raw formula divides by it.

The value 11 percent is the MAPE of a different evaluation set, the two cases with targets 5 and 50, whose percentage errors are 20 and 2 percent. Reporting it as the MAPE of all three cases hides the excluded case. The value of about 341 percent comes from a modified metric. The first case contributes $1/0.1 = 10$ , or 1000 percent, against 20 and 2 percent for the others, so the average $1022/3 \approx 340.7$ percent is driven almost entirely by an arbitrary floor on one denominator. Neither number describes relative accuracy on all three cases. A clear report gives the MAE, or a relative metric with its definition stated, together with the number of zero targets.

Regression Metrics and Baseline Comparisons

Key terms

Mean absolute error

Mean squared error and root mean squared error

R-squared

Mean baseline

Median baseline

Mean absolute percentage error

Check yourself

How this lesson connects

Builds on

Used later in