Simple Linear Regression, machine learning lesson

Three constructed delivery routes of 0, 10 and 20 kilometers took 1, 2 and 2 hours. No straight line passes through all three points, yet a line that predicts duration from distance is still useful, provided we can say which line is best and how far to trust it. This lesson fits that line by least squares, derives its slope and intercept, reads its residuals, and marks where a prediction stops being supported by the data.

Key terms

Simple linear regression

Definition Simple linear regression predicts a numerical target $y$ from one numerical feature $x$ with the line

\hat{y} = b + wx,

where the intercept $b$ and the slope $w$ are parameters fitted to $n$ training pairs $(x_1, y_1), \dots, (x_n, y_n)$ . When the line is read as a model of the population, it says that the conditional mean $E[Y \mid X = x]$ is a straight-line function of $x$ . That is an assumption, and a fitted line does not prove it. Computing the least-squares line needs no assumption about the distribution of the errors, in particular no Gaussian assumption. Such assumptions matter only for particular statements about uncertainty, such as confidence intervals.

In plain words One input, one output and a straight line between them. Finding the best line is a calculation on the data that works for any data with at least two different feature values. Whether a line describes the relationship well, and whether it predicts new cases well, are separate questions.

Example In the constructed data, $x$ is route distance in tens of kilometers and $y$ is duration in hours, with observations $(0, 1)$ , $(1, 2)$ and $(2, 2)$ . The intercept is then in hours and the slope in hours per ten kilometers. A route of distance zero can still take time for loading and handling, so a nonzero intercept is not absurd here.

Fitted value and residual

Definition For a line $\hat{y} = b + wx$ , the fitted value of training case $i$ is $\hat{y}_i = b + w x_i$ . On this page the residual is the fitted value minus the observed target,

r_i = \hat{y}_i - y_i ,

so a positive residual means the line overpredicts and a negative one means it underpredicts. Many texts, including the loss examples of Loss Functions and Empirical Risk, use the opposite sign $y_i - \hat{y}_i$ . Squared residuals are the same under both conventions, but the meaning of a positive residual flips. A residual is measured vertically, at the case’s own feature value, not as the perpendicular distance from the point to the line. The same difference for a case that was not used in fitting is a prediction error on new data, not a residual.

In plain words The fitted value is what the line says at a training point, and the residual says by how much and in which direction the line missed that point.

Example Try the candidate line $\hat{y} = 1 + 0.5x$ on the three routes. Its fitted values are 1, 1.5 and 2 hours, so its residuals are $0$ , $-0.5$ and $0$ hours, and it underpredicts the 10-kilometer route by half an hour. Its squared residuals add to 0.25. At the point $(1, 2)$ the vertical residual has size 0.5, while the perpendicular distance from the point to the line is about 0.45. Least squares uses the vertical one, because it compares the prediction and the observation at the same distance.

Least squares

Definition The least-squares line is the line whose intercept and slope minimize the sum of squared residuals

\operatorname{SSE}(b, w) = \sum_{i=1}^{n} (b + w x_i - y_i)^2 .

For any fixed slope $w$ , setting the derivative with respect to $b$ to zero gives $\sum_i r_i = 0$ , so the best intercept is $b = \bar{y} - w\bar{x}$ . With that intercept every residual equals $w(x_i - \bar{x}) - (y_i - \bar{y})$ , and

\operatorname{SSE} = S_{yy} - 2wS_{xy} + w^2 S_{xx},

where $S_{yy} = \sum_i (y_i - \bar{y})^2$ and

\begin{aligned} S_{xx} &= \textstyle\sum_i (x_i - \bar{x})^2, \\[0.4em] S_{xy} &= \textstyle\sum_i (x_i - \bar{x})(y_i - \bar{y}). \end{aligned}

If $S_{xx} > 0$ , which holds exactly when at least two feature values differ, this is an upward parabola in $w$ , and the unique least-squares line has

\begin{gathered} w = \frac{S_{xy}}{S_{xx}}, \\[0.4em] b = \bar{y} - w\bar{x}. \end{gathered}

The line passes through $(\bar{x}, \bar{y})$ , and its residuals satisfy $\sum_i r_i = 0$ and $\sum_i x_i r_i = 0$ , which are the two first-order conditions for $b$ and $w$ . These sums describe the fit and say nothing about whether the errors are independent. If $S_{xx} = 0$ , every line through $(\bar{x}, \bar{y})$ gives the same smallest SSE, so the slope is not determined.

In plain words For any slope, the best intercept puts the line through the point of averages. What remains is the tilt, and the best tilt compares how the two variables move together with how much the feature varies on its own. If the feature never varies, the data say nothing about the tilt.

Example For the three routes the means are $\bar{x} = 1$ and $\bar{y} = 5/3$ . The centered distances are $-1$ , $0$ and $1$ , and the centered durations are $-2/3$ , $1/3$ and $1/3$ , so $S_{xx} = 2$ and $S_{xy} = 2/3 + 0 + 1/3 = 1$ . The slope is $w = 1/2$ and the intercept is $b = 5/3 - 1/2 = 7/6$ . The fitted values are $7/6$ , $5/3$ and $13/6$ hours, and the residuals are $1/6$ , $-1/3$ and $1/6$ hour. They add to zero, their sum weighted by distance $0 - 1/3 + 2/6$ is also zero, and the SSE is $1/36 + 1/9 + 1/36 = 1/6$ , smaller than the 0.25 of the candidate line. The MSE is $1/18$ squared hour. A few lines of NumPy reproduce the fit.

import numpy as np

x = np.array([0.0, 1.0, 2.0])  # distance, tens of km
y = np.array([1.0, 2.0, 2.0])  # duration, hours

w = np.sum((x - x.mean()) * (y - y.mean())) / np.sum((x - x.mean()) ** 2)
b = y.mean() - w * x.mean()
r = (b + w * x) - y            # residuals, fitted minus observed
print(b, w)                    # 1.1666666666666667 0.5
print(r.sum(), (x * r).sum())  # both zero up to rounding, about 1e-16

Slope and intercept

Definition In a fitted line $\hat{y} = b + wx$ , the slope $w$ is the change in the predicted value when $x$ increases by one unit, measured in target units per feature unit. The intercept $b$ is the predicted value at $x = 0$ , measured in target units. Both describe the fitted model. The slope compares predictions for cases whose feature values differ by one unit. It is not by itself the effect of changing $x$ for a given case, which needs an experiment or a separate causal argument. When $x = 0$ lies outside the observed range of the feature, the intercept is a value of the extended line rather than a prediction the data support.

In plain words Read the slope with its units, as a rate inside the model. Read the intercept as the height of the line at zero, a situation the data may or may not cover. A pattern of association in the data is not the same as what would happen after an intervention.

Example The fitted slope of $1/2$ hour per ten kilometers, which is 3 minutes per kilometer, says that among these routes a route 10 kilometers longer has a predicted duration half an hour longer. It does not show that adding 10 kilometers to a particular route adds half an hour, because longer routes may also differ in road type, stops or traffic. The intercept of $7/6$ hours, or 70 minutes, is the predicted duration at distance zero, and here the value $x = 0$ was observed. For the routes $(1, 2)$ , $(2, 2)$ and $(3, 5)$ , by contrast, the least-squares line is $\hat{y} = 0 + 1.5x$ . Its intercept 0 does not show that a delivery over no distance takes no time, since no route shorter than 10 kilometers was observed.

Extrapolation

Definition Using a fitted model at a feature value outside the range of the training feature values is extrapolation. For one feature the range is the interval from the smallest to the largest observed $x_i$ , and a prediction inside it is an interpolation. Inside the range the prediction rests on data on both sides, although a large gap in the observed values can still leave it poorly supported. Outside the range the prediction relies entirely on the line continuing, which the data cannot check.

In plain words Inside the observed range, the line fills in between known cases. Beyond it, the line keeps going only because the formula does, and nothing in the data says the relationship stays straight. Neither kind of prediction is guaranteed, but an extrapolation carries an extra assumption that the data cannot test.

Example The line $\hat{y} = 1.5x$ was fitted to routes of 10 to 30 kilometers, so the observed range is $1 \le x \le 3$ . At $x = 2.5$ , a 25-kilometer route, it predicts 3.75 hours, an interpolation. At $x = 5$ , a 50-kilometer route, it predicts 7.5 hours, an extrapolation. A longer route may involve a highway, a driver break or a different vehicle, and the three routes contain no information about any of that. The intercept at $x = 0$ is itself an extrapolation for these data.

Check yourself

A line fitted to constructed routes is $\hat{y} = 0.5 + 0.8x$ , with $x$ in tens of kilometers and $y$ in hours. A route of 30 kilometers took 2.5 hours. Find its fitted value and its residual, fitted minus observed, and say whether the line overpredicts or underpredicts this route.

Show the answer

The distance is $x = 3$ , so the fitted value is $0.5 + 0.8 \cdot 3 = 2.9$ hours. The residual is $2.9 - 2.5 = 0.4$ hour. It is positive, so the line overpredicts this route by 0.4 hour, or 24 minutes.

Fit the least-squares line to the constructed routes $(1, 2)$ , $(2, 3)$ , $(3, 5)$ and $(4, 6)$ , in the same units. Check the two residual sums, predict the durations at $x = 2.5$ and at $x = 10$ , and say which prediction is an extrapolation. Does the slope show what adding 10 kilometers to a given route would do?

Show the answer

The means are $\bar{x} = 2.5$ and $\bar{y} = 4$ . The centered distances are $-1.5$ , $-0.5$ , $0.5$ and $1.5$ , and the centered durations are $-2$ , $-1$ , $1$ and $2$ . So $S_{xx} = 5$ and $S_{xy} = 3 + 0.5 + 0.5 + 3 = 7$ , which give $w = 7/5 = 1.4$ and $b = 4 - 1.4 \cdot 2.5 = 0.5$ . The line is $\hat{y} = 0.5 + 1.4x$ .

The fitted values are 1.9, 3.3, 4.7 and 6.1 hours, so the residuals, fitted minus observed, are $-0.1$ , $0.3$ , $-0.3$ and $0.1$ . They add to zero, and their sum weighted by distance is $-0.1 + 0.6 - 0.9 + 0.4 = 0$ . The SSE is 0.2 squared hour.

At $x = 2.5$ the prediction is $0.5 + 1.4 \cdot 2.5 = 4$ hours, an interpolation inside the observed range from 1 to 4. At $x = 10$ , a 100-kilometer route, it is $0.5 + 14 = 14.5$ hours, an extrapolation far beyond the longest observed route of 40 kilometers.

The slope of 1.4 hours per ten kilometers describes how predicted duration differs between routes of different lengths in these data. It does not show what lengthening one particular route would do, because routes of different lengths may differ in other ways too.

Three constructed routes all have $x = 3$ , with durations of 2, 4 and 6 hours. What goes wrong with the slope formula? Find the smallest possible SSE and every line that reaches it. What can these data say about the duration at $x = 5$ ?

Show the answer

All centered distances are zero, so $S_{xx} = 0$ and the formula for $w$ divides by zero. Every line predicts the same value $b + 3w$ for all three routes, and the SSE is smallest when that value is the mean duration 4, giving $\text{SSE} = 4 + 0 + 4 = 8$ . Every line with $b + 3w = 4$ reaches this minimum, for example $\hat{y} = 4$ , $\hat{y} = 1 + x$ and $\hat{y} = 7 - x$ . These are exactly the lines through the point $(3, 4)$ .

At $x = 5$ those three lines predict 4, 6 and 2 hours, and other minimizing lines predict any other value, so the data say nothing about the duration at $x = 5$ . More routes of the same distance would not help. Only routes of a different distance can determine the slope.

Simple Linear Regression

Key terms

Simple linear regression

Fitted value and residual

Least squares

Slope and intercept

Extrapolation

Check yourself

How this lesson connects

Builds on

Used later in