Least Squares and R-Squared

Definition Regression data consist of $n$ pairs $(x_1, y_1), \dots, (x_n, y_n)$ measured on the same individuals. The response $y$ is the variable to be described or predicted, and the predictor $x$, also called the explanatory variable, is the variable used to describe or predict it. Simple linear regression summarizes how the response changes across values of one predictor by a straight line $\hat{y} = b_0 + b_1 x$.

Definition For a line $\hat{y} = b_0 + b_1 x$ and observation $i$, the fitted value $\hat{y}_i = b_0 + b_1 x_i$ is the height of the line at that observation’s predictor value. The residual of observation $i$ is

the observed response minus the fitted value, in the units of the response. It is positive when the point lies above the line, negative when it lies below and zero when it lies on the line. A residual is a vertical distance taken at the fixed value $x_i$, not the perpendicular distance from the point to the line.

Definition For a line with intercept $b_0$ and slope $b_1$, the residual sum of squares is

where $e_i = y_i - b_0 - b_1 x_i$. It is measured in squared response units. Squaring keeps positive and negative residuals from canceling, and it makes a large miss count much more than a small one.

Definition For $n$ pairs with at least two distinct predictor values, the least-squares line is the line $\hat{y} = b_0 + b_1 x$ whose intercept and slope make the SSE as small as possible. It is unique, and its coefficients are

where

Equivalently $b_1 = s_{xy}/s_x^2$, which equals $r s_y / s_x$ when $s_y > 0$. The line passes through the point $(\bar{x}, \bar{y})$, and its residuals satisfy $\sum e_i = 0$ and $\sum x_i e_i = 0$. These are the two equations obtained by setting the partial derivatives of the SSE with respect to $b_0$ and $b_1$ equal to zero. If all $x_i$ are equal, then $S_{xx} = 0$, every line through $(\bar{x}, \bar{y})$ gives the same smallest SSE, and the slope is not determined.

Definition In the line $\hat{y} = b_0 + b_1 x$, the slope $b_1$ is the change in the fitted value when $x$ increases by one unit, measured in response units per predictor unit. The intercept $b_0$ is the fitted value at $x = 0$, measured in response units. For a least-squares line both are summaries of the data. The slope compares the fitted values of individuals whose predictor values differ by one unit, and without a suitable design it should not be read as the effect of changing $x$ for any one individual. When $x = 0$ lies outside the observed range of the predictor, the intercept comes from extending the line beyond the data and does not describe any observed situation. When $x = 0$ is not a possible value, the intercept describes no real situation at all.

Definition The total sum of squares $\text{SST} = \sum (y_i - \bar{y})^2$ is the SSE of the horizontal line $\hat{y} = \bar{y}$, which ignores the predictor. For a least-squares line with an intercept, fitted to data with $\text{SST} > 0$,

For such a fit $\text{SST} = \sum (\hat{y}_i - \bar{y})^2 + \text{SSE}$, so $0 \le R^2 \le 1$, and $R^2$ is the fraction of the total sum of squares accounted for by the fitted values. With at least two distinct predictor values, $R^2 = r^2$, the square of the Pearson correlation. When all responses are equal, $\text{SST} = 0$ and the ratio is undefined.