Regression Assumptions, Residuals and Influence

Definition Classical inference for a line uses the model $Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$, with unknown coefficients $\beta_0$ and $\beta_1$ and random errors $\varepsilon_i$. The linear conditional mean assumption says that $E(\varepsilon_i \mid x_i) = 0$, so that the mean response at each predictor value lies on a straight line,

for every $x$ in the range studied. A residual plot shows the residuals $e_i = y_i - \hat{y}_i$ of the fitted line on the vertical axis against the predictor values $x_i$ or against the fitted values $\hat{y}_i$, with a horizontal reference line at zero. When the conditional mean is not linear, the residuals tend to follow a systematic pattern, such as a curve, instead of scattering around zero with no trend across the plot.

Definition The constant variance assumption says that the errors have the same variance at every predictor value, $\operatorname{Var}(\varepsilon_i \mid x_i) = \sigma^2$. Heteroscedasticity, also spelled heteroskedasticity, means that this conditional variance changes with $x$. A common sign is a fan shape in a residual plot, with the vertical spread of the residuals growing or shrinking along the horizontal axis. When the linear conditional mean holds, heteroscedasticity does not bias the least-squares coefficients, but the usual standard errors, tests and confidence intervals, which use a single $\sigma$, can be wrong. Prediction intervals built on one common $\sigma$ tend to be too wide where the spread is small and too narrow where it is large.

Definition The errors are independent when, given the predictor values, the errors of some observations carry no information about the errors of the others. Dependence is common when several observations come from the same person, device or cluster, or when observations are taken close together in time or space. Independence is judged mainly from the design, meaning how the units were selected or assigned and how the measurements were made. A residual plot in the order of data collection can reveal runs of residuals with the same sign, or cycles, that a plot against $x$ may hide. When errors are positively correlated, the usual standard errors are often too small, so intervals are too narrow and tests reject too often.

Definition Exact small-sample t and F inference for a line also assumes normal errors, so that given the predictor values the $\varepsilon_i$ are independent with distribution $N(0, \sigma^2)$. This condition concerns the distribution of the errors around the conditional mean, not the distribution of the predictor values or of all the response values taken together. A normal Q-Q plot, short for quantile-quantile plot, shows the ordered residuals $e_{(1)} \le \dots \le e_{(n)}$ against standard normal quantiles at matching probabilities, for example the values $z_i$ with $P(Z \le z_i) = (i - 0.5)/n$. If the errors are normal, the points tend to lie near a straight line, and curvature at one or both ends suggests skewness or tails heavier or lighter than normal. Residuals are not the errors. A fit with an intercept forces them to sum to zero, and when the errors are independent with constant variance, the residual variances $\sigma^2(1 - h_i)$ generally differ from one observation to another, with $h_i$ the hat value defined below. So the plot is approximate evidence, and standardized residuals are often plotted instead.

Definition For a least-squares line with an intercept and at least two distinct predictor values, the hat value of observation $i$ is

also called the leverage of the observation. It depends only on the predictor values, not on the responses. Each fitted value is a weighted sum of all the responses, $\hat{y}_i = \sum_j h_{ij} y_j$ with $h_{ij} = 1/n + (x_i - \bar{x})(x_j - \bar{x})/S_{xx}$, and $h_i = h_{ii}$ is the weight of $y_i$ in its own fitted value. The hat values satisfy $1/n \le h_i \le 1$ and add up to 2, the number of coefficients, so their average is $2/n$. An observation whose hat value is large compared with that average is called a high-leverage point.

Definition Consider a least-squares fit with $p$ coefficients, so $p = 2$ for a line with an intercept, and residual mean square $s^2 = \text{SSE}/(n - p) > 0$. For an observation with $h_i < 1$, Cook’s distance is

where $\hat{y}_{j(i)}$ is the fitted value for observation $j$ from the fit with observation $i$ left out. It measures how much all the fitted values change together when that one observation is removed. It can be computed without refitting from

which combines the size of the residual with the hat value. A common rule of thumb treats values near or above 1 as large, and a value far above the others deserves a look at any size.