Inference for a Regression Slope

Definition The simple linear regression model says that the responses at predictor values $x_1, \dots, x_n$ satisfy

for $i = 1, \dots, n$, where $\beta_0$ and $\beta_1$ are unknown parameters and the $\varepsilon_i$ are random errors. Exact classical inference adds four conditions, all given the predictor values. The errors have mean zero, so that the mean response is the line $E(Y \mid x) = \beta_0 + \beta_1 x$. They are independent. They share one variance $\sigma^2$. And they are normally distributed. Together these say that the $\varepsilon_i$ are independent with distribution $N(0, \sigma^2)$. The predictor values are treated as fixed, or the analysis is carried out conditional on them, and at least two of them must differ. The slope $\beta_1$ is the change in the mean response per unit increase in $x$.

Definition In the simple linear regression model, the error term of observation $i$ is

the deviation of the response from the unknown population mean line. It cannot be observed, because $\beta_0$ and $\beta_1$ are unknown. The residual $e_i = y_i - (b_0 + b_1 x_i)$ is the deviation from the fitted line and serves as an estimate of the error. The two differ. The residuals of a fit with an intercept satisfy $\sum e_i = 0$ and $\sum x_i e_i = 0$, while the errors need not sum to zero. Under the model each residual has variance $\sigma^2(1 - h_i)$, where $h_i$ is its hat value, and this is smaller than the error variance $\sigma^2$.

Definition For a least-squares line fitted to $n \ge 3$ observations, the residual standard deviation is

where SSE is the residual sum of squares. The divisor $n - 2$ is the number of residual degrees of freedom. The $n$ residuals must satisfy the two equations $\sum e_i = 0$ and $\sum x_i e_i = 0$, one for each estimated coefficient. When the conditional mean is linear and the errors are independent with common variance $\sigma^2$, the residual variance $s^2$ is an unbiased estimator of $\sigma^2$, whether or not the errors are normal. The value $s$ is in response units and estimates $\sigma$. It is not the sample standard deviation $s_y$ of the responses.

Definition Under the simple linear regression model with independent errors of common variance $\sigma^2$, the least-squares slope $b_1$ has mean $\beta_1$ and standard deviation $\sigma/\sqrt{S_{xx}}$ over repeated samples at the same predictor values, where $S_{xx} = \sum (x_i - \bar{x})^2$. Replacing $\sigma$ by the residual standard deviation $s$ gives the standard error of the slope,

It has the units of the slope, response units per predictor unit.

Definition To test $H_0\colon \beta_1 = \beta_{1,0}$, most often with $\beta_{1,0} = 0$, compute

If $H_0$ and the conditions of the simple linear regression model hold, the statistic has the t distribution with $n - 2$ degrees of freedom, written $t_{n-2}$. For the two-sided alternative $H_1\colon \beta_1 \ne \beta_{1,0}$ the p-value is $2P(T \ge \lvert t \rvert)$ with $T \sim t_{n-2}$. It is the probability, computed assuming $H_0$ and the model, of a t statistic at least as far from zero as the one observed. For $\beta_{1,0} = 0$ the statistic equals $r\sqrt{n - 2}/\sqrt{1 - r^2}$ when $\lvert r \rvert < 1$, so for pairs sampled at random from a bivariate normal population the same statistic tests whether the population correlation is zero.

Definition Under the same conditions, the confidence interval for $\beta_1$ with level $1 - \alpha$ is

where $t_{1-\alpha/2,\, n-2}$ is the $1 - \alpha/2$ quantile of $t_{n-2}$. Over repeated samples at the same predictor values, the procedure captures $\beta_1$ in a proportion $1 - \alpha$ of the samples. The interval leaves out a value $\beta_{1,0}$ exactly when the two-sided t test of $H_0\colon \beta_1 = \beta_{1,0}$ gives $p < \alpha$. Its endpoints are in response units per predictor unit.