Multiple Regression and Adjusted Associations

Definition A multiple linear regression model describes one numerical response $Y$ with several predictors $x_1, \dots, x_p$,

where the error $\varepsilon_i$ has mean $0$ given the predictors. Least squares chooses $b_0, \dots, b_p$ to minimize $\text{SSE} = \sum (y_i - \hat{y}_i)^2$. The design has $p + 1$ columns, the column of ones and one column for each predictor. If none of them is a linear combination of the others, the minimizer is unique. The residuals $e_i = y_i - \hat{y}_i$ are orthogonal to every column, so $\sum e_i = 0$ and $\sum x_{ij} e_i = 0$ for each predictor. When the minimizer is unique and $n > p + 1$, the residual degrees of freedom are $n - p - 1$ and the residual variance is estimated by

Standard errors, tests and intervals need the same error conditions as in simple regression, namely independent errors with constant variance and, for exact t and F procedures, normal errors. With large samples, tests and intervals for the coefficients remain approximately valid without normal errors, but a prediction interval for one new observation does not.

Definition In a multiple regression, the coefficient $b_j$ of a predictor $x_j$ is called a partial regression coefficient. When $x_j$ appears in no other term of the model, such as an interaction or a power, $b_j$ is the change in the fitted response when $x_j$ increases by one unit while every other predictor in the model stays fixed, measured in response units per unit of $x_j$. Equivalently, $b_j$ is the least-squares slope of $y$ on the residuals from regressing $x_j$ on the other columns of the design, including the column of ones. So it uses only the part of $x_j$ that the other predictors do not explain linearly, and its value depends on which other predictors are in the model.

Definition The association between the response and a predictor adjusted for other predictors is the comparison of fitted mean responses between units that differ in that predictor but have the same values of the other predictors in the model. For a least-squares fit of $\hat{y} = b_0 + b_x x + b_g g$ with a group indicator $g$, the residuals sum to $0$ within each group, so the raw difference between the group mean responses splits exactly as

The adjusted difference $b_g$ is conditional on the predictors in the model and on its form, here a straight line in $x$ with the same slope in both groups. The data support it directly only where the groups overlap in $x$.

Definition An indicator variable for a category equals $1$ for units in that category and $0$ for all other units. To include a categorical predictor with $k$ categories in a model with an intercept, choose one category as the reference group and include $k - 1$ indicators, one for each other category. Units in the reference group have every indicator equal to $0$. In a model with no interactions involving the indicators, each indicator’s coefficient is the difference in fitted mean response between its category and the reference group at equal values of the other predictors. Including all $k$ indicators together with the intercept makes the design columns linearly dependent, because the indicators add up to the column of ones, and then the coefficients are not unique. Dropping the intercept and keeping all $k$ indicators is another valid coding, in which each indicator’s coefficient is that group’s own intercept.

Definition An omitted variable is a variable left out of a regression model that is related to the response and to one or more of the included predictors. Leaving it out can change the coefficients of those predictors. For least-squares fits with an intercept on the same data, let the long model contain an extra predictor $z$ with coefficient $b_z$, and let $\delta_x$ be the coefficient of an included predictor $x$ in the least-squares regression of $z$ on the included predictors. Then the coefficient of $x$ in the short model without $z$ is

so the coefficient changes exactly when both $b_z$ and $\delta_x$ are nonzero. An omitted common cause of a predictor and the response is a confounder of that comparison.