Categorical Predictors and Interactions

Definition A categorical predictor is a predictor whose values are categories, such as route, region or treatment, rather than measured quantities. A categorical predictor with $k$ categories enters a regression model through indicator variables, $k - 1$ of them when the model has an intercept. Coding the categories as the numbers $1, 2, \dots, k$ and fitting one slope instead imposes an order and equal spacing, because it forces the same change in fitted mean response from each category to the next.

Definition Reference coding chooses one category as the reference category and includes, next to the intercept, an indicator for each of the other $k - 1$ categories. In a model without interactions, the coefficient of each indicator is the difference in fitted mean response between its category and the reference category at the same values of the other predictors. Changing the reference category can change the coefficients and flip their signs. In a hierarchical model, described under model hierarchy below, it does not change any least-squares fitted value, because the columns of both codings then span the same space. Without hierarchy it can, since a model with only $1$, $g$ and $xg$ forces the reference group’s line to be flat, and switching the reference changes which line that is.

Definition For a numerical predictor $x$ and an indicator $g$ that is $1$ for group B and $0$ for the reference group A, the interaction term is the product $xg$. The model

has the mean line $\beta_0 + \beta_1 x$ in group A and $(\beta_0 + \beta_2) + (\beta_1 + \beta_3)x$ in group B, so $\beta_3$ is the slope in B minus the slope in A. More generally, two predictors interact in a model when the change in mean response that goes with one of them depends on the value of the other. When each group has at least two distinct $x$ values, the least-squares lines from this model are the same as separate least-squares lines fitted in each group. The single model still estimates one residual variance from both groups together, which assumes a common $\sigma^2$.

Definition In a model with an interaction, a conditional effect is the change in the fitted mean response that goes with a change in one predictor at a stated value of the other. In the fitted model $\hat{y} = b_0 + b_1 x + b_2 g + b_3 xg$, the group difference at $x$ is

and the slope of $x$ in group $g$ is $b_1 + b_3 g$. The coefficients $b_1$ and $b_2$, often called main-effect coefficients, are conditional effects too. The coefficient $b_1$ is the slope in the reference group, and $b_2$ is the group difference at $x = 0$. Writing the model in the centered predictor $z = x - c$ turns the group coefficient into the difference at $x = c$ and changes no fitted value. Here effect means a difference between fitted values under the model. It has a causal reading only when the design or further assumptions support one, for example when the groups were formed by random assignment.

Definition A regression model respects hierarchy if, whenever it contains a product of predictors or a power of a predictor, it also contains every lower-order term built from the same predictors, as well as the intercept. A categorical predictor counts as one predictor, so all of its indicators enter each term that involves it. For the interaction $xg$ this means including $x$ and $g$, and for $x^2$ it means including $x$. In a hierarchical model, recentering a numerical predictor or changing the reference category leaves every least-squares fitted value unchanged. A model that drops a lower-order term imposes a restriction whose meaning depends on where zero is placed or on which category is the reference.