Polynomial Features and Interactions

Definition Feature engineering is the construction of the columns of the design matrix from the raw inputs. A feature map $\boldsymbol{\phi}$ sends each raw input $\mathbf{x}$ to a feature vector $\boldsymbol{\phi}(\mathbf{x})$ in $\mathbb{R}^p$, and the model is then fitted on the rows $\boldsymbol{\phi}(\mathbf{x}_i)$. Every quantity the map takes from data, such as means and scales, a category vocabulary or knot locations, is part of the fitted procedure. It is learned from the training rows only, inside each training fold when cross-validation is used, saved, and applied unchanged to validation, test and future rows. Construction choices such as the polynomial degree or which interactions to include are made with validation data, never with the test set. Fixed arithmetic with constants chosen in advance needs no data, and every raw input that the map uses has to be available at prediction time.

Definition Basis functions are fixed functions $\phi_0, \phi_1, \dots, \phi_{p-1}$ of the raw input, and a basis expansion model predicts

Such a model is linear in parameters, meaning that for every fixed input the prediction is a linear function of the coefficient vector $\boldsymbol{\beta}$. Its dependence on the raw input can be strongly nonlinear. Let $\Phi$ be the design matrix with entries $\Phi_{ij} = \phi_j(\mathbf{x}_i)$. Fitting by squared error is then ordinary least squares on $\Phi$, its objective is a convex quadratic in $\boldsymbol{\beta}$, and the coefficients are unique exactly when $\Phi$ has full column rank. A model such as $f(x) = \beta_0 + e^{\beta_1 x}$ is not linear in parameters.

Definition A polynomial feature is a power of an input, such as $u^2$ or $u^3$, or a product of powers of several inputs. A polynomial model of degree $d$ in one input uses the basis $1, u, \dots, u^d$. With $q$ raw inputs, all products of powers with total degree at most $d$ give $\binom{q + d}{d}$ columns including the constant, so the count grows quickly, to $66$ columns for $q = 10$ and $d = 2$. The coefficient of a single power is not a slope. The rate of change is a derivative that combines several coefficients, for example $\beta_1 + 2\beta_2 u$ for the model $\beta_0 + \beta_1 u + \beta_2 u^2$.

Definition An interaction term is the product of two features, such as $uS$. In a model containing $u$, $S$ and $uS$, the change in prediction that goes with one of them depends on the value of the other. For the demand model

and the summer minus winter difference at the same temperature is $8 + 5u$. Without the product, the difference between seasons would be the same at every temperature. A model with an interaction usually also keeps both features of the product, because dropping one imposes a restriction whose meaning depends on where the zero of the other is placed. An interaction describes how predictions vary, and it has a causal reading only when the design of the study or further assumptions support one.

Definition An indicator variable equals $1$ when a condition holds and $0$ otherwise. A categorical input with $K$ categories enters a model with an intercept through $K - 1$ indicators, and the omitted category is the reference. Adding an indicator for the reference category as well makes the $K$ indicators sum to the column of ones, so the design matrix loses full column rank. The fitted values stay unique, but the coefficients do not. Dropping the intercept and keeping all $K$ indicators is the other valid coding. The vocabulary of categories is fixed during training and saved in order. A category that never appeared in training needs an explicit policy, such as raising an error or routing it to a designated category for rare or unknown values that already exists in training, because silently setting all indicators to $0$ treats it as the reference category.

Definition A spline in one input is a function built from polynomial pieces on the intervals between chosen points $\xi_1 < \dots < \xi_K$, called knots, with the pieces joined so that the function, and possibly some of its derivatives, is continuous at each knot. With the hinge functions $(u - \xi_k)_+ = \max(0, u - \xi_k)$, every continuous piecewise linear function with these knots can be written as

whose slope changes by $\gamma_k$ at the knot $\xi_k$. Cubic splines use pieces of degree three whose values and first two derivatives agree at each knot. Once the knots are fixed, a spline is linear in parameters and is fitted by least squares like any other basis. Knots placed using data, for example at quantiles of the input, are learned from the training rows only.