Feature Selection and Collinearity, machine learning lesson

A delivery table records each route’s distance twice, once in kilometers and once in meters. A linear model fitted to both columns predicts durations perfectly well, yet one fit gives the kilometer column a coefficient of 2 and another fit, with exactly the same predictions, gives it 0. Which number describes the effect of distance? This lesson explains why correlated features make individual coefficients unstable or impossible to pin down, how to read a coefficient correctly, and how to choose a subset of features without letting that choice leak into the evaluation.

Key terms

Collinearity

Definition Consider a linear model with an intercept and features $x_1, \dots, x_p$ fitted to $n$ rows. The features are exactly collinear when there are constants $c_0, c_1, \dots, c_p$ , not all zero, such that every row satisfies

c_0 + \sum_{j=1}^{p} c_j x_{ij} = 0 ,

which means the design matrix with its all-ones intercept column included does not have full column rank. Least squares then has infinitely many coefficient vectors that give the same fitted values. The fitted values are unique, and so are the combinations of coefficients that the fitted values depend on, but the individual coefficients are not. The features are approximately collinear when some feature is close to a linear function of the others in the data. The coefficients are then unique but unstable. If the errors are uncorrelated with a common variance, the variance of the estimated coefficient of $x_j$ is multiplied by the variance inflation factor $\operatorname{VIF}_j = 1/(1 - R_j^2)$ , compared with a feature of the same spread that is uncorrelated with the others, where $R_j^2$ is the R-squared of a least-squares regression of $x_j$ on the other features with an intercept.

In plain words Collinear columns carry overlapping information. The data show how much the shared information matters for the predictions, but not how to divide that amount among the columns. Exact collinearity leaves the division undetermined, and near collinearity leaves it poorly determined, so small changes in the data can move the coefficients a lot while predictions in the region the data cover barely change.

Example Let $x_{\text{km}}$ be the distance in kilometers and $x_{\text{m}} = 1000\,x_{\text{km}}$ the same distance in meters. The model

\begin{aligned} \hat{y} &= b + \beta_1 x_{\text{km}} + \beta_2 x_{\text{m}} \\[0.4em] &= b + (\beta_1 + 1000\beta_2)\, x_{\text{km}} \end{aligned}

depends on the two coefficients only through the effective slope $\beta_1 + 1000\beta_2$ in minutes per kilometer. On the constructed rows with distances 1, 2 and 3 km and durations 5, 7 and 9 minutes, the intercept $b = 3$ with $(\beta_1, \beta_2) = (2, 0)$ and with $(0, 0.002)$ gives the same predictions 5, 7 and 9. The first pair puts 2 minutes per kilometer on the kilometer column, and the second puts 0.002 minutes per meter on the meter column, which is the same 2 minutes per kilometer. The exact relation is $x_{\text{m}} - 1000\,x_{\text{km}} = 0$ in every row.

Feature redundancy

Definition A feature is redundant, given a set of other features, when it adds no information about the target beyond what those features already contain. The clearest case is a feature that is a fixed function of the others, such as a distance recorded twice in two units. Redundancy concerns information, while collinearity concerns linear relations among columns. A feature computed from others by a nonlinear formula, such as $x^2$ from $x$ , carries no new information, yet it is not collinear with $x$ and the intercept when $x$ takes at least three distinct values, and a linear model may need it as a separate column. Redundancy is also a property of the data at hand. Two measurements that agree in the training data can disagree in use, because of different noise, missing values or recording rules. How a method divides weight among redundant columns depends on the method. When the redundant columns are exactly collinear, least squares leaves the division open, the least-squares solution with the smallest Euclidean norm splits it equally between identical columns, and ridge regression with penalty weight $\lambda > 0$ always has a unique solution. With the intercept $b$ unpenalized, ridge minimizes

\begin{aligned} &\sum_{i=1}^{n} \bigl(y_i - b - \mathbf{w}^{\top}\mathbf{x}_i\bigr)^2 \\[0.4em] &\quad + \lambda \lVert \mathbf{w} \rVert_2^2 , \end{aligned}

and its penalty depends on the units of each column.

In plain words A redundant feature repeats what the model already knows. Including it rarely changes what the model can predict, but it changes how the model’s weight is spread across columns, and different fitting methods spread it differently. Whether a feature is truly redundant should be checked on the data the model will meet, not only on the training table.

Example Put both distance columns in kilometers, so the two columns are identical, and use the rows above. Centering gives the feature values $-1$ , $0$ and $1$ in both columns and the target values $-2$ , $0$ and $2$ . Any coefficients with $w_1 + w_2 = 2$ fit the data exactly, for example $(3, -1)$ , $(2, 0)$ and $(1, 1)$ , and the smallest-norm choice is $(1, 1)$ . Ridge with $\lambda = 2$ adds 2 to the diagonal of the centered cross-product matrix, and the normal equations become

\begin{bmatrix} 4 & 2 \\ 2 & 4 \end{bmatrix} \mathbf{w} = \begin{bmatrix} 4 \\ 4 \end{bmatrix} ,

so $w_1 = w_2 = 2/3$ and the effective slope is $4/3$ minutes per kilometer. At a new distance of 4 km the ridge prediction is $7 + \tfrac23 \cdot 2 + \tfrac23 \cdot 2 = 29/3 \approx 9.67$ minutes, while every least-squares fit predicts 11. If the second column is left in meters, the same $\lambda = 2$ puts almost all the weight on it, about 0.002 minutes per meter against about 0.000002 minutes per kilometer, and the effective slope stays at about 1.999998 minutes per kilometer, so the penalty barely shrinks anything. Features are therefore usually standardized, with means and standard deviations computed on the training data, before a penalty is applied.

Coefficient interpretation

Definition In a fitted linear model $\hat{y} = b + w_1 x_1 + \dots + w_p x_p$ , the coefficient $w_j$ is the change in the prediction when $x_j$ increases by one unit while every other feature in the model stays fixed. It is measured in target units per unit of $x_j$ . It depends on which other features are in the model, because they decide what is held fixed, and adding or removing a correlated feature can change its size and even its sign. It describes the fitted model, and it reflects a comparison the data support only when they contain cases in which $x_j$ varies while the other features stay about the same. It is not a causal effect unless the study design or a separate causal argument justifies that reading. If another feature is computed from $x_j$ , such as a duplicate or $x_j^2$ , the other features cannot stay fixed, and $w_j$ alone has no such reading.

In plain words A coefficient answers the question of what the model predicts if only this feature changes. When features move together, that question is about combinations the data rarely or never show, so the answer comes from the straight-line form of the model rather than from observed cases. A negative coefficient on a duplicated measurement shows how the weight happened to be split. It is not evidence that a longer distance shortens a delivery.

Example Four constructed deliveries have route distances 2, 4, 6 and 8 km, straight-line distances 1, 3, 3 and 5 km, and durations 8, 10, 16 and 18 minutes, built to follow $\hat{y} = 4 + 3x_1 - 2x_2$ exactly, with $x_1$ the route distance and $x_2$ the straight-line distance. Regressing duration on straight-line distance alone gives a slope of 2.5 minutes per kilometer. With route distance in the model, the coefficient of straight-line distance is $-2$ . Both numbers correctly describe their own models. The negative one says that among deliveries with the same route distance, the model predicts 2 fewer minutes for each extra kilometer of straight-line distance, as for a more direct route. Yet the two distances have correlation about 0.95, so regressing either one on the other gives $R^2 = 0.9$ and a VIF of 10, and no two of these deliveries share a route distance. The comparison the coefficient describes is never observed here, and it says nothing about what would happen if a route were changed.

Feature selection

Definition Feature selection chooses a subset $S \subseteq \{1, \dots, p\}$ of the available features for the model. Filter methods rank features by a statistic, such as the absolute correlation with the target. Wrapper methods fit the model on candidate subsets and compare their validation scores. Embedded methods make selection part of fitting, as when a lasso penalty sets some coefficients exactly to zero. Every selection step learned from data, and above all every step that looks at the targets, is part of training. In cross-validation it is refitted on each training part together with scaling, imputation and the model, so the held-out fold never influences which features are kept. The cross-validation score then estimates the performance of the whole procedure, selection followed by fitting, rather than of one fixed subset, and different folds may keep different subsets. If many subsets or settings are compared, the best validation score is optimistic, and the final estimate needs nested cross-validation or a test set that played no part in the choice. A kept feature has shown predictive usefulness within this procedure, not causal importance, and a dropped feature may still matter, for example when it was dropped only because a correlated kept feature carries the same information.

In plain words Choosing features is fitting by another name. If the choice is made on all rows before cross-validation, the held-out targets have already helped pick the features, and the scores look better than they will on new data. A smaller subset with a similar score can still be the better choice, because it is cheaper to collect and simpler to maintain.

Example To decide whether the meter column is worth keeping, compare a model with one distance column, the two-column model with ridge and $\lambda$ chosen inside each training part, and a filter that keeps the $k$ features of a larger candidate set most correlated with duration, all on the same folds, split the way the model will be used, for example by time. In each fold, fit the scaler, the selector and the model on the training part only, then score the held-out part. An exhaustive wrapper over 10 candidate features would compare $2^{10} - 1 = 1023$ nonempty subsets, and the winner’s validation score is the best of 1023 noisy numbers, so it needs an outer evaluation before it is reported. If the one-column model scores about as well as the others, dropping the redundant column simplifies the system.

Post-selection inference

Definition Post-selection inference means drawing conclusions such as p-values, confidence intervals or standard errors about a model that was chosen with the same data. The standard formulas assume that the model was fixed before the data were seen. When a search keeps the features that look strongest, the kept coefficients tend to lie farther from zero than the values they estimate, nominal p-values tend to be too small and nominal intervals too narrow, so they lose their stated error rates and coverage. Valid approaches include sample splitting, which selects on one part of the data and computes the inference on an independent part, methods designed for inference after a specified selection rule, and reporting the results as exploratory until new data confirm them.

In plain words Picking the most impressive feature and then testing it on the data that made it look impressive grades the search with its own answers. The search has to be accounted for, either by setting aside data it never touched or by using a method that knows how the choice was made.

Example Ten candidate features, none related to the target, are each tested at the 0.05 level. If the ten tests were independent, the probability that at least one comes out significant would be

1 - 0.95^{10} \approx 0.40 .

Reporting only that feature with its p-value below 0.05 makes a search result look like a finding. With sample splitting, the feature is chosen on the first half of the rows and tested on the second half. If the rows are independent and the test’s usual conditions hold, that test keeps its 0.05 level, because the second half had no part in the choice.

Check yourself

A model uses two identically scaled copies of a feature. One fit has coefficients $(4, -1)$ and another has $(1, 2)$ , with the same intercept. Do their predictions differ on rows where the two columns agree? Does the coefficient $-1$ mean that the feature lowers the target? What happens on a row where the copies read 5 and 4?

Show the answer

The effective slopes are $4 + (-1) = 3$ and $1 + 2 = 3$ . On a row where both columns equal $x$ , each fit predicts the intercept plus $3x$ , so the predictions are identical on every such row. The coefficient $-1$ only records how the first fit split the slope between two copies of the same information. It is not evidence that the feature lowers the target. Reading it as a partial effect would require changing one copy while the other stays fixed, which never happens in these rows, so it has no separate meaning.

On a row where the copies read 5 and 4, the first fit adds $4 \cdot 5 - 1 \cdot 4 = 16$ to the intercept and the second adds $1 \cdot 5 + 2 \cdot 4 = 13$ . Once the copies stop agreeing, for example through a recording error in use, the two fits make different predictions, which is one reason to check whether the redundancy holds where the model will be used.

Repeat the ridge computation for the two identical kilometer columns with $\lambda = 4$ . Find the coefficients, the effective slope and the prediction at 4 km. Does the larger penalty give a better estimate of the effect of distance?

Show the answer

Adding 4 to the diagonal of the centered cross-product matrix gives the system

\begin{bmatrix} 6 & 2 \\ 2 & 6 \end{bmatrix} \mathbf{w} = \begin{bmatrix} 4 \\ 4 \end{bmatrix} .

By symmetry $w_1 = w_2 = w$ , and each row reads $8w = 4$ , so $w = 1/2$ and the effective slope is 1 minute per kilometer. The prediction at 4 km is $7 + \tfrac12 \cdot 2 + \tfrac12 \cdot 2 = 9$ minutes, compared with $29/3 \approx 9.67$ at $\lambda = 2$ and 11 for least squares.

The larger penalty does not give a better estimate of any effect. It shrinks the slope further toward zero, and on these rows, which a slope of 2 fits exactly, the shrinkage only moves predictions away from the data. Ridge makes the coefficients unique, but uniqueness comes from the penalty, not from new information about distance. The penalty weight is a hyperparameter, chosen for prediction by validation within the development data and never on the test set.

A team has 200 rows and 50 candidate features. They compute each feature’s correlation with the target on all 200 rows, keep the 5 strongest, run 5-fold cross-validation of a linear model on those 5 features and report the mean score as the performance to expect. They also report the usual p-values of the 5 coefficients from a fit on all rows. What is wrong, and how should the comparison be set up?

Show the answer

The filter used the targets of all 200 rows, including every row that later served as a held-out fold, so the held-out targets helped choose the features. With 50 candidates, some features correlate with the target by chance, and the filter keeps such features because they fit the very targets that are later used for scoring. The cross-validation score is therefore optimistic. The p-values have the same problem, since they treat the 5 features as if they had been chosen before seeing the data.

An honest setup puts the filter inside the pipeline. For each of the 5 folds, compute the correlations on the training part only, keep the 5 strongest there, fit the model on the training part and score the held-out part. The mean score then estimates the performance of the procedure “keep the 5 strongest, then fit”, and the kept subsets may differ between folds. Compare it, on the same folds, with alternatives such as all 50 features with a ridge penalty. If the number of kept features or the method is chosen by these scores, report the result from nested cross-validation or a protected test set. If the rows have groups or a time order, the folds must respect them. For the coefficients, either select on one part of the data and estimate on an independent part, or label the p-values as exploratory.

Feature Selection and Collinearity

Key terms

Collinearity

Feature redundancy

Coefficient interpretation

Feature selection

Post-selection inference

Check yourself

How this lesson connects

Builds on

Used later in