Ridge, Lasso, and Elastic Net, machine learning lesson

Two sensors measure nearly the same quantity. Least squares can give one of them a large positive coefficient and the other a large negative one, and a slightly different training sample can change that split completely while the predictions hardly move. A penalty on the size of the coefficients removes much of this freedom. This lesson states one convention for the ridge, lasso and elastic-net penalties, works out how each one shrinks coefficients and when the lasso sets them exactly to zero, and chooses the penalty strength with validation data rather than with the training fit.

Key terms

Regularization

Definition Regularization changes the training objective so that it prefers some parameter values over others. On this page the features are standardized with the training means and standard deviations, giving a matrix $Z$ with $n$ rows and centered columns, and a linear model $\hat{y} = b + \mathbf{z}^{\top}\mathbf{w}$ is fitted by minimizing

\begin{aligned} &\frac{1}{2n} \lVert \mathbf{y} - b\mathbf{1} - Z\mathbf{w} \rVert_2^2 \\[0.4em] &\quad + \lambda P(\mathbf{w}) , \end{aligned}

where $P(\mathbf{w})$ is the ridge, lasso or elastic-net penalty defined below and the regularization strength $\lambda \ge 0$ sets its weight. The intercept $b$ is not penalized, and because the columns of $Z$ are centered, its optimal value is the training mean $\bar{y}$ for every $\lambda$ . Setting $\lambda = 0$ gives least squares. As $\lambda \to \infty$ , the coefficients go to zero and every prediction approaches $\bar{y}$ . Software differs in these constants. For example, an objective $\lVert \mathbf{y} - b\mathbf{1} - Z\mathbf{w} \rVert_2^2 + a \lVert \mathbf{w} \rVert_2^2$ has the same minimizer as the ridge objective below when $a = n\lambda$ , so the same printed number can mean different penalties.

In plain words The penalty adds a cost for large coefficients, so the fit gives up some training accuracy in exchange for smaller and more stable coefficients. A penalty is a choice about how to fit, not a finding that small coefficients are true. Because one penalty weighs the coefficients of all features, their scales matter, which is why the features are standardized first, with statistics from the training rows only.

Example A length recorded in meters is re-recorded in centimeters, so its values are multiplied by $100$ . The coefficient that gives the same predictions is divided by $100$ , so its square falls by a factor of $10000$ , and so does its ridge penalty, which then barely restrains that feature. After standardization both versions give the same column, because the standard deviation is multiplied by $100$ as well.

Ridge regression

Definition Ridge regression uses the penalty $P(\mathbf{w}) = \tfrac12 \lVert \mathbf{w} \rVert_2^2$ . With the intercept at $\bar{y}$ and $\lambda > 0$ , the minimizing coefficient vector is the unique solution of

\bigl( Z^{\top}Z + n\lambda I \bigr)\hat{\mathbf{w}} = Z^{\top}\mathbf{y} ,

since $Z^{\top}Z + n\lambda I$ is positive definite even when $Z$ does not have full column rank. Because the columns of $Z$ are centered, using $\mathbf{y}$ or the centered targets on the right gives the same result.

In plain words Ridge pulls all coefficients toward zero smoothly and does not choose between correlated features. It tends to spread the weight among them, which keeps the solution stable when features nearly repeat each other.

Example Two constructed standardized features are identical, with training rows $(-1, -1)$ and $(1, 1)$ and targets $-2$ and $2$ , so $\bar{y} = 0$ . Predictions depend only on the sum $s = w_1 + w_2$ , the data term of the objective is $\tfrac12 (s - 2)^2$ , and least squares fits perfectly with any pair whose sum is $2$ . For a given sum the ridge penalty is smallest when $w_1 = w_2 = s/2$ , which leaves

\tfrac12 (s - 2)^2 + \tfrac{\lambda}{4} s^2

to minimize. Setting the derivative $s - 2 + \lambda s/2$ to zero gives $s = 4/(2 + \lambda)$ , so each coefficient is $2/(2 + \lambda)$ . At $\lambda = 1$ both are $2/3$ . The training fit is worse than with least squares, but the coefficients are now unique.

Lasso

Definition The lasso uses the penalty $P(\mathbf{w}) = \lVert \mathbf{w} \rVert_1 = \sum_j \lvert w_j \rvert$ . Its objective is convex but not differentiable where a coefficient is zero. Write $\mathbf{z}_j$ for column $j$ of $Z$ and $\mathbf{r} = \mathbf{y} - \bar{y}\mathbf{1} - Z\mathbf{w}$ for the residual vector. A coefficient vector is a minimizer exactly when, for every $j$ , the average product $\tfrac1n \mathbf{z}_j^{\top}\mathbf{r}$ equals $\lambda \operatorname{sign}(w_j)$ if $w_j \ne 0$ and lies in $[-\lambda, \lambda]$ if $w_j = 0$ . So a coefficient can sit exactly at zero, and every coefficient is zero once $\lambda \ge \max_j \lvert \tfrac1n \mathbf{z}_j^{\top}\mathbf{y} \rvert$ . Because of the corner at zero, plain gradient descent is not a general lasso solver. Coordinate descent with soft thresholding and proximal gradient methods treat the corner exactly. When the columns of $Z$ are linearly dependent, for example with identical features or with more features than rows, the minimizer need not be unique.

In plain words A coefficient stays at zero unless its feature reduces the squared error by more than the penalty charges, so the lasso fits and selects features at the same time. With correlated features the selection is unstable. Which of two similar features is kept can depend on the sample or even on the order in which an algorithm visits them, and a zero coefficient only says that the penalized fit did not use that feature, given the others, this scaling and this $\lambda$ .

Example For the identical features with $\lambda = 0.5$ , the penalty for a given sum is smallest when both coefficients have the same sign, and then it equals $0.5\lvert s \rvert$ . The objective $\tfrac12 (s - 2)^2 + 0.5s$ for $s \ge 0$ is smallest at $s = 1.5$ , so every pair of nonnegative coefficients with sum $1.5$ is a minimizer, and all of them give the objective value $0.875$ . Coordinate descent that updates $w_1$ first returns $(1.5, 0)$ , and updating $w_2$ first returns $(0, 1.5)$ . Neither zero shows that the dropped feature is unrelated to the target, since both features have exactly the same relationship with it.

Elastic net

Definition The elastic net uses the penalty

\begin{aligned} P(\mathbf{w}) &= \alpha \lVert \mathbf{w} \rVert_1 \\[0.4em] &\quad + \tfrac{1 - \alpha}{2} \lVert \mathbf{w} \rVert_2^2 \end{aligned}

with a mixing weight $0 \le \alpha \le 1$ , so $\alpha = 1$ gives the lasso and $\alpha = 0$ gives ridge. For $\lambda > 0$ and $\alpha < 1$ the objective is strictly convex, so its minimizer is unique even with identical features. For $\alpha > 0$ it can set coefficients exactly to zero, as the lasso does, and like ridge it tends to give strongly correlated features similar coefficients.

In plain words The elastic net combines a threshold that can remove features with a smooth pull that shares weight among similar ones. It has two settings to choose, the strength $\lambda$ and the mix $\alpha$ .

Example For the identical features with $\lambda = 1$ and $\alpha = 0.5$ , the penalty for a given positive sum is smallest with equal coefficients, so the objective becomes

\tfrac12 (s - 2)^2 + \tfrac12 s + \tfrac18 s^2 .

Its derivative $1.25s - 1.5$ is zero at $s = 1.2$ , so the unique solution is $(0.6, 0.6)$ . Unlike the lasso, the elastic net does not pick one of the two identical features.

Coefficient shrinkage

Definition Coefficient shrinkage is the movement of fitted coefficients toward zero, relative to least squares, as the regularization strength grows. Take one standardized feature, with average square $1$ and average product $c$ with the centered target. The data term is $\tfrac12 w^2 - cw$ plus a constant, least squares gives $w = c$ , and the three penalized minimizers are

\begin{gathered} w_{\text{ridge}} = \frac{c}{1 + \lambda}, \\[0.4em] w_{\text{lasso}} = S_{\lambda}(c), \\[0.4em] w_{\text{elastic}} = \frac{S_{\lambda\alpha}(c)}{1 + \lambda(1 - \alpha)}, \end{gathered}

where $S_a(c) = \operatorname{sign}(c)\max(\lvert c \rvert - a, 0)$ is soft thresholding at level $a \ge 0$ . With several features, the Euclidean norm of the ridge coefficient vector never increases as $\lambda$ grows, but single coefficients need not move steadily toward zero, and with correlated features one of them can grow or change sign along the way.

In plain words Ridge divides the least-squares coefficient by $1 + \lambda$ , so it never reaches zero unless $c = 0$ . The lasso subtracts $\lambda$ from its size and stops at zero. The elastic net subtracts and then divides. These formulas are exact for a single standardized feature, and with correlated features each coefficient also depends on the others.

Example With $c = 0.3$ and $\lambda = 0.5$ , ridge gives $0.3/1.5 = 0.2$ , the lasso gives $0$ because $\lvert c \rvert \le \lambda$ , and the elastic net with $\alpha = 0.5$ gives $(0.3 - 0.25)/1.25 = 0.04$ . For two standardized features with correlation $0.9$ and average products $1$ and $0.5$ with the centered target, least squares gives about $2.89$ and $-2.11$ , while ridge with $\lambda = 1$ gives about $0.49$ and $0.03$ . The second coefficient changes sign as $\lambda$ passes $0.8$ .

Regularization path

Definition A regularization path is the collection of fitted coefficient vectors $\hat{\mathbf{w}}(\lambda)$ over a range of regularization strengths, computed on fixed training data, in practice on a grid of values. The strength is chosen by comparing the fits on validation data, or by cross-validation within the training data, with one metric on the same cases for every candidate. Training error cannot make this choice, because the unpenalized training loss of the penalized solution never decreases as $\lambda$ grows. Comparing penalized objective values across $\lambda$ is no fairer, since the objective itself changes with $\lambda$ . The test set is used once, for the procedure finally chosen.

In plain words The path shows how each coefficient moves as the penalty tightens, which coefficients reach zero and in which order, and where the fit starts to underfit. Validation error often falls at first and rises later along the path, but its shape depends on the data.

Example For the identical features, the ridge path gives each coefficient $2/(2 + \lambda)$ , which is $1$ , $0.8$ , $2/3$ , $0.5$ and $1/3$ at $\lambda = 0$ , $0.5$ , $1$ , $2$ and $4$ , where the value at $\lambda = 0$ is the limit from positive $\lambda$ , the least-squares solution of smallest norm. On the lasso path the coefficient sum is $\max(2 - \lambda, 0)$ , which reaches zero at $\lambda = 2$ , while its split between the features is not unique. Two constructed validation rows $(-0.5, -0.5)$ and $(0.5, 0.5)$ , standardized with the training statistics, have targets $-0.6$ and $0.6$ . Along the ridge path their mean squared errors are $0.16$ , $0.04$ , about $0.0044$ , $0.01$ and about $0.071$ , so $\lambda = 1$ is best among these candidates. Two validation rows are far too few for a real choice, and this ranking says nothing about other data.

Check yourself

A single standardized feature has average product $c = -1.2$ with the centered target. With $\lambda = 0.5$ , find the ridge, lasso and elastic-net coefficients, using $\alpha = 0.5$ for the elastic net. For which values of $\lambda$ is each coefficient exactly zero?

Show the answer

Ridge gives $-1.2/1.5 = -0.8$ . The lasso reduces the size $1.2$ by $0.5$ and keeps the sign, giving $-0.7$ . The elastic net thresholds at $\lambda\alpha = 0.25$ , giving $-0.95$ , and divides by $1 + 0.5 \cdot 0.5 = 1.25$ , giving $-0.76$ . The lasso coefficient is zero exactly when $\lambda \ge 1.2$ , and the elastic-net coefficient with $\alpha = 0.5$ exactly when $0.5\lambda \ge 1.2$ , that is $\lambda \ge 2.4$ . The ridge coefficient $-1.2/(1 + \lambda)$ is never zero for a finite $\lambda$ , because $c \ne 0$ .

A lasso fit on the two identical standardized features with $\lambda = 0.5$ returns $(1.5, 0)$ . A colleague concludes that the second sensor has no relationship with the target. What is wrong with this conclusion, and what would you report instead?

Show the answer

Every pair of nonnegative coefficients with sum $1.5$ minimizes the same objective, with value $0.875$ , and coordinate descent returns $(1.5, 0)$ or $(0, 1.5)$ depending only on which coefficient it updates first. The second feature is identical to the first, so its relationship with the target is exactly as strong. The zero says only that this penalized fit, with this scaling, data and $\lambda$ , did not need the second feature once the first was in the model. Report that the data cannot distinguish the two features, so the split between them is arbitrary. With real sensors that are only nearly identical, refit on resampled training data and report how often each one is selected. Also consider the elastic net, which with $\lambda = 1$ and $\alpha = 0.5$ gives the unique answer $(0.6, 0.6)$ . No penalized fit turns either coefficient into a causal effect.

For the identical-feature training data, one analyst picks the ridge strength among $\lambda = 0$ , $1$ and $2$ by the smallest training mean squared error, and another by the smallest test error. Compute the training and validation mean squared errors of the three fits, using the validation rows of the regularization path card, and explain which way of choosing is sound.

Show the answer

The coefficient sums are $s = 2$ , $4/3$ and $1$ . The training mean squared error is $(s - 2)^2$ , which gives $0$ , $4/9 \approx 0.444$ and $1$ . The validation predictions are $\pm 0.5s$ , so the validation mean squared error is $(0.5s - 0.6)^2$ , which gives $0.16$ , $1/225 \approx 0.0044$ and $0.01$ . Training error can never favor a larger $\lambda$ over a smaller one, because the training loss of the penalized solution never decreases as $\lambda$ grows, so it cannot detect overfitting. Validation error picks $\lambda = 1$ . Choosing by test error turns the test set into development data, and the test error of the chosen $\lambda$ is then optimistically biased. Choose $\lambda$ on validation data or by cross-validation within the training data, and evaluate the chosen procedure once on the test set.

Ridge, Lasso, and Elastic Net

Key terms

Regularization

Ridge regression

Lasso

Elastic net

Coefficient shrinkage

Regularization path

Check yourself

How this lesson connects

Builds on

Used later in