Neural Regularization and Early Stopping, machine learning lesson

A network’s training loss can keep falling long after its loss on held-out cases has started to rise. Which of the parameter vectors met along the way should be kept, and would a weight penalty or dropout have done better? This lesson defines weight decay and its decoupled form for adaptive optimizers, explains dropout, separates a network’s training mode from its evaluation mode, and turns early stopping into a declared rule for selecting a checkpoint by validation loss. A constructed example then shows what a fair comparison of these choices can and cannot establish.

Key terms

Weight decay

Definition Let $L(\boldsymbol{\theta})$ be the mean training loss of a network with parameters $\boldsymbol{\theta}$ , and let $\mathcal{P}$ be the set of penalized weights. On this page $\mathcal{P}$ contains every entry of the weight matrices and of the output weights, and the biases are left unpenalized by a stated choice. An L2 penalty with strength $\lambda \ge 0$ trains on the objective

\begin{aligned} J(\boldsymbol{\theta}) &= L(\boldsymbol{\theta}) \\[0.4em] &\quad + \frac{\lambda}{2} \sum_{j \in \mathcal{P}} w_j^2 . \end{aligned}

For a penalized weight $w$ with data gradient $g = \partial L / \partial w$ , the partial derivative of $J$ is $g + \lambda w$ , so one plain gradient descent step with learning rate $\eta > 0$ is

\begin{aligned} w &\leftarrow w - \eta (g + \lambda w) \\[0.4em] &= (1 - \eta\lambda) w - \eta g . \end{aligned}

Weight decay is the rule on the second line. It multiplies each penalized weight by the factor $1 - \eta\lambda$ and then takes the ordinary gradient step, and when $0 < \eta\lambda < 1$ the factor lies strictly between $0$ and $1$ . For plain gradient descent, and for stochastic gradient descent without momentum, the L2 penalty and weight decay with this factor produce exactly the same updates, even when the learning rate changes from step to step. Setting $\lambda = 0$ gives unregularized training. What a weight penalty prefers depends on how the inputs are scaled, so rescaling a feature changes which fits the penalty favors.

In plain words Each step pulls every penalized weight a fixed fraction of the way toward zero, and the data gradient then pushes it where the training data want it. A weight stays large only if the data keep pushing it there. The penalty is the one of ridge regression, now applied to the weights of a network. Smaller weights are a preference written into the objective, not proof of a better model, so the strength $\lambda$ is chosen on validation data like any other setting.

Example Take a weight $w = 2$ with data gradient $g = 0.3$ , learning rate $\eta = 0.1$ and $\lambda = 0.2$ . The shrink factor is $1 - 0.1 \cdot 0.2 = 0.98$ , so the new weight is $0.98 \cdot 2 - 0.1 \cdot 0.3 = 1.96 - 0.03 = 1.93$ . Without decay the step gives $2 - 0.03 = 1.97$ . The difference $0.04$ equals $\eta\lambda w$ , so the pull toward zero is proportional to the current size of the weight.

Decoupled weight decay

Definition Adam keeps, for each weight, running averages of its gradient $g_t$ and of the squared gradient, starting from $m_0 = v_0 = 0$ ,

\begin{gathered} m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \\[0.4em] v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2, \end{gathered}

with $0 \le \beta_1 < 1$ and $0 \le \beta_2 < 1$ . It divides them by $1 - \beta_1^t$ and $1 - \beta_2^t$ to get $\hat{m}_t$ and $\hat{v}_t$ , and it moves the weight by $\eta \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$ with a small $\epsilon > 0$ . An L2 penalty enters Adam through the gradient, since $g_t$ is replaced by $g_t + \lambda w_{t-1}$ in both averages, so the pull of the penalty is divided by the same coordinate-wise scale as the data gradient. Decoupled weight decay, the rule of AdamW, computes the averages from the data gradient alone and shrinks the weight in a separate term,

\begin{aligned} w_t &= w_{t-1} - \eta \, \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} \\[0.4em] &\quad - \eta\lambda w_{t-1} . \end{aligned}

For Adam and other adaptive optimizers, and also for gradient descent with momentum, the coupled penalty and the decoupled decay generally produce different updates, and a value of $\lambda$ tuned for one does not carry over to the other. Libraries differ in whether the decay coefficient is multiplied by the learning rate, and an option named weight decay may implement either rule, so a report names the optimizer, the regularization rule actually applied and its value.

In plain words Adam divides each weight’s step by a running measure of that weight’s own gradient size, so weights with large and small gradients move by steps of similar size. A penalty added to the gradient is divided along with it, which means that weights with a history of large gradients are decayed less than the others. Decoupled decay shrinks every penalized weight by the same fraction $\eta\lambda$ per step, whatever its gradient history.

Example Take the weight $w = 2$ with data gradient $0.3$ , $\eta = 0.1$ and $\lambda = 0.2$ again, now as the first step of Adam. At $t = 1$ the corrections give $\hat{m}_1 = g_1$ and $\hat{v}_1 = g_1^2$ , so the step is $\eta g_1 / (\lvert g_1 \rvert + \epsilon)$ , almost exactly $\eta$ times the sign of $g_1$ . Without any penalty the new weight is $1.9$ . With the L2 penalty the gradient becomes $0.3 + 0.2 \cdot 2 = 0.7$ , which has the same sign, so the new weight is again $1.9$ , up to a difference below $10^{-8}$ , and the penalty has had no visible effect. AdamW gives about $2 - 0.1 - 0.1 \cdot 0.2 \cdot 2 = 1.86$ , while plain gradient descent with either form gave $1.93$ .

Dropout

Definition Dropout acts on the activations $h_j$ of a layer during training, with a dropout probability $p$ such that $0 \le p < 1$ and keep probability $q = 1 - p$ . In every training forward pass it draws independent masks $m_j \sim \operatorname{Bernoulli}(q)$ , a fresh one for each unit of each example, and inverted dropout passes on

\begin{gathered} \tilde{h}_j = \frac{m_j h_j}{q}, \\[0.4em] E[\tilde{h}_j \mid h_j] = h_j . \end{gathered}

The backward pass uses the same mask, so no gradient flows back through a dropped unit and the gradient through a kept unit is multiplied by $1/q$ . At evaluation, inverted dropout is switched off and passes $h_j$ unchanged. The original form of dropout skips the division by $q$ in training and multiplies by $q$ at evaluation instead, and the two scale conventions must not be mixed. The equality of expectations holds unit by unit. It does not make the average network output over random masks equal to the output of the evaluation network, because the computation after the dropout layer is in general nonlinear.

In plain words Every training step uses a randomly thinned network, so no hidden unit can count on a particular partner being present. Dividing by the keep probability keeps each unit’s average output at its full value, so the network used for evaluation needs no rescaling. Dropout changes how training computes, not which network is evaluated.

Example A layer with keep probability $q = 0.75$ produces the activations $2$ and $-1$ , and the training pass draws the mask $(1, 0)$ . The outputs are $2/0.75 = 8/3$ and $0$ . Over many masks the first unit gives $8/3$ with probability $0.75$ and $0$ otherwise, which averages to $2$ . At evaluation the outputs are simply $2$ and $-1$ . Averages do not pass through a nonlinearity, though. Suppose the logit is $z = \tilde{h}$ for a single unit with activation $2$ and keep probability $0.5$ . The logit is $0$ or $4$ with equal probability and averages to the evaluation logit $2$ , but the predicted probabilities $\sigma(0) = 0.5$ and $\sigma(4) \approx 0.982$ average to about $0.741$ , while the evaluation network gives $\sigma(2) \approx 0.881$ .

Training and evaluation modes

Definition Layers such as dropout and batch normalization compute differently in a network’s two modes. In training mode, dropout draws random masks, and batch normalization standardizes each feature with the mean and the variance of the current mini-batch and updates stored running averages of these statistics. In evaluation mode, dropout passes its input unchanged, and batch normalization standardizes with the stored running averages, so the output for one example does not depend on which other examples share its batch, and nothing is updated. Validation losses, checkpoint selection, test results and deployed predictions all use evaluation mode. The mode is a separate switch from gradient tracking. Turning off gradient computation leaves dropout active, and forward passes in training mode can update the batch normalization statistics even when no gradient is computed.

In plain words The mode tells each layer whether it is being trained or used. A model scored in training mode is a random, batch-dependent variant of the model that will actually be used, and running validation data through batch normalization in training mode even writes validation statistics into the model.

Example A batch normalization unit with scale $1$ , shift $0$ and a negligible $\epsilon$ receives the value $3$ . In training mode, in a batch with the values $1$ and $3$ , it uses the batch mean $2$ and the batch variance $1$ , computed with divisor $2$ , and outputs $(3 - 2)/1 = 1$ . In a batch with the values $3$ and $5$ the mean is $4$ , the variance is again $1$ , and the same value $3$ comes out as $-1$ . In evaluation mode with stored running mean $2$ and running variance $1$ , it outputs $1$ in either batch. The dropout layer of the previous card likewise gives $8/3$ and $0$ for one draw in training mode and always $2$ and $-1$ in evaluation mode. The constructed comparison on this page draws masks only inside its training updates and computes every reported loss with a separate deterministic prediction function.

Validation loss

Definition For parameters $\boldsymbol{\theta}$ and $m$ validation examples $(\mathbf{x}'_j, y'_j)$ that took no part in fitting, the validation loss is the mean loss

L_{\text{val}}(\boldsymbol{\theta}) = \frac{1}{m} \sum_{j=1}^{m} \ell(y'_j, z'_j),

where $z'_j$ is the logit the network computes for $\mathbf{x}'_j$ in evaluation mode. On this page $\ell$ is the binary cross-entropy written from the logit, $\ell(y, z) = \ln(1 + e^{z}) - yz$ , measured in nats per example, and no penalty term is included. For a fixed $\boldsymbol{\theta}$ chosen without these examples, if they are independent draws from the population of interest, the validation loss is an unbiased estimate of the population risk of that network. Once it has been used to choose among checkpoints or candidates, the smallest value is an optimistically biased estimate for the one chosen. Comparisons across epochs or candidates need the same examples, the same loss with the same reduction, such as the mean over examples, and the same mode. The training loss reported next to it on this page is computed the same way on the training examples, which differs from the objective the optimizer minimizes when a penalty or dropout is in use.

In plain words Validation loss scores the current parameters on cases the optimizer never saw. Training loss and validation loss answer different questions, and they can move in opposite directions. Cross-entropy also registers changes in confidence that accuracy ignores, so validation loss can rise while validation accuracy stays the same.

Example The constructed comparison on this page has $20$ training cases and $80$ validation cases, with two features drawn uniformly from the square $[-1, 1]^2$ and the label $1$ when $x_1 x_2 \ge 0$ . Three training labels were deliberately flipped from $1$ to $0$ to model label errors, and the validation labels follow the unflipped rule. A network with $32$ tanh hidden units and a logistic output is trained by full-batch gradient descent with $\eta = 0.1$ for $1000$ epochs, one gradient step per epoch, from a fixed seed. Without regularization the validation loss starts at $0.7062$ , reaches its minimum $0.3901$ at epoch $789$ , where the training loss is $0.2296$ , and rises to $0.4002$ by epoch $1000$ , while the training loss falls to $0.1903$ . The validation accuracy is $64/80 = 0.8$ at both epochs, so accuracy alone would not show the change. The comparison also trains the same network with an L2 penalty of strength $\lambda = 0.03$ on its weights and, separately, with dropout probability $0.25$ on its hidden layer. For the weight decay candidate, the training loss at its selected checkpoint, epoch $1000$ , is $0.6284$ and the penalty adds $0.0471$ , so its training objective there is $0.6755$ . Plotting that objective next to the unpenalized training losses of other candidates would compare two different quantities.

Early stopping

Definition Early stopping evaluates the validation loss at declared checks during training, saves a copy of the parameters whenever the loss improves, and returns the saved checkpoint instead of the final parameters. Checkpoint selection is the rule that names the returned checkpoint. For a set $C$ of checked epochs, write $\boldsymbol{\theta}_t$ for the parameters after epoch $t$ and $V_t = L_{\text{val}}(\boldsymbol{\theta}_t)$ for their validation loss. This page selects the earliest epoch with the smallest validation loss,

\begin{gathered} V^{*} = \min_{s \in C} V_s, \\[0.4em] \hat{t} = \min \{ t \in C : V_t = V^{*} \} . \end{gathered}

A patience rule with patience $P \ge 1$ and minimum improvement $\delta \ge 0$ counts a check as an improvement when its validation loss is below the best value so far by more than $\delta$ . It stops training after $P$ consecutive checks without improvement or when the budget of epochs runs out, whichever comes first, and returns the last checkpoint that counted as an improvement. With $\delta = 0$ that is the earliest minimum among the checks actually made. The check frequency, $P$ , $\delta$ , the tie rule and the budget are declared before training. The saved checkpoint must be a copy of the parameters, not a reference to arrays the optimizer keeps changing, and resuming training from it also needs the optimizer state, such as momentum or Adam averages and the step count, the position in the learning rate schedule and the random number generator states. The stopping epoch is chosen on validation data, so it is a hyperparameter, and the chosen procedure is assessed on a protected test set.

In plain words Training traces a path through parameter space, and early stopping keeps the point on that path that predicted the validation cases best. The training loss usually keeps falling after that point as the network fits details of its own sample that do not carry over. Patience trades computation against the risk of quitting on a temporary plateau.

Example On the full recorded curve of the unregularized run, the selected checkpoint is epoch $789$ , and keeping the final parameters of epoch $1000$ instead would cost $0.4002 - 0.3901 = 0.0101$ nats of validation loss per case. A patience rule with $\delta = 0$ behaves very differently depending on $P$ . The validation loss dips to $0.6906$ at epoch $13$ and first goes below that value again at epoch $137$ . A patience of $100$ therefore stops training at epoch $113$ and returns epoch $13$ , a network with validation accuracy $29/80 \approx 0.36$ , worse than the accuracy $44/80 = 0.55$ of always predicting class $0$ , the more common training class. A patience of $150$ stops at epoch $939$ and returns epoch $789$ . Whether a flat stretch is temporary is known only in hindsight, which is one reason to declare a generous patience or to record the full curve when computation allows.

Check yourself

A run records validation losses $0.61$ , $0.47$ , $0.44$ , $0.44$ , $0.46$ and $0.45$ at epochs $1$ to $6$ , and its training loss falls at every epoch. Which checkpoint does the earliest-minimum rule select on the full curve? With patience $P = 2$ , $\delta = 0$ and a check after every epoch, when does training stop and which checkpoint is returned? Is the falling training loss a reason to prefer epoch $6$ ?

Show the answer

The smallest validation loss, $0.44$ , occurs at epochs $3$ and $4$ , and the earliest-minimum rule breaks the tie in favor of epoch $3$ .

Under the patience rule, epochs $1$ , $2$ and $3$ each improve on the best value so far. At epoch $4$ the loss $0.44$ is not below the best value $0.44$ , so it is the first check without improvement, and epoch $5$ , with $0.46$ , is the second. Training stops after epoch $5$ and returns epoch $3$ . Epoch $6$ is never computed, so its loss of $0.45$ plays no part.

The training loss is not a reason to prefer epoch $6$ . It measures the fit to the cases the optimizer used, and it keeps falling while the validation evidence says the predictions for new cases have stopped improving.

A layer uses inverted dropout with dropout probability $0.2$ . In one training pass it receives the activations $1.2$ , $-0.4$ and $0.5$ and draws the mask $(1, 1, 0)$ . Find its outputs in that pass and in evaluation mode. Then explain what is wrong with two shortcuts, computing the validation loss with the model left in training mode, and multiplying the activations by $0.8$ at evaluation.

Show the answer

The keep probability is $q = 0.8$ . In the training pass the kept activations are divided by $0.8$ and the dropped one becomes $0$ , so the outputs are $1.5$ , $-0.5$ and $0$ . In evaluation mode the layer passes $1.2$ , $-0.4$ and $0.5$ unchanged.

With the model left in training mode, every validation pass draws new masks, so the validation loss becomes a random number that describes a thinned network instead of the one that will be used, and checkpoint selection partly follows mask noise. If the network also has batch normalization, its outputs then depend on which cases share a batch, and the stored running statistics are updated with validation data.

Multiplying by $0.8$ at evaluation scales twice. Inverted dropout already divided by $q$ during training, so the evaluation network should use the activations as they are. Multiplying by $q$ belongs to the original form of dropout, which does not divide during training. Mixing the two conventions feeds the next layer activations at $80$ percent of the scale it saw on average during training.

The three candidates of this page use the same data, network, seed, learning rate and budget of $1000$ epochs, and each keeps its own best validation checkpoint. All losses are mean cross-entropy in evaluation mode without the penalty. A colleague concludes that regularization does not help this network. What does the evidence support, and what should be checked next?

Candidate	Selected epoch	Training loss	Validation loss
No regularizer	$789$	$0.2296$	$0.3901$
Weight decay $\lambda = 0.03$	$1000$	$0.6284$	$0.6575$
Dropout $p = 0.25$	$998$	$0.2951$	$0.3961$

Show the answer

The weight decay candidate has high training and validation losses, both close to the value $\ln 2 \approx 0.693$ of a score that always says $0.5$ , so within this budget it underfits. Its selected epoch is the last one, which means its validation loss was still falling when the budget ran out.

The dropout candidate trails the unregularized one by only $0.0061$ nats per case. On these $80$ validation cases the standard error of the mean per-case difference between the two selected networks is about $0.016$ , more than twice the difference, and both checkpoints were chosen on these same cases. The data do not separate the two. Its selected epoch $998$ also sits at the end of the budget.

A checkpoint at the edge of the budget is a warning that the budget, not the stopping rule, ended the search. An additional run of the same three candidates with a budget of $3000$ epochs, everything else unchanged, shows the effect. The unregularized network still selects epoch $789$ at $0.3901$ , and its validation loss climbs to $0.6018$ by epoch $3000$ . The dropout network reaches $0.3665$ at epoch $1689$ , and the weight decay network is still improving at epoch $3000$ , at $0.6475$ . So the evidence supports a narrow statement. Under this seed and budget, these particular settings did not lower the selected validation loss, and with $\lambda = 0.03$ the network underfits within both budgets. It does not support a general verdict on regularization.

A sound next step declares one budget for all candidates in advance, long enough that every selected epoch falls well inside it, gives each technique a comparable search over its strength, repeats the runs over several seeds and compares on validation data. Choosing the budget after seeing these results is itself a selection, so the final claim comes from one evaluation of the chosen procedure on a protected test set.

Neural Regularization and Early Stopping

Key terms

Weight decay

Decoupled weight decay

Dropout

Training and evaluation modes

Validation loss

Early stopping

Check yourself

How this lesson connects

Builds on

Used later in