Bias, Variance, and Irreducible Error

Definition Fix an input $\mathbf{x}$ and let $Y$ be a new outcome at this input. Write $Y = f(\mathbf{x}) + \varepsilon$, where $f(\mathbf{x}) = E[Y \mid \mathbf{X} = \mathbf{x}]$ is the conditional mean, so the noise has $E[\varepsilon \mid \mathbf{X} = \mathbf{x}] = 0$, and assume its variance $\sigma^2(\mathbf{x}) = \operatorname{Var}(\varepsilon \mid \mathbf{X} = \mathbf{x})$ is finite. A learning procedure trained on a random training set $D$, possibly with further random choices during training, returns a predictor $\hat{f}_D$. Assume that $E[\hat{f}_D(\mathbf{x})^2]$ is finite and that the new outcome is independent of the training set and of every random choice made in training. The average fitted prediction is

where $E_D$ averages over training sets and training randomness. At the fixed input abbreviate $f = f(\mathbf{x})$, $m = m(\mathbf{x})$, $\sigma^2 = \sigma^2(\mathbf{x})$ and $\hat{f} = \hat{f}_D(\mathbf{x})$. Then the expected squared prediction error, averaged over the training randomness and the new outcome, is

which is squared bias plus prediction variance plus irreducible error. To derive it, write the error as $Y - \hat{f} = (f - m) + (m - \hat{f}) + \varepsilon$ and square. The first part is a constant at this input, the second has expectation $0$ by the definition of $m$, and the noise has mean $0$ and is independent of $\hat{f}$, so all three cross terms have expectation $0$. If the input $\mathbf{X}$ of a new case is drawn independently of the training data, averaging the three terms over its distribution gives the integrated decomposition of the expected squared error, provided these expectations are finite. The identity belongs to squared loss. For class labels under zero-one loss, bias and variance are commonly defined through the most frequent prediction, and those terms do not add up to the error rate. Extra variability can even lower the error there. If $P(Y = 1 \mid \mathbf{x}) = 0.8$, a classifier that always predicts $0$ errs with probability $0.8$. One that predicts $1$ for $40$ percent of training draws still predicts the wrong label $0$ most often, yet it errs with probability $0.6 \cdot 0.8 + 0.4 \cdot 0.2 = 0.56$.

Definition Under the setup of the decomposition, the bias of a learning procedure at the input $\mathbf{x}$ is

the average fitted prediction minus the conditional mean, and the squared bias is its square. A positive and a negative bias of the same size give the same squared bias. Bias belongs to the whole experiment, the procedure together with the data-generating process, the training-set size and the training randomness. It is not a property of one fitted model, which gives a single prediction at $\mathbf{x}$ and no average. It usually changes with the input, so a procedure can be unbiased at some inputs and biased at others, and the integrated squared bias averages it over the input distribution. Computing it requires $f$, which is known in simulations and constructed examples but unknown for real data.

Definition With the abbreviations of the decomposition, the prediction variance of a procedure at the input $\mathbf{x}$ is

the expected squared distance of a fitted prediction from the average fitted prediction, with the expectation over the training set and all training randomness. It is the variance of a fully specified probability experiment, so when finitely many equally likely predictions are possible, the average of their squared deviations divides by their number and not by one less. It measures how sensitive the fitted prediction is to the particular training sample and to random choices in training. It contains no noise from the new outcome. It equals $0$ exactly when the prediction at $\mathbf{x}$ is the same for every training draw, apart from events of probability $0$.

Definition The irreducible error at the input $\mathbf{x}$ is the conditional variance of the outcome. With the two expectations below taken conditionally on $\mathbf{X} = \mathbf{x}$, it is

For every number $a$, the expected squared error of the constant prediction $a$ at this input is

and only $a = f(\mathbf{x})$ reaches $\sigma^2(\mathbf{x})$. The same holds for a random prediction, such as a fitted model’s, that is independent of the new outcome. Its expected squared error at this input is $\sigma^2(\mathbf{x})$ plus the expected squared distance of the prediction from $f(\mathbf{x})$, so no such prediction gets below $\sigma^2(\mathbf{x})$. The floor is irreducible relative to the information in $\mathbf{x}$ and to squared loss. It does not depend on the learning procedure, so procedures compared at the same input under the same outcome process differ only in squared bias plus variance. A further feature $Z$ known at prediction time changes the conditional distribution, and by the law of total variance $E\bigl[\operatorname{Var}(Y \mid \mathbf{X}, Z) \mid \mathbf{X} = \mathbf{x}\bigr] \le \sigma^2(\mathbf{x})$, so new information can lower the floor on average and can never raise it on average.

Definition The bias-variance tradeoff is the frequently observed tendency, within a family of models indexed by a complexity setting, for more flexible settings to lower squared bias and raise prediction variance, and for restrictions such as a larger ridge penalty, fewer features or more neighbors in nearest-neighbor regression to do the opposite, so that expected squared error is often smallest at an intermediate setting. It is a tendency and not a theorem. Neither term has to move monotonically with the number of parameters or any other complexity count, and the error curve need not be U-shaped. The amount of data, the optimization method, the architecture, the regularization and the training randomness all change both terms. One procedure can have both smaller bias and smaller variance than another, so lowering the error does not always require giving up one term for the other. The decomposition behind the picture is the squared-loss identity under the conditions stated above.

Definition Training randomness is every source of chance that makes the fitted predictor random. It includes the draw of the training set $D$ itself and random choices inside training, such as initial weights, the order of mini-batches, random subsamples of rows or features and random fold assignments inside a tuning search. The expectation $E_D$ in the decomposition averages over all the sources that are allowed to vary, so a statement of bias or variance has to say which sources were varied and which were held fixed, for example by recording a seed. A single trained model on a single data set is one draw from this experiment, and it reveals neither $m(\mathbf{x})$ nor the variance. Estimating them needs repeated training on fresh samples from a process whose $f$ is known, as in a simulation, or further assumptions. Rerunning training with new seeds on one fixed data set measures only the part of the variance that the seeds cause.