Estimation, Bias, Precision and Mean Squared Error, statistics lesson

Two methods estimate the same quantity, whose true value is 10. Method A returns 8 or 12, each half of the time, so it is centered on the target but never hits it. Method B returns 10 or 12, each half of the time, so it sometimes hits the target but is centered one unit too high. Which is better? The word “accurate” hides several different questions, and this lesson separates them into bias, variance, mean squared error and consistency, using these two constructed methods as the running example.

Key terms

Estimator

Definition An estimator of a parameter $\theta$ is a rule that computes a number from the sample data, $\hat\theta = g(X_1, \dots, X_n)$ , without using the unknown value of $\theta$ . Before the data are collected the observations are random, so the estimator is a random variable with a sampling distribution. An estimate is the number $g(x_1, \dots, x_n)$ that the rule gives for the data actually observed, and it is fixed once the data are in. Bias, variance and mean squared error are properties of the estimator, and consistency is a property of the estimators for growing sample sizes, all computed under a stated sampling model. They describe how the rule behaves over repeated samples, not how close one particular estimate is to $\theta$ .

In plain words The estimator is the method, and the estimate is the number the method produced for your data. A method is judged by how it performs over many repetitions, which one result cannot show.

Example The sample mean $\bar X = (X_1 + \dots + X_n)/n$ is an estimator of the population mean $\mu$ . For the observed data 3, 5 and 10, its estimate is $\bar x = 6$ . The sample median is another estimator, and its estimate for these data is 5. If the population is symmetric about $\mu$ , then $\mu$ is also a population median, so the two estimators aim at the same center and can be compared. Methods A and B from the introduction are hypothetical estimators of $\theta = 10$ whose sampling distributions are given directly, so their properties can be computed exactly.

Bias

Definition When $E(\hat\theta)$ exists, the bias of an estimator $\hat\theta$ of $\theta$ is

\operatorname{Bias}(\hat\theta) = E(\hat\theta) - \theta,

with the expectation taken under the stated model. It generally depends on the true value of $\theta$ . The estimator is unbiased if its bias is $0$ for every value of $\theta$ that the model allows. Under independent random sampling from a population with mean $\mu$ , $E(\bar X) = \mu$ , so the sample mean is unbiased for $\mu$ . If the population variance $\sigma^2$ is finite and $n \ge 2$ , the sample variance $S^2$ with denominator $n - 1$ , viewed as an estimator, is unbiased for $\sigma^2$ , while the version with denominator $n$ has expectation $(n - 1)\sigma^2/n$ and bias $-\sigma^2/n$ . Bias depends on the model. A formula that is unbiased under random sampling can be biased for a convenience sample or for readings from a miscalibrated instrument.

In plain words Bias is the systematic offset, the signed gap between where the estimator lands on average and the target. Unbiased does not mean accurate in each sample. It means that the errors balance out on average over repeated samples.

Example Method A takes the values 8 and 12 with probability $\tfrac12$ each, so $E(A) = 10$ and its bias is $0$ . Yet neither possible estimate equals 10, and every estimate misses by 2. Method B takes the values 10 and 12 with probability $\tfrac12$ each, so $E(B) = 11$ and its bias is $11 - 10 = 1$ .

Variance of an estimator

Definition The variance of an estimator measures the spread of its sampling distribution around its own expected value,

\begin{aligned} &\operatorname{Var}(\hat\theta) \\[0.4em] &\quad = E\Bigl[\bigl(\hat\theta - E(\hat\theta)\bigr)^2\Bigr], \end{aligned}

not around the target $\theta$ . Its square root is the standard error $\operatorname{SE}(\hat\theta)$ . An estimator with smaller variance is more precise. Precision says nothing about where the distribution is centered, so a precise estimator can be concentrated around the wrong value. For independent observations with variance $\sigma^2$ , the sample mean has $\operatorname{Var}(\bar X) = \sigma^2/n$ .

In plain words The variance measures how much the estimate varies from sample to sample. Precision is about repeatability, not about hitting the target.

Example The deviations of A from its own mean 10 are $-2$ and $2$ , so $\operatorname{Var}(A) = 4$ and $\operatorname{SE}(A) = 2$ . The deviations of B from its own mean 11 are $-1$ and $1$ , so $\operatorname{Var}(B) = 1$ and $\operatorname{SE}(B) = 1$ . Method B is more precise than A and is also biased.

Mean squared error

Definition When $E(\hat\theta^2)$ is finite, the mean squared error of $\hat\theta$ is

\operatorname{MSE}(\hat\theta) = E\bigl[(\hat\theta - \theta)^2\bigr].

It is the expected loss under squared-error loss, which charges $e^2$ for an error $e$ of either sign. Squaring keeps positive and negative errors from canceling and weighs large errors more heavily than small ones. Like the bias, the MSE generally depends on the true value of $\theta$ . Under this loss, an estimator whose MSE is smaller at every allowed value of $\theta$ is preferred, while two estimators can each have the smaller MSE for different values of $\theta$ . Other losses, such as absolute error with expected loss $E\lvert \hat\theta - \theta \rvert$ , can rank the same estimators differently, so the loss should be stated whenever estimators are compared.

In plain words The MSE is the average squared miss, measured from the target rather than from the estimator’s own center. It puts being off-center and being spread out into one number.

Example The errors of A are $-2$ and $2$ , and the errors of B are $0$ and $2$ , each with probability $\tfrac12$ , so

\begin{gathered} \operatorname{MSE}(A) = \tfrac12 (4 + 4) = 4, \\[0.4em] \operatorname{MSE}(B) = \tfrac12 (0 + 4) = 2. \end{gathered}

Under squared-error loss B is better, despite its bias. The loss can matter. An estimator G whose error is $0$ with probability $0.9$ and $10$ with probability $0.1$ has MSE $10$ and expected absolute error $1$ . An estimator H whose error is always $2$ has MSE $4$ and expected absolute error $2$ . Squared error prefers H, and absolute error prefers G.

Bias-variance decomposition

Definition If $E(\hat\theta^2)$ is finite, then

\begin{aligned} \operatorname{MSE}(\hat\theta) &= \operatorname{Var}(\hat\theta) \\[0.4em] &\quad + \operatorname{Bias}(\hat\theta)^2 . \end{aligned}

To see why, split the error into a fluctuating part and a fixed part,

\begin{aligned} \hat\theta - \theta &= \bigl(\hat\theta - E(\hat\theta)\bigr) \\[0.4em] &\quad + \bigl(E(\hat\theta) - \theta\bigr). \end{aligned}

Square and take expectations. The first part gives the variance, and the second part is the constant bias, which gives the squared bias. The cross term is twice the bias times $E\bigl(\hat\theta - E(\hat\theta)\bigr)$ , which is $0$ .

In plain words The expected squared miss is the spread around the estimator’s own center plus the squared distance from that center to the target. A biased estimator can have the smaller MSE when it removes more variance than it adds in squared bias. The identity does not say that lowering one term always raises the other. That depends on the estimators being compared.

Example For A the decomposition gives $4 + 0^2 = 4$ , and for B it gives $1 + 1^2 = 2$ , matching the direct calculations. A standard case behaves the same way. For $n \ge 2$ independent observations from a normal population, $S^2$ is unbiased for $\sigma^2$ with variance $2\sigma^4/(n - 1)$ , and the version with denominator $n$ , $\hat\sigma^2_n = (n - 1)S^2/n$ , has bias $-\sigma^2/n$ and variance $2(n - 1)\sigma^4/n^2$ . With $n = 5$ and $\sigma^2 = 1$ their MSEs are

\begin{aligned} \operatorname{MSE}(S^2) &= \tfrac{2}{4} = 0.5, \\[0.4em] \operatorname{MSE}(\hat\sigma^2_n) &= \tfrac{8}{25} + \tfrac{1}{25} \\[0.4em] &= 0.36 . \end{aligned}

The biased version has the smaller MSE for normal data. The denominator $n - 1$ is chosen because it makes $S^2$ unbiased, and the example shows that unbiasedness and MSE are different criteria that can disagree.

Consistency

Definition Consistency is a property of a sequence of estimators $T_1, T_2, \dots$ , one for each sample size $n$ , as the sample grows. The sequence is consistent for $\theta$ if for every $\varepsilon > 0$

\lim_{n \to \infty} P\bigl(\lvert T_n - \theta \rvert > \varepsilon\bigr) = 0

for every value of $\theta$ that the model allows. In other words, $T_n$ converges in probability to $\theta$ . A single sampling distribution at one $n$ cannot show consistency. A sufficient condition is that $\operatorname{MSE}(T_n) \to 0$ for every allowed value of $\theta$ , because $P(\lvert T_n - \theta \rvert \ge \varepsilon) \le \operatorname{MSE}(T_n)/\varepsilon^2$ , but it is not necessary. Neither consistency nor unbiasedness implies the other.

In plain words A consistent estimator is, with probability approaching 1, within any fixed tolerance of the target once the sample is large enough. It is a promise about large samples and says nothing about how good the estimate is at the sample size you have.

Example For independent, identically distributed observations with mean $\mu$ and finite variance $\sigma^2$ , the sample mean has $\operatorname{MSE}(\bar X_n) = \sigma^2/n \to 0$ , so it is consistent for $\mu$ . The estimator $\bar X_n + 1$ has MSE $\sigma^2/n + 1$ , which tends to $1$ , and it concentrates around $\mu + 1$ , so it is not consistent. The estimator $\bar X_n + 1/n$ is biased at every $n$ , with bias $1/n$ , yet its MSE $\sigma^2/n + 1/n^2$ tends to $0$ , so it is consistent.

Check yourself

Estimator C has bias $-2$ and variance $0.5$ . Estimator D is unbiased and has variance $3$ . Which has the smaller MSE, and which is more precise?

Show the answer

By the bias-variance decomposition,

\begin{aligned} \operatorname{MSE}(C) &= 0.5 + (-2)^2 \\[0.4em] &= 4.5, \\[0.4em] \operatorname{MSE}(D) &= 3 + 0^2 = 3. \end{aligned}

D has the smaller MSE. C has the smaller variance, so it is more precise, but its squared bias of 4 outweighs that advantage under squared-error loss. Here the unbiased estimator wins, while for methods A and B the biased one won, so unbiasedness alone cannot rank estimators.

An estimator of $\theta = 5$ takes the values 4, 5 and 9 with probabilities $0.5$ , $0.25$ and $0.25$ . Find its bias, variance and MSE, and check that the decomposition holds.

Show the answer

The expected value is $0.5 \cdot 4 + 0.25 \cdot 5 + 0.25 \cdot 9 = 5.5$ , so the bias is $0.5$ . The deviations from $5.5$ are $-1.5$ , $-0.5$ and $3.5$ , and the variance is

\begin{aligned} &0.5 \cdot 2.25 + 0.25 \cdot 0.25 \\[0.4em] &\quad + 0.25 \cdot 12.25 \\[0.4em] &= 4.25 . \end{aligned}

The errors from the target 5 are $-1$ , $0$ and $4$ , so the MSE is $0.5 \cdot 1 + 0.25 \cdot 0 + 0.25 \cdot 16 = 4.5$ . The decomposition agrees, since $4.25 + 0.5^2 = 4.5$ .

Observations $X_1, X_2, \dots$ are independent with mean $\mu$ and variance $4$ . Compare the estimators $T_n = X_1$ , which ignores every observation after the first, and $W_n = \bar X_n + 1/n$ . Which is unbiased, and which is consistent?

Show the answer

The estimator $T_n$ has $E(T_n) = \mu$ for every $n$ , so it is unbiased. Its distribution is the distribution of $X_1$ whatever the value of $n$ . Because the variance $4$ is positive, $X_1$ is not constant, so there is some $\varepsilon > 0$ with $P(\lvert X_1 - \mu \rvert > \varepsilon) > 0$ , and this probability does not shrink as $n$ grows. So $T_n$ is not consistent.

The estimator $W_n$ has $E(W_n) = \mu + 1/n$ , so it is biased at every $n$ . Its MSE is $4/n + 1/n^2$ , for example $0.0401$ at $n = 100$ , and it tends to $0$ , so $W_n$ is consistent. Unbiasedness and consistency answer different questions, and neither implies the other.

Estimation, Bias, Precision and Mean Squared Error

Key terms

Estimator

Bias

Variance of an estimator

Mean squared error

Bias-variance decomposition

Consistency

Check yourself

How this lesson connects

Builds on

Used later in