Variance, introduction to probability lesson

Two service desks both keep a caller waiting two minutes on average. At one desk the wait is exactly two minutes every time, while at the other a call is sometimes answered at once and sometimes after four minutes. Knowing only the mean, you cannot tell which desk you are calling. This lesson defines variance, the expected squared distance from the mean, computes it in two ways that must agree, and explains why it is measured in squared units and when it is finite.

Key terms

Deviation from the mean

Definition Let $X$ have finite expectation $\mu = E[X]$ . The deviation of $X$ from its mean is the random variable $X - \mu$ . Its value is positive on outcomes where $X$ lies above the mean, negative where $X$ lies below it and zero where $X = \mu$ . By linearity,

E[X - \mu] = E[X] - \mu = 0.

In plain words A deviation says how far one outcome lies from the center and on which side. The signed deviations of any variable with a finite mean balance exactly, so their average cannot measure spread. A measure of spread built by averaging deviations has to remove the signs first.

Example Let a waiting time $X$ be $0$ , $2$ or $4$ minutes with probabilities $\frac14$ , $\frac12$ and $\frac14$ . Its mean is $0 \cdot \frac14 + 2 \cdot \frac12 + 4 \cdot \frac14 = 2$ minutes, and the deviations are $-2$ , $0$ and $2$ minutes. Their weighted average is $(-2) \cdot \frac14 + 0 \cdot \frac12 + 2 \cdot \frac14 = 0$ , although half of the probability sits away from the mean. The sign describes position relative to the mean, so a deviation of $-2$ minutes is not a negative waiting time.

Variance

Definition Let $X$ have finite expectation $\mu$ . The squared deviation $(X - \mu)^2$ is a nonnegative random variable, and the variance of $X$ is its expectation,

\operatorname{Var}(X) = E\bigl[(X - \mu)^2\bigr].

For a discrete variable with PMF $p_X$ ,

\operatorname{Var}(X) = \sum_x (x - \mu)^2 \, p_X(x).

The variance is either a number in $[0, \infty)$ or $+\infty$ , and it is finite exactly when $X$ has a finite second moment. It equals $0$ exactly when $P(X = \mu) = 1$ , so for a discrete variable exactly when every value with positive probability equals the mean. Variance is a property of a probability model. Estimating an unknown variance from data, with a divisor such as $n - 1$ , is a different task.

In plain words Square each deviation, so that distances above and below the mean count alike and cannot cancel, then average the squares with the probabilities as weights. Squaring also makes large deviations count heavily, since doubling a distance multiplies its square by four.

Example For the waiting time above, the squared deviations are $4$ , $0$ and $4$ , so $\operatorname{Var}(X) = 4 \cdot \frac14 + 0 \cdot \frac12 + 4 \cdot \frac14 = 2$ square minutes. A wait of exactly $2$ minutes has the same mean and variance $0$ . A wait of $0$ or $4$ minutes with probability $\frac12$ each also has mean $2$ , but both of its squared deviations are $4$ , so its variance is $4$ square minutes. Moving probability from the center to the ends, with the mean unchanged, raised the variance from $2$ to $4$ .

Second moment

Definition The second moment of $X$ is $E[X^2]$ . For a discrete variable,

E[X^2] = \sum_x x^2 \, p_X(x).

All its terms are nonnegative, so it is either a finite number or $+\infty$ , and $X$ has a finite second moment when $E[X^2] < \infty$ . A finite second moment implies a finite expectation, because $\lvert x \rvert \le \frac{1 + x^2}{2}$ for every real $x$ . For $X$ with finite expectation $\mu$ , the inequalities $(x - \mu)^2 \le 2x^2 + 2\mu^2$ and $x^2 \le 2(x - \mu)^2 + 2\mu^2$ show that the variance is finite exactly when the second moment is finite, and the variance is $+\infty$ when $E[X^2] = \infty$ . If $X$ has no finite expectation, there is no real number $\mu$ to center at, and the definition of variance does not apply. Every variable with finitely many values, and more generally every bounded variable, has a finite second moment.

In plain words The second moment averages the squared outcomes themselves, measured from zero rather than from the mean. A distribution with unbounded values can have a finite mean and an infinite second moment, because squaring makes rare large values count heavily enough for the sum to diverge.

Example For the waiting time above, the squared outcomes are $0$ , $4$ and $16$ square minutes, so $E[X^2] = 0 \cdot \frac14 + 4 \cdot \frac12 + 16 \cdot \frac14 = 6$ square minutes. For an infinite second moment, let $Y$ take the value $2^k$ with probability $3 \cdot 4^{-k}$ for $k = 1, 2, 3, \dots$ . The probabilities $\frac34, \frac{3}{16}, \frac{3}{64}, \dots$ add to $1$ , and

\begin{aligned} E[Y] &= \sum_{k=1}^{\infty} 2^k \cdot 3 \cdot 4^{-k} = 3, \\[0.4em] E[Y^2] &= \sum_{k=1}^{\infty} 4^k \cdot 3 \cdot 4^{-k} = \infty . \end{aligned}

The first sum is $\sum_k 3 \cdot 2^{-k} = 3$ , while every term of the second equals $3$ . So $Y$ has a finite mean and $\operatorname{Var}(Y) = +\infty$ .

Computational formula for variance

Definition If $X$ has a finite second moment and $\mu = E[X]$ , then

\operatorname{Var}(X) = E[X^2] - \bigl(E[X]\bigr)^2 .

To see why, expand the square and apply linearity, which is allowed because $X^2$ , $X$ and the constant $\mu^2$ all have finite expectations,

\begin{aligned} &E\bigl[(X - \mu)^2\bigr] \\[0.4em] &\quad = E[X^2] - 2\mu E[X] + \mu^2 \\[0.4em] &\quad = E[X^2] - \mu^2 . \end{aligned}

Because a variance is never negative, it follows that $E[X^2] \ge (E[X])^2$ , with equality exactly when $X$ is constant with probability one.

In plain words The variance is the mean of the squares minus the square of the mean. This route is often quicker by hand, since it avoids subtracting the mean from every value. When the two terms are large and nearly equal, subtracting rounded decimals can lose much of the accuracy, and exact fractions or the centered definition are safer.

Example For the waiting time above, $E[X^2] = 6$ and $(E[X])^2 = 4$ square minutes, so $\operatorname{Var}(X) = 6 - 4 = 2$ square minutes, which matches the calculation from squared deviations.

Squared units

Definition If $X$ is measured in some unit, the deviation $X - \mu$ has the same unit, and $\operatorname{Var}(X)$ is measured in the square of that unit, since probabilities carry no unit. The standard deviation $\operatorname{SD}(X) = \sqrt{\operatorname{Var}(X)}$ returns to the original unit. It is the root-mean-square deviation, the square root of the mean of the squared deviations. If $X$ has finite variance and $a$ and $b$ are constants, then

\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X),

because the deviation of $aX + b$ from its mean $a\mu + b$ is $a(X - \mu)$ . A shift leaves the variance unchanged, and a factor $a$ multiplies it by $a^2$ .

In plain words A variance of $2$ square minutes is not an average distance of $2$ minutes. To compare the spread with the mean or with the outcomes themselves, take the square root. Changing units from minutes to seconds multiplies every deviation by $60$ and the variance by $3600$ .

Example The waiting time above has variance $2$ square minutes and standard deviation $\sqrt{2} \approx 1.41$ minutes. Adding a fixed $5$ minutes to every wait gives $X + 5$ , whose mean is $7$ minutes and whose variance is still $2$ square minutes. Tripling every wait gives $3X$ , with variance $9 \cdot 2 = 18$ square minutes and standard deviation $3\sqrt{2} \approx 4.24$ minutes.

Check yourself

A waiting time is $1$ , $2$ or $6$ minutes with probabilities $0.2$ , $0.5$ and $0.3$ . Find its mean, then its variance, once from the squared deviations and once from the second moment. Give units.

Show the answer

The mean is $1 \cdot 0.2 + 2 \cdot 0.5 + 6 \cdot 0.3 = 0.2 + 1 + 1.8 = 3$ minutes. The deviations are $-2$ , $-1$ and $3$ minutes, with squares $4$ , $1$ and $9$ , so the variance is $4 \cdot 0.2 + 1 \cdot 0.5 + 9 \cdot 0.3 = 0.8 + 0.5 + 2.7 = 4$ square minutes.

The second moment is $1 \cdot 0.2 + 4 \cdot 0.5 + 36 \cdot 0.3 = 0.2 + 2 + 10.8 = 13$ square minutes, and $13 - 3^2 = 4$ square minutes again. The standard deviation is $2$ minutes.

A waiting time has variance $4$ square minutes. Someone converts the times to seconds and reports a variance of $240$ . What went wrong, and what are the correct variance and standard deviation in seconds?

Show the answer

If $X$ is the waiting time in minutes, the time in seconds is $60X$ . Every deviation is multiplied by $60$ , so every squared deviation, and with them the variance, is multiplied by $60^2 = 3600$ . The correct variance is $3600 \cdot 4 = 14400$ square seconds. The reported value multiplies by $60$ only once, as if the variance were measured in minutes. The standard deviation is $60 \cdot 2 = 120$ seconds, in agreement with $\sqrt{14400} = 120$ .

Can a random variable have mean $2$ and second moment $3$ ? Can it have mean $2$ and second moment $5$ ? Give an example where one exists.

Show the answer

With mean $2$ and second moment $3$ , the computational formula would give a variance of $3 - 2^2 = -1$ . A variance is the expectation of a nonnegative variable, so it cannot be negative, and no such random variable exists.

With mean $2$ and second moment $5$ , the variance would be $5 - 4 = 1$ , which is possible. For example, let $X$ be $1$ or $3$ with probability $\frac12$ each. Then $E[X] = 2$ and $E[X^2] = \frac{1 + 9}{2} = 5$ .

Variance

Key terms

Deviation from the mean

Variance

Second moment

Computational formula for variance

Squared units

Check yourself

How this lesson connects

Builds on

Used later in