Variance and Standard Deviation, statistics lesson

Two delivery services both take 30 minutes on average. One arrives within a minute or two of that time on almost every order, while the other is sometimes a quarter of an hour early and sometimes a quarter of an hour late. The averages agree, yet anyone planning around a delivery cares a great deal about the difference. This lesson builds a measure of spread from the distances to the mean, explains why a complete population and a sample use different divisors, and returns the result to the original units with the standard deviation.

Key terms

Deviation

Definition Let $x_1, \dots, x_n$ be numerical observations with mean $\bar{x}$ . The deviation of the $i$ th observation is $x_i - \bar{x}$ , the observation minus the mean, in the same units as the data. When the values form a complete population with mean $\mu$ , the deviations are $x_i - \mu$ . Deviations from the mean always sum to zero, because

\begin{aligned} \sum_{i=1}^{n} (x_i - \bar{x}) &= \sum_{i=1}^{n} x_i - n\bar{x} \\[0.4em] &= 0 . \end{aligned}

In plain words A deviation says how far one observation sits from the center and on which side, positive above the mean and negative below it. The positive and negative deviations always balance, so their average is zero for every dataset, a tightly clustered one and a widely spread one alike. To measure spread, the signs have to be removed somehow.

Example Four constructed delivery times are $2$ , $4$ , $6$ and $8$ minutes, with mean $\bar{x} = 5$ minutes. Their deviations are $-3$ , $-1$ , $1$ and $3$ minutes, which add to $0$ . Four times that all equal $5$ minutes have the same mean, and every one of their deviations is $0$ . The means cannot tell the two sets apart, but the deviations can.

Population variance

Definition Let $x_1, \dots, x_N$ be all $N$ values of a complete finite population, with population mean $\mu$ . The population variance is the average squared deviation,

\sigma^2 = \frac{1}{N} \sum_{i=1}^{N} (x_i - \mu)^2 .

It is measured in the square of the data’s unit. It is never negative, and it equals $0$ exactly when all $N$ values are equal.

In plain words Square every deviation so that each one counts as a nonnegative amount, then average the squares over the whole population. Squaring gives large deviations much more weight than small ones. Dividing by $N$ is right when the values in hand are the entire group you want to describe, rather than a sample from something larger.

Example Suppose the times $2$ , $4$ , $6$ and $8$ minutes are every delivery a courier made on one day, and that day is all we want to describe. The squared deviations are $9$ , $1$ , $1$ and $9$ , with total $20$ , so

\sigma^2 = \frac{20}{4} = 5 \,\text{min}^2 .

The unit is square minutes, which is not a length of time anyone waits. For four times that all equal $5$ minutes, the population variance is $0$ .

Sample variance

Definition Let $x_1, \dots, x_n$ be a sample of $n \ge 2$ observations with sample mean $\bar{x}$ . The sample variance is

s^2 = \frac{1}{n - 1} \sum_{i=1}^{n} (x_i - \bar{x})^2 ,

in squared units. With a single observation the formula gives $0/0$ , and the sample variance is not defined. If the observations are independent draws from a population or process with finite variance $\sigma^2$ , then

E(s^2) = \sigma^2 ,

so $s^2$ is an unbiased estimator of $\sigma^2$ . Dividing the same sum by $n$ instead gives an estimator with expected value $\frac{n-1}{n}\sigma^2$ , which falls short of $\sigma^2$ on average whenever $\sigma^2 > 0$ .

In plain words The sample mean comes from the same data, and no other center gives a smaller sum of squared deviations for those data. Measuring from it therefore never gives a larger sum than measuring from the unknown population mean would, and it typically gives a smaller one. Dividing by $n - 1$ instead of $n$ makes up for this exactly on average. The correction concerns chance variation between random samples. It does not repair a sample collected in a biased way, and the guarantee assumes independent observations.

Example Treat the times $2$ , $4$ and $6$ minutes as independent observations of a courier’s delivery process. The mean is $4$ minutes, the deviations are $-2$ , $0$ and $2$ , and the squared deviations total $8$ . The sample variance is $s^2 = 8/2 = 4$ square minutes, so the sample standard deviation is $s = 2$ minutes. If the same three times were instead the complete population of interest, the population variance would be $8/3 \approx 2.67$ square minutes. The numerator is the same, and the divisor depends on what the number is meant to describe.

Standard deviation

Definition A standard deviation is the nonnegative square root of a variance. The population standard deviation is $\sigma = \sqrt{\sigma^2}$ , and the sample standard deviation is

\begin{aligned} s &= \sqrt{s^2} \\[0.4em] &= \sqrt{\frac{1}{n - 1} \sum_{i=1}^{n} (x_i - \bar{x})^2} . \end{aligned}

It has the same units as the data. Adding a constant to every observation leaves it unchanged, and multiplying every observation by a constant $b$ multiplies it by $\lvert b \rvert$ and the variance by $b^2$ . Although $s^2$ is unbiased for $\sigma^2$ under independent sampling, $s$ is not an unbiased estimator of $\sigma$ in general. Because the square root is a concave function, $E(s) \le \sqrt{E(s^2)} = \sigma$ .

In plain words The standard deviation puts spread back into the units of the data and can be read as a typical size of a deviation, in a root-mean-square sense. It is not the average absolute distance from the mean, which is never larger and usually smaller. On its own it also does not say what share of the data lies within one standard deviation of the mean. The familiar figure of about 68 percent belongs to normal distributions, and for other data the share can be quite different.

Example Treated as a complete population, the four times $2$ , $4$ , $6$ and $8$ minutes have $\sigma = \sqrt{5} \approx 2.24$ minutes. Treated as a sample, they give $s^2 = 20/3$ square minutes and $s = \sqrt{20/3} \approx 2.58$ minutes. The average absolute distance from the mean is $(3 + 1 + 1 + 3)/4 = 2$ minutes, smaller than both. The interval $5 \pm 2.24$ minutes contains the times $4$ and $6$ but not $2$ or $8$ , so exactly half of these data lie within one standard deviation of the mean.

Degrees of freedom

Definition Degrees of freedom count the values that remain free to vary after the constraints created by estimated quantities. The $n$ deviations $x_i - \bar{x}$ satisfy one linear constraint, since they sum to zero, so any $n - 1$ of them determine the last one. The sum of squared deviations about $\bar{x}$ therefore has $n - 1$ degrees of freedom, and the sample variance divides by that number. The count depends on the constraints. A straight line fitted with an intercept and a slope, for example, leaves $n - 2$ degrees of freedom for its residuals. Behind the divisor $n - 1$ is an identity that holds for every number $c$ ,

\begin{aligned} \sum_{i=1}^{n} (x_i - c)^2 &= \sum_{i=1}^{n} (x_i - \bar{x})^2 \\[0.4em] &\quad + n(\bar{x} - c)^2 . \end{aligned}

For independent draws with mean $\mu$ and variance $\sigma^2$ , take $c = \mu$ . The left side has expected value $n\sigma^2$ , and the last term has expected value $\sigma^2$ because $\bar{x}$ has variance $\sigma^2/n$ . So the sum of squared deviations about $\bar{x}$ has expected value $(n - 1)\sigma^2$ .

In plain words Once the sample mean is fixed, only $n - 1$ of the deviations can be chosen freely. The identity shows that centering at $\bar{x}$ gives the smallest possible sum of squares, since the extra term is never negative. Measured from the true mean instead, the sum would on average be larger by exactly $\sigma^2$ , one observation’s worth of variance, and dividing by $n - 1$ accounts for that loss. The true mean is not needed to compute $s^2$ . It only appears in the argument for why $n - 1$ is the right divisor.

Example For the times $2$ , $4$ , $6$ and $8$ minutes with mean $5$ , the deviations $-3$ , $-1$ and $1$ force the fourth to be $3$ , because the four must add to $0$ . To check the identity with $c = 4$ , the distances from $4$ are $-2$ , $0$ , $2$ and $4$ , and their squares add to $4 + 0 + 4 + 16 = 24$ . The right side gives the same total, $20 + 4 \cdot (5 - 4)^2 = 24$ .

Check yourself

The waiting times of five customers are $3$ , $5$ , $7$ , $9$ and $11$ minutes. Find the population variance and standard deviation, then the sample variance and standard deviation. When is each pair the right summary?

Show the answer

The mean is $7$ minutes, the deviations are $-4$ , $-2$ , $0$ , $2$ and $4$ , and the squared deviations total $16 + 4 + 0 + 4 + 16 = 40$ . Dividing by $N = 5$ gives $\sigma^2 = 8$ square minutes and $\sigma = \sqrt{8} \approx 2.83$ minutes. Dividing by $n - 1 = 4$ gives $s^2 = 10$ square minutes and $s = \sqrt{10} \approx 3.16$ minutes. Use the first pair when these five customers are the whole group you want to describe, for example everyone served during one short event. Use the second when the five are a random sample used to estimate the variability of the shop’s waiting times in general.

Start again from the times $2$ , $4$ , $6$ and $8$ minutes, treated as a complete population. Find the population variance and standard deviation after adding $10$ minutes to every time, and separately after doubling every time. Someone claims that doubling the data doubles the variance. Is that right?

Show the answer

After adding $10$ minutes the times are $12$ , $14$ , $16$ and $18$ with mean $15$ . The deviations are still $-3$ , $-1$ , $1$ and $3$ , so the variance stays $5$ square minutes and the standard deviation stays $\sqrt{5} \approx 2.24$ minutes. After doubling, the times are $4$ , $8$ , $12$ and $16$ with mean $10$ , and the deviations double to $-6$ , $-2$ , $2$ and $6$ . Their squares total $80$ , so the variance is $80/4 = 20$ square minutes and the standard deviation is $\sqrt{20} = 2\sqrt{5} \approx 4.47$ minutes. The claim is wrong. Doubling multiplies every squared deviation by $4$ , so the variance is multiplied by $4$ , while the standard deviation doubles.

A process produces the value $0$ or $2$ , each with probability $\tfrac12$ , independently each time, so its mean is $1$ and its variance is $\sigma^2 = 1$ . Take samples of size $n = 2$ . Find the expected value of $s^2$ , of the version that divides by $n$ , and of $s$ .

Show the answer

The four samples $(0, 0)$ , $(0, 2)$ , $(2, 0)$ and $(2, 2)$ are equally likely. Two of them have no spread, and for the other two the mean is $1$ and the squared deviations total $2$ . So $s^2$ takes the values $0$ , $2$ , $2$ and $0$ , and its expected value is $1$ , equal to $\sigma^2$ . The version that divides by $n = 2$ takes the values $0$ , $1$ , $1$ and $0$ , with expected value $\tfrac12$ , which is $\frac{n-1}{n}\sigma^2$ and too small. The standard deviation $s$ takes the values $0$ , $\sqrt{2}$ , $\sqrt{2}$ and $0$ , with expected value $\sqrt{2}/2 \approx 0.71$ , less than $\sigma = 1$ . The divisor $n - 1$ makes the variance estimate unbiased, but its square root still underestimates $\sigma$ on average.

Variance and Standard Deviation

Key terms

Deviation

Population variance

Sample variance

Standard deviation

Degrees of freedom

Check yourself

How this lesson connects

Builds on

Used later in