Mean and Variance of the Sample Mean, introduction to probability lesson

In a constructed model, each delay at a service desk is $0$ or $4$ minutes with equal probability, so the model mean is $2$ minutes. Suppose that one morning six delays are recorded, $0$ , $4$ , $4$ , $4$ , $0$ and $4$ , and their average is $8/3$ of a minute. Does that average show the model is wrong, and how far from $2$ should an average of $n$ delays usually be? This lesson separates the random sample mean from the number one sample produces, shows that the sample mean is centered at the model mean, and finds how its spread shrinks with the sample size when the observations are independent.

Key terms

Realized average

Definition Let $X_1, \dots, X_n$ , with $n \ge 1$ , be random variables that record $n$ observations not yet made. The sample mean is the random variable

\bar{X}_n = \frac{1}{n} \sum_{i=1}^{n} X_i .

Once the observations are made and take the values $x_1, \dots, x_n$ , the realized average is the number

\bar{x}_n = \frac{1}{n} \sum_{i=1}^{n} x_i ,

which is the value of $\bar{X}_n$ for that one outcome. Capital letters stand for random variables and lowercase letters for their observed values.

In plain words The sample mean is a rule applied to data that are still random, so it has a probability distribution. A realized average is the single number the rule returns for one data set. Collecting the data again can give a different realized average.

Example In the delay model each $X_i$ is $0$ or $4$ minutes with probability $\tfrac12$ each, so $\mu = E[X_i] = 2$ minutes and $\sigma^2 = \operatorname{Var}(X_i) = 4$ square minutes. The recorded delays above have sum $16$ minutes and realized average

\bar{x}_6 = \frac{16}{6} = \frac{8}{3} \approx 2.67

minutes. This number differs from $\mu = 2$ without contradicting the model. The value $\mu$ is the center of the distribution of $\bar{X}_6$ , not a value that each sample has to reproduce.

Unbiasedness

Definition A statistic $T$ used to estimate a quantity $\theta$ of the model is unbiased for $\theta$ if $E[T] = \theta$ for every distribution the model allows. If each $X_i$ has the same finite mean $\mu$ , linearity of expectation gives

\begin{aligned} E[\bar{X}_n] &= \frac{1}{n} \sum_{i=1}^{n} E[X_i] \\[0.4em] &= \frac{n\mu}{n} = \mu , \end{aligned}

so the sample mean is unbiased for $\mu$ , whatever the value of $\mu$ . Linearity holds for dependent as well as independent variables, so this conclusion does not need independence.

In plain words Unbiased means centered. Averaged over all the samples the model can produce, weighted by their probabilities, the sample mean lands on $\mu$ . Unbiasedness says nothing about how far one realized average falls from $\mu$ , and an unbiased average can still be imprecise when the sample is small or the observations vary a lot.

Example With three independent delays, the sample mean equals $0$ , $4/3$ , $8/3$ or $4$ minutes with probabilities $\tfrac18$ , $\tfrac38$ , $\tfrac38$ and $\tfrac18$ . Its expectation is

\begin{aligned} E[\bar{X}_3] &= 0 \cdot \tfrac18 + \tfrac43 \cdot \tfrac38 \\[0.4em] &\quad + \tfrac83 \cdot \tfrac38 + 4 \cdot \tfrac18 = 2, \end{aligned}

although no realized average of three delays equals $2$ . Now copy one delay $X$ into all three positions. Each position still has mean $2$ , so the average is still unbiased, but it equals $0$ or $4$ with probability $\tfrac12$ each.

Variance of the sample mean

Definition Let $X_1, \dots, X_n$ have finite variances and let $S_n = X_1 + \cdots + X_n$ . Because $\operatorname{Var}(cY) = c^2 \operatorname{Var}(Y)$ for a constant $c$ ,

\operatorname{Var}(\bar{X}_n) = \frac{\operatorname{Var}(S_n)}{n^2},

and the variance of a sum collects every variance and every covariance,

\begin{gathered} \operatorname{Var}(S_n) = \sum_{i=1}^{n} \operatorname{Var}(X_i) \\[0.4em] + 2 \sum_{i<j} \operatorname{Cov}(X_i, X_j). \end{gathered}

If the $X_i$ are independent, or merely uncorrelated, every covariance is $0$ . If in addition they share the variance $\sigma^2$ , as IID observations with finite variance do, then $\operatorname{Var}(S_n) = n\sigma^2$ and

\operatorname{Var}(\bar{X}_n) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}.

In plain words Dividing by $n$ divides the variance by $n^2$ , while adding $n$ uncorrelated observations with a common variance multiplies that variance only by $n$ , so the variance of the average is the single-observation variance divided by $n$ . Independence does its work by removing the covariance terms. Positive covariances add variance back, and identical distributions alone do not remove them.

Example For two independent delays, the ordered samples $(0, 0)$ , $(0, 4)$ , $(4, 0)$ and $(4, 4)$ are equally likely and have averages $0$ , $2$ , $2$ and $4$ . So $\operatorname{Var}(\bar{X}_2) = \tfrac14 \cdot 4 + \tfrac12 \cdot 0 + \tfrac14 \cdot 4 = 2$ , which is $4/2$ . For four independent delays the average equals the number of $4$ -minute delays, a $\operatorname{Bin}(4, \tfrac12)$ count, so it takes the values $0$ to $4$ with probabilities $\tfrac{1}{16}$ , $\tfrac{4}{16}$ , $\tfrac{6}{16}$ , $\tfrac{4}{16}$ and $\tfrac{1}{16}$ , and its variance is $1 = 4/4$ . If instead one delay $X$ is copied into all $n$ positions, every covariance equals $\operatorname{Var}(X) = 4$ , and

\begin{aligned} \operatorname{Var}(\bar{X}_n) &= \frac{4n + 4n(n - 1)}{n^2} \\[0.4em] &= 4 \end{aligned}

for every $n$ , as it must be, since then $\bar{X}_n = X$ .

Standard error of the mean

Definition The standard error of the sample mean is its standard deviation,

\begin{aligned} \operatorname{SE}(\bar{X}_n) &= \operatorname{SD}(\bar{X}_n) \\[0.4em] &= \sqrt{\operatorname{Var}(\bar{X}_n)} . \end{aligned}

For IID observations with finite standard deviation $\sigma$ it equals

\operatorname{SE}(\bar{X}_n) = \frac{\sigma}{\sqrt{n}} .

When the observations share the mean $\mu$ , so that $E[\bar{X}_n] = \mu$ , the square of the standard error is $E\bigl[(\bar{X}_n - \mu)^2\bigr]$ , the mean squared error of the sample mean as an estimate of $\mu$ . Here $\sigma$ comes from a fully specified model. When $\sigma$ is estimated from data, the result is an estimate of the standard error.

In plain words The standard error is the typical size of the miss $\bar{X}_n - \mu$ across repeated samples, measured as a root mean square. It describes averages, not single observations, whose standard deviation stays $\sigma$ however many are collected. It is not a maximum error. Unless the miss has the same size on every outcome, some samples miss by more than one standard error.

Example For IID delays the standard error is $2/\sqrt{n}$ minutes, which is $0.4$ minutes at $n = 25$ and $0.2$ minutes at $n = 100$ , while each delay still has standard deviation $2$ minutes. At $n = 4$ the standard error is $1$ minute, and the miss $\lvert \bar{X}_4 - 2 \rvert$ equals $0$ , $1$ or $2$ minutes with probabilities $\tfrac38$ , $\tfrac12$ and $\tfrac18$ . So the miss exceeds one standard error with probability $\tfrac18$ . The inequality in an accuracy question matters too, because

\begin{gathered} P(\lvert \bar{X}_4 - 2 \rvert \le 1) = \tfrac78, \\[0.4em] P(\lvert \bar{X}_4 - 2 \rvert < 1) = \tfrac38, \end{gathered}

and the difference is the probability $\tfrac12$ that sits on the boundary values $1$ and $3$ .

Square-root scaling

Definition For IID observations with standard deviation $\sigma > 0$ , the standard error $\sigma/\sqrt{n}$ is proportional to $1/\sqrt{n}$ . Multiplying the sample size by a factor $c > 0$ , with $cn$ a whole number, divides the standard error by $\sqrt{c}$ , so dividing the standard error by $k > 0$ takes $k^2$ times as many observations. A target $s > 0$ is met, meaning $\sigma/\sqrt{n} \le s$ , exactly when

n \ge \frac{\sigma^2}{s^2}.

If $\sigma = 0$ , every observation equals $\mu$ with probability $1$ and the standard error is $0$ for every $n$ .

In plain words Precision gets more expensive as it improves. Each halving of the standard error costs four times as many independent observations, so doubling the sample size reduces the standard error by a factor of only about $1.41$ .

Example For the delays the standard error is $0.4$ , $0.2$ and $0.1$ minutes at $n = 25$ , $100$ and $400$ . Each step multiplies $n$ by $4$ and halves the standard error. A standard error of at most $0.1$ minutes needs $n \ge 4/0.01 = 400$ . Doubling $n$ from $25$ to $50$ gives $2/\sqrt{50} \approx 0.28$ minutes, not $0.2$ . The scaling counts independent observations, and $n$ copies of one delay keep the standard error at $2$ minutes for every $n$ .

Check yourself

IID measurements have mean $12$ units and standard deviation $6$ units. Find the mean, the variance and the standard error of $\bar{X}_n$ for $n = 9$ and for $n = 36$ . How many observations make the standard error at most $1$ unit?

Show the answer

The mean of $\bar{X}_n$ is $12$ for both sample sizes. The variance of one measurement is $6^2 = 36$ square units, so $\operatorname{Var}(\bar{X}_9) = 36/9 = 4$ and $\operatorname{SE}(\bar{X}_9) = 2$ units, while $\operatorname{Var}(\bar{X}_{36}) = 36/36 = 1$ and $\operatorname{SE}(\bar{X}_{36}) = 1$ unit. The condition $6/\sqrt{n} \le 1$ means $n \ge 36$ , so $36$ observations are the fewest that suffice. Using $6$ as the variance would give the wrong standard error $\sqrt{6/9} \approx 0.82$ for $n = 9$ .

In the same model, a technician makes one measurement $X$ and enters it $36$ times. Is the average of the $36$ entries unbiased for $12$ ? What is its standard error, and why does the answer to question 1 not apply?

Show the answer

Each entry equals $X$ , so the average equals $X$ and its mean is $12$ . It is unbiased, because linearity of expectation does not need independence. Its standard error is $\operatorname{SD}(X) = 6$ units, not $1$ . In the covariance formula every pair has $\operatorname{Cov}(X_i, X_j) = \operatorname{Var}(X) = 36$ , so

\begin{aligned} \operatorname{Var}(S_{36}) &= 36 \cdot 36 \\[0.4em] &\quad + 2 \cdot 630 \cdot 36 \\[0.4em] &= 46656 , \end{aligned}

using $\binom{36}{2} = 630$ pairs, and $\operatorname{Var}(\bar{X}_{36}) = 46656/36^2 = 36$ . The formula $\sigma^2/n$ assumes zero covariances, and copies of one measurement carry no new information.

Three observations are independent, and each equals $3$ with probability $\tfrac13$ and $0$ otherwise. Find the distribution, the mean, the variance and the standard error of $\bar{X}_3$ . How likely is the miss $\lvert \bar{X}_3 - 1 \rvert$ to exceed the standard error? Compare $P(\lvert \bar{X}_3 - 1 \rvert \le 1)$ with $P(\lvert \bar{X}_3 - 1 \rvert < 1)$ .

Show the answer

One observation has mean $3 \cdot \tfrac13 = 1$ and variance $9 \cdot \tfrac13 - 1^2 = 2$ . Let $K \sim \operatorname{Bin}(3, \tfrac13)$ count the observations equal to $3$ . The sum is $3K$ , so $\bar{X}_3 = K$ , which takes the values $0$ , $1$ , $2$ and $3$ with probabilities $\tfrac{8}{27}$ , $\tfrac{12}{27}$ , $\tfrac{6}{27}$ and $\tfrac{1}{27}$ . Its mean is $1$ , and its variance is $\sigma^2/n = 2/3$ , which agrees with the direct calculation $E[(K - 1)^2] = (8 + 0 + 6 + 4)/27 = 18/27$ . The standard error is $\sqrt{2/3} \approx 0.816$ . The miss $\lvert K - 1 \rvert$ is $0$ when $K = 1$ and at least $1$ otherwise, so it exceeds $0.816$ with probability $1 - \tfrac{12}{27} = \tfrac59$ , more than half the time. Finally $P(\lvert \bar{X}_3 - 1 \rvert \le 1) = 1 - \tfrac{1}{27} = \tfrac{26}{27}$ , while $P(\lvert \bar{X}_3 - 1 \rvert < 1) = P(K = 1) = \tfrac49$ , because the values $0$ and $2$ sit exactly on the boundary.

Mean and Variance of the Sample Mean

Key terms

Realized average

Unbiasedness

Variance of the sample mean

Standard error of the mean

Square-root scaling

Check yourself

How this lesson connects

Builds on

Used later in

Where this leads in Statistics