Sample Means and Sampling Distributions, introduction to probability lesson

One measurement varies from trial to trial, and so does an average of several measurements, because a new sample can give a new average. Before the data arrive, the average is a random variable with its own probability distribution, and one observed average is only a single value from it. This lesson builds that distribution exactly for small discrete models, keeps the observations, the sample, the statistic and its observed value apart, and shows how the spread of the average depends on the sample size and on independence.

Key terms

Independent and identically distributed

Definition Random variables $X_1, \dots, X_n$ are independent and identically distributed, abbreviated IID, if they are mutually independent and all have the same distribution. For discrete variables that share the PMF $p_X$ of a variable $X$ , this means

\begin{aligned} &P(X_1 = x_1, \dots, X_n = x_n) \\[0.4em] &\quad = \prod_{i=1}^{n} p_X(x_i) \end{aligned}

for all values $x_1, \dots, x_n$ . Having the same distribution does not make variables independent, and independent variables need not have the same distribution.

In plain words Every observation follows the same probability law, and learning the values of some observations does not change the probabilities for the others. IID is a property of the model for how the data arise, not a description of numbers that happen to look alike.

Example In a constructed model, each measurement equals $0$ or $4$ with probability $\tfrac12$ each, so one measurement has mean $\mu = 2$ and variance $\sigma^2 = 4$ . For two IID measurements, each of the ordered pairs $(0, 0)$ , $(0, 4)$ , $(4, 0)$ and $(4, 4)$ has probability $\tfrac12 \cdot \tfrac12 = \tfrac14$ . If instead the second measurement is a copy of the first, so that $X_2 = X_1$ , the two still have the same distribution, but they are dependent, since $P(X_1 = 0, X_2 = 4) = 0$ while $P(X_1 = 0)\,P(X_2 = 4) = \tfrac14$ .

Random sample

Definition In this course, a random sample of size $n$ from a distribution is an ordered list $X_1, \dots, X_n$ of IID random variables, each with that distribution. Before observation the $X_i$ are random. After observation they have particular values, written $x_1, \dots, x_n$ , which form one realization of the sample. The order is part of the outcome, so the samples $(0, 4)$ and $(4, 0)$ are different outcomes.

In plain words A random sample describes, with probabilities, a plan for collecting data before it is carried out. The numbers recorded afterward are one result of that plan, and running the plan again could give different numbers.

Example Two measurements from the $0$ or $4$ model form a random sample $(X_1, X_2)$ with four equally likely outcomes, and a run that records $x_1 = 4$ and $x_2 = 0$ is one realization. By contrast, draw two tickets without replacement from a box holding one ticket marked $0$ and one marked $4$ . Each draw on its own is $0$ or $4$ with probability $\tfrac12$ , but the second draw is always the ticket left over. The draws are identically distributed and dependent, so they do not form a random sample in this sense.

Statistic

Definition A statistic is a random variable $T = g(X_1, \dots, X_n)$ computed from the sample by a fixed rule $g$ that depends on the sample values alone. Before sampling, $T$ is random. After the sample is observed, the number $t = g(x_1, \dots, x_n)$ is the observed value of the statistic.

In plain words A statistic is a recipe, fixed in advance, for turning data into a number. Because the data are random, the result of the recipe is a random variable too. A single observed value of a statistic is one number, not its whole distribution.

Example For a sample $(X_1, X_2)$ from the $0$ or $4$ model, the sample maximum $\max(X_1, X_2)$ , the number of observations equal to $4$ and the average $\tfrac12(X_1 + X_2)$ are all statistics. For the realization $(4, 0)$ their observed values are $4$ , $1$ and $2$ . Before sampling, the maximum equals $4$ with probability $\tfrac34$ , because only the pair $(0, 0)$ gives a maximum of $0$ .

Sample mean

Definition For a sample $X_1, \dots, X_n$ of fixed size $n \ge 1$ , the sample mean is the statistic

\bar{X} = \frac{X_1 + X_2 + \cdots + X_n}{n}.

If every $X_i$ has the same finite mean $\mu$ , linearity of expectation gives

\begin{aligned} E[\bar{X}] &= \frac{1}{n} \sum_{i=1}^{n} E[X_i] \\[0.4em] &= \frac{n\mu}{n} = \mu . \end{aligned}

This step uses only the common mean, not independence.

In plain words The sample mean is the ordinary average of the observations, but before they are observed it is a new random variable. Its possible values can differ from the possible values of a single observation, and when all observations share the same finite mean, its expected value equals that mean.

Example For two IID measurements from the $0$ or $4$ model, the four equally likely pairs have averages $0$ , $2$ , $2$ and $4$ . So $\bar{X} = 2$ with probability $\tfrac12$ , even though a single measurement never equals $2$ . Its expected value is $0 \cdot \tfrac14 + 2 \cdot \tfrac12 + 4 \cdot \tfrac14 = 2$ , which is $\mu$ .

Sampling distribution

Definition The sampling distribution of a statistic $T$ is the probability distribution of $T$ determined by the model for the sample. For a discrete sample, $P(T = t)$ is the total probability of all ordered outcomes $(x_1, \dots, x_n)$ with $g(x_1, \dots, x_n) = t$ . It follows exactly from the model. A histogram of simulated or recorded values of $T$ only approximates it and carries sampling error.

In plain words The sampling distribution assigns probabilities to the values the statistic could take if the whole sampling plan were carried out again. It can be computed from the model without collecting any data.

Example For two observations from the $0$ or $4$ model, the sampling distribution of $\bar{X}$ puts probabilities $\tfrac14$ , $\tfrac12$ and $\tfrac14$ on the values $0$ , $2$ and $4$ . For four IID observations there are $16$ equally likely ordered records. The number of $4$ s in a record is $0$ , $1$ , $2$ , $3$ or $4$ in $1$ , $4$ , $6$ , $4$ and $1$ of them, and the sample mean equals that number, since the sum is $4$ times it.

$\bar{x}$	$0$	$1$	$2$	$3$	$4$
$P(\bar{X} = \bar{x})$	$\tfrac{1}{16}$	$\tfrac{4}{16}$	$\tfrac{6}{16}$	$\tfrac{4}{16}$	$\tfrac{1}{16}$

The center is still $2$ . The probability that $\bar{X}$ lies within $1$ of the center has grown from $\tfrac12$ for two observations to $\tfrac{14}{16}$ , although $P(\bar{X} = 2)$ itself fell from $\tfrac12$ to $\tfrac{6}{16}$ . The distribution is still discrete, with five possible values, and it is not a normal distribution.

Standard error

Definition The standard error of a statistic is the standard deviation of its sampling distribution. Let $X_1, \dots, X_n$ be independent, each with mean $\mu$ and finite variance $\sigma^2$ , and let $S_n = X_1 + \cdots + X_n$ . Independence makes the variances add, so $\operatorname{Var}(S_n) = n\sigma^2$ , and

\begin{aligned} \operatorname{Var}(\bar{X}) &= \tfrac{1}{n^2} \operatorname{Var}(S_n) \\[0.4em] &= \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n} . \end{aligned}

The standard error of the sample mean is therefore

\operatorname{SD}(\bar{X}) = \frac{\sigma}{\sqrt{n}} .

Here $\sigma$ comes from a fully specified model. When $\sigma$ is unknown and estimated from data, the result is only an estimate of the standard error.

In plain words The standard deviation $\sigma$ describes how single observations vary, and the standard error describes how averages of $n$ observations vary from sample to sample. When $\sigma > 0$ , halving the standard error takes four times as many independent observations, because of the square root. A smaller standard error describes the spread of the average across possible samples, and it does not promise that each new observation moves an observed average closer to $\mu$ .

Example In the $0$ or $4$ model $\sigma = 2$ . The standard error of $\bar{X}$ is $2/\sqrt{2} = \sqrt{2}$ for $n = 2$ , matching the variance $\tfrac14 \cdot 4 + \tfrac12 \cdot 0 + \tfrac14 \cdot 4 = 2$ of the sampling distribution for two observations. It is $1$ for $n = 4$ , matching the variance $\tfrac{4 + 4 + 0 + 4 + 4}{16} = 1$ from the table, and $\tfrac12$ for $n = 16$ . The derivation of $\sigma^2/n$ used independence, and without it the rule can fail. If $X_2 = X_1$ , then $\bar{X} = X_1$ and its variance stays $4$ instead of dropping to $2$ . Along a single run, the observations $4$ and $0$ give the average $2$ , which equals $\mu$ , but a third observation $4$ moves the average to $\tfrac83$ , farther from $\mu$ .

Check yourself

Observations are IID with mean $10$ and standard deviation $6$ . Find the mean and the standard error of $\bar{X}$ for $n = 9$ , and the sample size that gives a standard error of $1$ .

Show the answer

The mean of $\bar{X}$ is $10$ for every sample size. For $n = 9$ the standard error is $6/\sqrt{9} = 2$ . A standard error of $1$ needs $6/\sqrt{n} = 1$ , so $\sqrt{n} = 6$ and $n = 36$ , four times the sample size that gave a standard error of $2$ .

Each observation is independently uniform on $\{1, 3, 5\}$ , and the sample size is $2$ . Find the sampling distribution of $\bar{X}$ , its mean and variance, and $P(\bar{X} \ge 4)$ .

Show the answer

The nine ordered pairs are equally likely. The average is $1$ for one pair, $2$ for the two pairs $(1, 3)$ and $(3, 1)$ , $3$ for the three pairs $(1, 5)$ , $(3, 3)$ and $(5, 1)$ , $4$ for two pairs and $5$ for one pair. So $\bar{X}$ takes the values $1$ to $5$ with probabilities $\tfrac19$ , $\tfrac29$ , $\tfrac39$ , $\tfrac29$ and $\tfrac19$ , and its mean is $3$ by symmetry. One observation has mean $3$ and variance $\tfrac{4 + 0 + 4}{3} = \tfrac83$ , so $\operatorname{Var}(\bar{X}) = \tfrac{8/3}{2} = \tfrac43$ . A direct calculation agrees, since $\tfrac{4 \cdot 1 + 1 \cdot 2 + 0 \cdot 3 + 1 \cdot 2 + 4 \cdot 1}{9} = \tfrac{12}{9} = \tfrac43$ . Finally, $P(\bar{X} \ge 4) = \tfrac{2 + 1}{9} = \tfrac13$ .

Now draw two tickets without replacement from a box holding tickets marked $1$ , $3$ and $5$ , each draw equally likely among the tickets still in the box. Do the two draws form a random sample? Find the sampling distribution and the variance of their average, and compare with the answer to question 2.

Show the answer

Each draw on its own is uniform on $\{1, 3, 5\}$ , so the draws are identically distributed. They are dependent, because $P(X_1 = 1, X_2 = 1) = 0$ while $P(X_1 = 1)\,P(X_2 = 1) = \tfrac19$ . So they are not IID and do not form a random sample in the sense of this course. The six ordered pairs $(1, 3)$ , $(1, 5)$ , $(3, 1)$ , $(3, 5)$ , $(5, 1)$ and $(5, 3)$ each have probability $\tfrac16$ , with averages $2$ , $3$ , $2$ , $4$ , $3$ and $4$ . So the average equals $2$ , $3$ or $4$ with probability $\tfrac13$ each. Its mean is still $3$ , since linearity does not need independence, but its variance is $\tfrac{1 + 0 + 1}{3} = \tfrac23$ , half the IID value $\tfrac43$ . The formula $\sigma^2/n$ assumes independence and does not apply here. Dependence lowered the variance in this case, while a copied observation raises it.

Sample Means and Sampling Distributions

Key terms

Independent and identically distributed

Random sample

Statistic

Sample mean

Sampling distribution

Standard error

Check yourself

How this lesson connects

Builds on

Used later in

Where this leads in Statistics