Sampling Distributions of Means and Proportions, statistics lesson

Two polling organizations each draw a random sample of adults from the same city in the same week. One reports 52 percent support for a proposal and the other reports 48 percent. Neither has to be wrong, because each sample is only one of the many samples its procedure could have produced, and the percentage moves with the sample. This lesson separates the distribution of individual values in a population from the distribution of a statistic over repeated samples, and works out both exactly in an example small enough to list every possible sample.

Key terms

Population distribution

Definition For a population of $N$ units and a numerical variable, the population distribution gives each value the proportion of units that have it. Equivalently, it is the probability distribution of the value of one unit selected uniformly at random, each unit having probability $1/N$ . Its mean $\mu$ and standard deviation $\sigma$ are parameters, fixed numbers that describe the population. In this lesson the population variance uses the denominator $N$ ,

\sigma^2 = \frac{1}{N}\sum_{j=1}^{N} (x_j - \mu)^2 ,

which makes it the variance of one randomly selected value. When the data come from an ongoing process rather than a list of units, the population distribution is a probability model for a single observation.

In plain words The population distribution is what you would see if you could measure every unit. Drawing a sample does not change it, and it describes single observations, not summaries of several.

Example A constructed population has three cards labeled $0$ , $2$ and $4$ . Each value has probability $1/3$ , so the mean is $\mu = 2$ . The squared deviations from the mean are $4$ , $0$ and $4$ , so $\sigma^2 = 8/3$ and $\sigma \approx 1.63$ . A graph of this population has three equal bars, at $0$ , $2$ and $4$ .

Repeated sampling

Definition Repeated sampling means carrying out the same sampling procedure again and again, always from the same population, with the same sample size and the same selection rule, and computing the same statistic each time. A statistic is any quantity computed from the sample data alone, such as a sample mean $\bar{x}$ or a sample proportion $\hat{p}$ . The repetitions are usually a thought experiment or a simulation. When the possible samples are few, they can be listed completely together with their probabilities, which gives exact results.

In plain words A study usually observes one sample. Repeated sampling asks what the same procedure could have produced instead, and that is what makes it possible to say how much a statistic tends to move from sample to sample.

Example Draw two cards from the population $0$ , $2$ , $4$ with replacement, so the first card goes back before the second draw, and let the two draws be independent and uniformly random. There are nine ordered pairs, each with probability $\tfrac13 \cdot \tfrac13 = \tfrac19$ . The pairs $(0, 2)$ and $(2, 0)$ count as different outcomes. The table gives the sample mean of every pair.

First card	Second card $0$	Second card $2$	Second card $4$
$0$	$0$	$1$	$2$
$2$	$1$	$2$	$3$
$4$	$2$	$3$	$4$

Sampling distribution

Definition The sampling distribution of a statistic is the probability distribution of the values the statistic takes over repeated sampling. When the possible samples can be listed, it gives each possible value the total probability of the samples that produce it. It depends on the population, the sample size, the sampling design and the statistic itself. A histogram of the observations in one sample is not a sampling distribution. It shows individual values, and the whole sample contributes just one value of the statistic.

In plain words The population distribution answers which individual values are common. A sampling distribution answers which values of a summary are common when the whole procedure is repeated. Before reading any such graph, name the quantity on its horizontal axis, the sample size and the selection rule.

Example For the nine equally likely pairs above, let $M$ be the larger of the two cards. It equals $0$ only for $(0, 0)$ , it equals $2$ for $(0, 2)$ , $(2, 0)$ and $(2, 2)$ , and it equals $4$ for the other five pairs.

Value of $M$	$0$	$2$	$4$
Probability	$1/9$	$3/9$	$5/9$

Its mean is $E(M) = 26/9 \approx 2.89$ , above the population mean $2$ . Each statistic computed from the same nine samples has its own sampling distribution.

Sampling distribution of the sample mean

Definition For a sample $X_1, \dots, X_n$ , the sample mean is the random variable

\bar{X} = \frac{1}{n}\sum_{i=1}^{n} X_i ,

and an observed value of it is written $\bar{x}$ . If each $X_i$ has the population distribution, with mean $\mu$ , then $E(\bar{X}) = \mu$ . If in addition the observations are independent and the population variance $\sigma^2$ is finite, as with random sampling with replacement, then

\begin{gathered} \operatorname{Var}(\bar{X}) = \frac{\sigma^2}{n}, \\[0.4em] \operatorname{SD}(\bar{X}) = \frac{\sigma}{\sqrt{n}} . \end{gathered}

The shape of the sampling distribution can differ from the shape of the population distribution, and averaging can produce values that no single observation can take.

In plain words The sample mean is centered on the population mean, but it need not equal the population mean in a particular sample. The variance of an average of $n$ independent draws from the population is the variance of a single draw divided by $n$ , while each individual observation stays exactly as variable as before.

Example Counting the nine means in the table gives the sampling distribution of $\bar{X}$ for two draws with replacement.

$\bar{x}$	$0$	$1$	$2$	$3$	$4$
$P(\bar{X} = \bar{x})$	$1/9$	$2/9$	$3/9$	$2/9$	$1/9$

Its mean is $2$ , equal to $\mu$ . The squared deviations from $2$ are $4$ , $1$ , $0$ , $1$ and $4$ , and weighting them by the counts $1$ , $2$ , $3$ , $2$ and $1$ gives the variance $(4 + 2 + 0 + 2 + 4)/9 = 4/3$ , which matches $\sigma^2/2 = (8/3)/2$ . The means $1$ and $3$ are not the value of any card. The samples $(0, 0)$ and $(4, 4)$ give the means $0$ and $4$ , so a statistic centered on its target can still land far from it in a particular sample.

Sampling distribution of the sample proportion

Definition When each observation records whether a unit has some property, code it as $1$ for yes and $0$ for no. The sample proportion $\hat{p}$ is the number $X$ of ones divided by $n$ , which is the sample mean of the zeros and ones. If the $n$ observations are independent and each equals $1$ with probability $p$ , the population proportion, then $X \sim \operatorname{Binomial}(n, p)$ and, for $k = 0, 1, \dots, n$ ,

\begin{aligned} &P\bigl(\hat{p} = \tfrac{k}{n}\bigr) = P(X = k) \\[0.4em] &\quad = \binom{n}{k} p^k (1 - p)^{n - k} . \end{aligned}

Its mean and variance are

\begin{gathered} E(\hat{p}) = p, \\[0.4em] \operatorname{Var}(\hat{p}) = \frac{p(1 - p)}{n} . \end{gathered}

In plain words A proportion is a mean of zeros and ones, so what holds for sample means holds for it, with $p(1 - p)$ as the variance of one observation. Its possible values are only $0, 1/n, 2/n, \dots, 1$ , and its sampling distribution is centered on the population proportion.

Example In the card population, call a card flagged when its value is $4$ , so $p = 1/3$ . With two independent draws, neither card is flagged with probability $\tfrac23 \cdot \tfrac23 = \tfrac49$ , exactly one is flagged with probability $2 \cdot \tfrac13 \cdot \tfrac23 = \tfrac49$ , and both are flagged with probability $\tfrac19$ . So $\hat{p}$ takes the values $0$ , $1/2$ and $1$ with probabilities $4/9$ , $4/9$ and $1/9$ . Its mean is $\tfrac12 \cdot \tfrac49 + 1 \cdot \tfrac19 = \tfrac13 = p$ , and its variance is $(1/3)(2/3)/2 = 1/9$ . These are the same nine samples as before, summarized by a different statistic.

Finite population correction

Definition Suppose a simple random sample of $n$ units is drawn without replacement from a population of $N \ge 2$ units with mean $\mu$ and variance $\sigma^2$ computed with the denominator $N$ . Then $E(\bar{X}) = \mu$ still holds, but

\operatorname{Var}(\bar{X}) = \frac{\sigma^2}{n} \cdot \frac{N - n}{N - 1} .

The factor $(N - n)/(N - 1)$ is the finite population correction for the variance, and its square root is the correction for the standard deviation. Some books define the population variance with the denominator $N - 1$ ,

\begin{aligned} S^2 &= \frac{1}{N - 1}\sum_{j=1}^{N}(x_j - \mu)^2 \\[0.4em] &= \frac{N}{N - 1}\,\sigma^2 , \end{aligned}

and then the same variance reads $(S^2/n)(1 - n/N)$ . The two formulas agree, but combining one convention’s variance with the other convention’s factor in general gives a wrong answer. The factor equals $1$ when $n = 1$ , it is close to $1$ when $n$ is a small fraction of $N$ , and it equals $0$ when $n = N$ . For a sample proportion the same factor multiplies $p(1 - p)/n$ .

In plain words Without replacement, a sample that covers a large share of a finite population leaves little of it unseen, so the sample mean varies less than the independent-draw formula says. A census has no sampling variability at all, although measurement errors can remain. When the population is large compared with the sample, the correction hardly matters.

Example Draw two of the three cards without replacement. The possible samples are $\{0, 2\}$ , $\{0, 4\}$ and $\{2, 4\}$ , each with probability $1/3$ , and their means are $1$ , $2$ and $3$ . The extreme means $0$ and $4$ are now impossible. The mean of $\bar{X}$ is still $2$ , and its variance is $(1 + 0 + 1)/3 = 2/3$ . The formula agrees, since

\frac{8/3}{2} \cdot \frac{3 - 2}{3 - 1} = \frac{2}{3},

and so does the other convention, with $S^2 = 4$ and $(4/2)(1 - 2/3) = 2/3$ . The with-replacement variance $4/3$ is twice as large.

Check yourself

A population has three cards labeled $1$ , $4$ and $7$ . Two cards are drawn with replacement, independently and uniformly at random. Find the sampling distribution of the sample mean, its mean and its variance.

Show the answer

The nine ordered pairs are equally likely. Their means are $1$ once, $2.5$ twice, $4$ three times, $5.5$ twice and $7$ once, so these values have probabilities $1/9$ , $2/9$ , $3/9$ , $2/9$ and $1/9$ . The distribution is symmetric about $4$ , so $E(\bar{X}) = 4 = \mu$ .

The population variance, with denominator $3$ , is $(9 + 0 + 9)/3 = 6$ , so $\operatorname{Var}(\bar{X}) = 6/2 = 3$ . A direct check from the distribution agrees, since $(9 \cdot 1 + 2.25 \cdot 2 + 0 \cdot 3 + 2.25 \cdot 2 + 9 \cdot 1)/9 = 27/9 = 3$ .

Now draw two of the same three cards without replacement. Find the variance of the sample mean directly and check it with the finite population correction. A classmate instead computes $(9/2) \cdot (1/2)$ . What went wrong?

Show the answer

The three equally likely samples $\{1, 4\}$ , $\{1, 7\}$ and $\{4, 7\}$ have means $2.5$ , $4$ and $5.5$ . The mean of $\bar{X}$ is $4$ , and its variance is $(2.25 + 0 + 2.25)/3 = 1.5$ . With $\sigma^2 = 6$ the correction gives $(6/2) \cdot (3 - 2)/(3 - 1) = 3/2$ , which agrees.

The classmate used $S^2 = 18/2 = 9$ , the variance with denominator $N - 1$ , inside the factor $(N - n)/(N - 1)$ that belongs to the denominator $N$ convention, and got $9/4$ . With $S^2$ the matching formula is $(9/2)(1 - 2/3) = 3/2$ .

A researcher draws one simple random sample of 400 households from a large city, plots a histogram of the 400 household incomes, and labels it “sampling distribution of the mean income”. What does the histogram show, and how does the sampling distribution of the mean differ from it?

Show the answer

The histogram shows the incomes of individual households in one sample. It is evidence about the shape and spread of the population distribution, and the whole sample supplies only one value $\bar{x}$ of the mean.

The sampling distribution of $\bar{X}$ is the distribution of the mean over all samples of 400 households that the same design could draw. It is centered at the population mean $\mu$ . Since the city is much larger than the sample, the finite population correction is close to $1$ , so its standard deviation is about $\sigma/\sqrt{400} = \sigma/20$ , one twentieth of the standard deviation of individual incomes. Its shape need not match the histogram either. The horizontal axis of the correct graph is the sample mean, not the income of one household.

Sampling Distributions of Means and Proportions

Key terms

Population distribution

Repeated sampling

Sampling distribution

Sampling distribution of the sample mean

Sampling distribution of the sample proportion

Finite population correction

Check yourself

How this lesson connects

Builds on

Used later in