The Central Limit Theorem, introduction to probability lesson

In a constructed model, $100$ waits at a service desk are independent exponential times with rate $1$ per minute. A single wait is strongly skewed, often short and occasionally long, and the law of large numbers says that the average wait is probably close to $1$ minute. It does not say how the remaining fluctuation is distributed, for example how likely the average is to land between $0.8$ and $1.2$ minutes. The central limit theorem gives an approximate answer once the sum or the average is centered and scaled. This lesson states the theorem precisely, uses it for sums and averages, and marks where it gives no guarantee.

Key terms

Standardized sum

Definition Let $X_1, \dots, X_n$ be IID with mean $\mu$ and variance $\sigma^2$ , where $0 < \sigma^2 < \infty$ , and let $S_n = X_1 + \cdots + X_n$ and $\bar{X}_n = S_n/n$ . Then $E[S_n] = n\mu$ and $\operatorname{SD}(S_n) = \sigma\sqrt{n}$ , while $E[\bar{X}_n] = \mu$ and $\operatorname{SD}(\bar{X}_n) = \sigma/\sqrt{n}$ , the standard error of the sample mean. The standardized sum is

\begin{aligned} Z_n &= \frac{S_n - n\mu}{\sigma\sqrt{n}} \\[0.4em] &= \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}}, \end{aligned}

so the standardized sum and the standardized average are the same random variable. It has $E[Z_n] = 0$ and $\operatorname{Var}(Z_n) = 1$ for every $n$ .

In plain words Measure the distance of the sum from its mean in units of its own standard deviation. Without this rescaling the spread of the sum grows like $\sqrt{n}$ and the spread of the average shrinks like $1/\sqrt{n}$ , so neither keeps a fixed scale on which a shape could settle. Three distributions must be kept apart, the population distribution of one observation, the distribution of the sum and the sampling distribution of the average.

Example In the waiting model $X_i \sim \operatorname{Exp}(1)$ in minutes, so $\mu = 1$ and $\sigma = 1$ . For $n = 100$ the total wait $S_{100}$ has mean $100$ and standard deviation $10$ , and the average $\bar{X}_{100}$ has mean $1$ and standard error $0.1$ . A total of $120$ minutes and an average of $1.2$ minutes describe the same outcome, and both standardize to

\frac{120 - 100}{10} = \frac{1.2 - 1}{0.1} = 2.

Each single wait still has the exponential distribution, whatever $n$ is.

Convergence in distribution

Definition Random variables $Y_1, Y_2, \dots$ with CDFs $F_1, F_2, \dots$ converge in distribution to a random variable $Y$ with CDF $F$ if

\lim_{n \to \infty} F_n(y) = F(y)

at every real $y$ where $F$ is continuous. When the limit is standard normal, its CDF $\Phi$ is continuous everywhere, so the convergence holds at every real number. Then also $P(a < Y_n \le b) \to \Phi(b) - \Phi(a)$ for all $a < b$ , with the same limit for open or closed endpoints. Convergence in distribution is a statement about probabilities, not a claim that the values of $Y_n$ approach the values of some particular random variable.

In plain words Away from the jumps of the limiting CDF, the CDFs settle down to it, and so do probabilities of intervals whose endpoints avoid those jumps. The jump points are excused, because probability that approaches a jump point from the right is never counted at that point along the way.

Example Let $Y_n = 1/n$ with probability $1$ , which ought to converge to the constant $0$ . For $y > 0$ the CDF $F_n(y)$ equals $1$ once $1/n \le y$ , and for $y < 0$ it equals $0$ , matching the CDF of the constant $0$ . At $y = 0$ , however, $F_n(0) = P(1/n \le 0) = 0$ for every $n$ , while the limiting CDF has $F(0) = 1$ . The point $0$ is where $F$ jumps, so the definition does not test it, and $Y_n$ converges in distribution to $0$ .

Central limit theorem

Definition Let $X_1, X_2, \dots$ be IID with mean $\mu$ and variance $\sigma^2$ , where $0 < \sigma^2 < \infty$ , and let $S_n = X_1 + \cdots + X_n$ . Then for every real $z$

\begin{aligned} &P\Bigl(\frac{S_n - n\mu}{\sigma\sqrt{n}} \le z\Bigr) \\[0.4em] &\quad \to \Phi(z) \quad \text{as } n \to \infty. \end{aligned}

That is, the standardized sum $Z_n$ , which equals the standardized average, converges in distribution to a standard normal variable. The theorem is a limit statement. For a fixed $n$ the distribution of $Z_n$ is generally not exactly normal, and the theorem by itself gives no bound on the error at any particular $n$ . Independence and a finite positive variance are part of the hypothesis. Other central limit theorems cover some dependent observations or observations with different distributions, under conditions of their own.

In plain words Once centered and scaled, the sum of many independent copies of a random variable with finite positive variance has probabilities of intervals close to standard normal ones, whatever the shape of a single observation. How close depends on the distribution, on $n$ and on the event.

Example For the $100$ exponential waits, the event $0.8 < \bar{X}_{100} < 1.2$ is the event $-2 < Z_{100} < 2$ , so

\begin{aligned} &P(0.8 < \bar{X}_{100} < 1.2) \\[0.4em] &\quad \approx \Phi(2) - \Phi(-2) \\[0.4em] &\quad \approx 0.9545. \end{aligned}

The exact value under the exponential model, from the distribution of a sum of independent exponential waits, is about $0.9550$ , so here the approximation is off by about $0.0005$ . Independence matters. If every observation equals the same wait $W \sim \operatorname{Exp}(1)$ , then $S_n = nW$ has standard deviation $n$ , which exceeds $\sqrt{n}$ for every $n \ge 2$ . The formula above then gives $\sqrt{n}\,(W - 1)$ , whose variance $n$ grows without bound, and standardizing with the true standard deviation gives $W - 1$ , which keeps the skewed exponential shape for every $n$ .

Normal approximation for an average

Definition Under the hypotheses of the central limit theorem and for large $n$ , the sample mean is approximately normal,

\bar{X}_n \approx N\Bigl(\mu, \frac{\sigma^2}{n}\Bigr).

This means that for $a < b$

\begin{aligned} &P(a < \bar{X}_n < b) \\[0.4em] &\quad \approx \Phi(z_b) - \Phi(z_a), \end{aligned}

with the standardized endpoints

\begin{gathered} z_a = \frac{a - \mu}{\sigma/\sqrt{n}}, \\[0.4em] z_b = \frac{b - \mu}{\sigma/\sqrt{n}}. \end{gathered}

Equivalently, $S_n$ is approximately $N(n\mu, n\sigma^2)$ . Both statements are approximations. When the observations are themselves normal, $\bar{X}_n$ is exactly $N(\mu, \sigma^2/n)$ for every $n \ge 1$ , which is a separate fact about sums of independent normal variables. For a discrete population the average stays discrete, and an endpoint adjustment can improve the approximation, as the next lesson shows.

In plain words Turn the event about the average into an event about the standardized variable, then read the probability from $\Phi$ . The approximation describes the average of many observations, not a single observation. It is usually least accurate, relative to the size of the probability, far out in the tails.

Example A rule such as $n \ge 30$ cannot guarantee accuracy for every distribution and every event. For $30$ IID $\operatorname{Exp}(1)$ waits the standard error is $1/\sqrt{30} \approx 0.1826$ . The table compares the normal approximation with exact values from the distribution of the sum.

Event	Normal approximation	Exact
$0.8 < \bar{X}_{30} < 1.2$	$0.7267$	$0.7300$
$\bar{X}_{30} > 1.5$	$0.0031$	$0.0073$
$\bar{X}_{30} < 0.5$	$0.0031$	$0.0004$

Near the center the approximation is close. In the upper tail it gives less than half the true value, and in the lower tail it is about seven times too large, because the skew of the exponential distribution is still visible at $n = 30$ .

Finite positive variance

Definition The theorem requires $0 < \sigma^2 < \infty$ . A finite variance means $E[X^2] < \infty$ , which also makes the mean finite, and it is needed so that $\sigma$ exists for the standardization. A positive variance rules out a constant observation. If $\sigma^2 = 0$ , each $X_i$ equals $\mu$ with probability $1$ , the sum equals $n\mu$ with probability $1$ , and the standardization would divide by $0$ .

In plain words The theorem describes fluctuations measured in units of $\sigma$ , so $\sigma$ must be a genuine positive number. With no fluctuation there is nothing to describe, and with infinite variance there is no unit to measure in.

Example For $\operatorname{Exp}(\lambda)$ the variance $1/\lambda^2$ is finite and positive for every rate $\lambda > 0$ , and for $\operatorname{Uniform}(a, b)$ the variance $(b - a)^2/12$ is finite and positive. A Bernoulli variable with $p = 1$ has variance $0$ , and then $S_n = n$ with probability $1$ . The density $f(x) = 2/x^3$ for $x \ge 1$ has mean

\int_1^\infty x \cdot \frac{2}{x^3}\,dx = 2,

but $E[X^2] = \int_1^\infty 2/x \, dx = \infty$ . Its variance is infinite, so the theorem as stated does not apply to it.

Heavy tails

Definition Heavy tails is an informal description rather than one precise condition. A distribution is called heavy-tailed when it puts comparatively much probability far from its center, often with tail probabilities that shrink like a power of $t$ , such as $P(X > t) = t^{-3}$ for $t \ge 1$ , rather than exponentially fast. Some heavy-tailed distributions have finite variance, and then the central limit theorem applies, although a large $n$ may be needed. Others have infinite variance, and then this theorem does not apply. For the standard Cauchy density $1/(\pi(1 + x^2))$ , which has no mean at all, the average of $n$ IID copies has the same standard Cauchy distribution for every $n$ , so averaging does not concentrate it.

In plain words A few rare, very large values can dominate a sum. Before using a normal approximation, check that the variance exists, and do not rely on a rule of thumb for the sample size when extreme values are plausible.

Example The density $f(x) = 3/x^4$ for $x \ge 1$ has $P(X > t) = t^{-3}$ for $t \ge 1$ , mean $\tfrac32$ , $E[X^2] = 3$ and variance $3 - \tfrac94 = \tfrac34$ . It is heavy-tailed, since $P(X > 10) = 0.001$ , while a normal variable with the same mean and variance exceeds $10$ with probability about $5 \times 10^{-23}$ . Its variance is finite, so the theorem applies to its averages. The density $2/x^3$ of the previous card has $P(X > t) = t^{-2}$ for $t \ge 1$ and infinite variance, so the theorem stated here does not apply to it.

Check yourself

Waits are IID exponential with mean $3$ minutes, and $n = 225$ . Find the mean and the standard error of the average, approximate $P(2.6 < \bar{X} < 3.4)$ , and give the equivalent event for the sum $S_{225}$ .

Show the answer

An exponential wait with mean $3$ has rate $\tfrac13$ , variance $9$ and standard deviation $3$ . The average has mean $3$ and standard error $3/\sqrt{225} = 3/15 = 0.2$ minutes. The standardized endpoints are $(2.6 - 3)/0.2 = -2$ and $(3.4 - 3)/0.2 = 2$ , so

\begin{aligned} &P(2.6 < \bar{X} < 3.4) \\[0.4em] &\quad \approx \Phi(2) - \Phi(-2) \\[0.4em] &\quad \approx 0.9545. \end{aligned}

Multiplying by $225$ gives the same event for the sum, $585 < S_{225} < 765$ . The sum has mean $675$ and standard deviation $3 \cdot 15 = 45$ , and $(585 - 675)/45 = -2$ and $(765 - 675)/45 = 2$ give the same approximation. Both answers are approximations. The exact probability under the exponential model is about $0.9547$ .

In the example with $100$ waits from $\operatorname{Exp}(1)$ , a student concludes that a single wait satisfies $P(0.8 < X_1 < 1.2) \approx 0.9545$ . Find the actual probability and explain the mistake.

Show the answer

A single wait has $P(X_1 > t) = e^{-t}$ for $t \ge 0$ , so

\begin{aligned} &P(0.8 < X_1 < 1.2) \\[0.4em] &\quad = e^{-0.8} - e^{-1.2} \\[0.4em] &\quad \approx 0.1481. \end{aligned}

The central limit theorem describes the average of many waits, which has standard error $0.1$ . A single wait keeps its exponential distribution with standard deviation $1$ , whatever the number of waits in the sample, so the population distribution and the sampling distribution of the average must not be confused.

Decide in each case whether the central limit theorem of this lesson justifies a normal approximation for $\bar{X}_n$ when $n$ is large. (a) IID $\operatorname{Uniform}(0, 1)$ observations. (b) Observations $X_1 = \cdots = X_n = W$ with $W \sim \operatorname{Exp}(1)$ . (c) IID observations with density $2/x^3$ for $x \ge 1$ . (d) IID Bernoulli observations with $p = 1$ .

Show the answer

Case (a) qualifies, since the observations are IID with variance $\tfrac{1}{12}$ , which is finite and positive. Case (b) does not, because the observations are not independent. The average equals $W$ for every $n$ , so it keeps the exponential distribution and never becomes approximately normal. Case (c) does not, because the variance is infinite, so there is no $\sigma$ for the standardization, and other limit theorems with different scaling are needed for such sums. Case (d) does not, because the variance is $0$ . The average equals $1$ with probability $1$ , so there is nothing to approximate and the standardization would divide by $0$ .

The Central Limit Theorem

Key terms

Standardized sum

Convergence in distribution

Central limit theorem

Normal approximation for an average

Finite positive variance

Heavy tails

Check yourself

How this lesson connects

Builds on

Used later in

Where this leads in Statistics