The Central Limit Theorem

Definition Let $X_1, X_2, \dots$ be independent and identically distributed random variables with mean $\mu$ and finite positive variance $\sigma^2$, and let $\bar X_n$ be the mean of the first $n$ of them. Form the standardized sample mean

The central limit theorem says that for every real number $z$

where $\Phi$ is the cumulative distribution function of the standard normal distribution $N(0, 1)$. In other words, the distribution of $Z_n$ converges to $N(0, 1)$. The statement is about the distribution of the mean over repeated samples of size $n$. It says nothing about the shape of the observations inside one sample, and as a limit it names no sample size at which the approximation becomes good. Independence is a real condition. When observations are positively correlated, as measurements taken in related groups often are, the true standard error of the mean is larger than $\sigma/\sqrt{n}$, and this version of the theorem does not apply. Other versions allow observations that are not identically distributed or are weakly dependent, but they replace these conditions with others.

Definition For the mean $\bar X_n$ of $n$ independent observations with mean $\mu$ and finite positive SD $\sigma$, the standardized sample mean is $Z_n$ above. It measures the deviation of $\bar X_n$ from $\mu$ in units of the standard error

Since $E(\bar X_n) = \mu$ and $\operatorname{Var}(\bar X_n) = \sigma^2/n$, the standardized mean has mean $0$ and variance $1$ for every $n$. With the sum $S_n = X_1 + \dots + X_n$ the same quantity is

Standardizing is needed for a limit with a shape. By the law of large numbers the distribution of $\bar X_n$ itself piles up at $\mu$ as $n$ grows, so its limit is a single point. The standardized mean keeps its spread at $1$, which lets its shape be compared with one fixed curve. The law of large numbers says where the mean settles, and the central limit theorem describes the shape and size of its fluctuations around that point. Neither guarantees that one particular sample mean is close to $\mu$.

Definition The condition $0 < \sigma^2 < \infty$ on the population variance. A finite variance means $E(X^2) < \infty$, which also makes the mean finite. The condition rules out two kinds of populations. If $\sigma^2 = 0$, every observation equals $\mu$ with probability $1$, so $\bar X_n = \mu$ exactly and the standardized mean is undefined. If the variance is infinite, there is no $\sigma$ to standardize with, and the fluctuations of the mean need not be approximately normal at any sample size.

Definition Under the conditions of the theorem and for a fixed, large $n$, the normal approximation treats $\bar X_n$ as approximately $N(\mu, \sigma^2/n)$, so that

where $z_a = (a - \mu)/(\sigma/\sqrt{n})$ and $z_b = (b - \mu)/(\sigma/\sqrt{n})$ are the standardized endpoints. Its accuracy at a given $n$ depends on the population, especially its skewness and tails, and on the event. A central interval can be approximated very well while a far-tail probability at the same $n$ is off by a factor of two or more. No single sample size, such as 30, guarantees a good approximation for every population. When the population itself is normal and the observations are independent, $\bar X_n$ has the distribution $N(\mu, \sigma^2/n)$ exactly for every $n \ge 1$, so no limit is needed. When $\sigma$ is unknown and replaced by the sample SD $s$, the calculation changes, and later lessons treat that case.

Definition Informally, a distribution is heavy-tailed when values far from its center keep much more probability than a normal distribution would give them. A common precise version for the right tail requires that, for every $t > 0$, the product $e^{tx} P(X > x)$ is unbounded as $x$ grows. Equivalently, there are no constants $C$ and $t > 0$ for which

holds for every $x$, so no exponential function bounds the tail probability. In this sense the exponential distribution is not heavy-tailed, although its right tail is heavier than a normal tail. Power-law tails, such as the Pareto tail $P(X > x) = x^{-\alpha}$ for $x \ge 1$ with $\alpha > 0$, are the standard example. This Pareto distribution has a finite mean exactly when $\alpha > 1$ and a finite variance exactly when $\alpha > 2$. Independent draws from a heavy-tailed population with finite positive variance still satisfy the central limit theorem, but the normal approximation may need a much larger sample. With infinite variance the elementary theorem does not apply.