Normal Approximations and Their Limits, statistics lesson

Suppose 100 independent trials each succeed with probability $0.5$ . The exact chance of at most 50 successes is a sum of 51 binomial terms, while a normal curve with the same mean and standard deviation gives an estimate in a few steps. How good is that estimate, and when does the shortcut fail? This lesson states the exact model first, builds the matching normal curve, uses a continuity correction to respect integer counts, and measures the error against the exact binomial probability, including a rare-event case where the approximation is poor.

Key terms

Exact binomial probability

Definition Suppose $X$ counts the successes in $n$ independent trials that each succeed with the same probability $p$ , so $X \sim \operatorname{Binomial}(n, p)$ . For each integer $k$ with $0 \le k \le n$ ,

\begin{aligned} &P(X = k) \\[0.4em] &\quad = \binom{n}{k} p^k (1 - p)^{n - k}, \end{aligned}

and the probability of any event about $X$ is the sum of these masses over the integers in the event. For an integer $c$ with $0 \le c \le n$ , for example, $P(X \le c)$ is the sum of the masses for $k = 0, 1, \dots, c$ . This sum is the exact binomial probability. Writing it as a rounded decimal for reporting does not make the method an approximation.

In plain words Under the binomial model the answer is a finite sum, and software computes it directly. The normal approximation is still worth learning, because it shows how counts connect to the normal curve and to later inference. The exact value is exact for the binomial model only, so it is as trustworthy as the assumptions of independent trials with a common $p$ .

Example For $X \sim \operatorname{Binomial}(100, 0.5)$ every sequence of outcomes has probability $2^{-100}$ , so $P(X = 50) = \binom{100}{50}/2^{100} \approx 0.0795892$ . The distribution is symmetric about $50$ , so $P(X \le 49) = P(X \ge 51)$ , and each of these equals half of $1 - P(X = 50)$ . Hence

\begin{aligned} &P(X \le 50) \\[0.4em] &\quad = \tfrac12 + \tfrac12 P(X = 50) \\[0.4em] &\quad \approx 0.5397946 . \end{aligned}

Normal approximation

Definition For $X \sim \operatorname{Binomial}(n, p)$ with $0 < p < 1$ , the normal approximation replaces $X$ by

Y \sim N\bigl(np, np(1 - p)\bigr),

which has the same mean $np$ and the same variance $np(1 - p)$ as $X$ , so its standard deviation is $\sqrt{np(1 - p)}$ . The justification is the de Moivre-Laplace theorem. For fixed $p$ with $0 < p < 1$ and every real $z$ ,

\begin{aligned} &P\Bigl(\frac{X - np}{\sqrt{np(1 - p)}} \le z\Bigr) \\[0.4em] &\quad \to \Phi(z) \end{aligned}

as $n \to \infty$ . The theorem describes a limit and gives no error bound for any particular $n$ .

In plain words The normal curve is matched to the count through the mean and variance of the exact model, not chosen because it looks convenient. As $n$ grows, the histogram of binomial probabilities, with bars of width one, looks more and more like this curve. A common slip is to use $np(1 - p)$ as the standard deviation, when it is the variance.

Example For $X \sim \operatorname{Binomial}(100, 0.5)$ the mean is $np = 50$ and the variance is $np(1 - p) = 25$ , so the approximating variable is $Y \sim N(50, 25)$ , with standard deviation $5$ . Using $25$ as the standard deviation would make the curve five times too wide.

Continuity correction

Definition When an integer-valued count $X$ is approximated by a continuous variable $Y$ , the continuity correction treats each integer $k$ as the interval from $k - 0.5$ to $k + 0.5$ and places the cutoff at the outer edge of the integers in the event. For integers $a$ , $b$ and $k$ ,

\begin{aligned} &P(X \le b) \\[0.4em] &\quad \approx P(Y \le b + 0.5), \\[0.8em] &P(X \ge a) \\[0.4em] &\quad \approx P(Y \ge a - 0.5), \\[0.8em] &P(X = k) \\[0.4em] &\quad \approx P(\lvert Y - k \rvert \le 0.5). \end{aligned}

A strict inequality is first rewritten with the integers it includes, so $X < b$ becomes $X \le b - 1$ and $X > a$ becomes $X \ge a + 1$ .

In plain words Each bar of the binomial histogram has width one and is centered at its count. Shade the bars in the event, then put the normal cutoff at the outer edge of the shaded bars. Drawing the event is safer than memorizing whether to add or subtract one half.

Example For $X \sim \operatorname{Binomial}(100, 0.5)$ and $Y \sim N(50, 25)$ , the event $X \le 50$ includes the whole bar at $50$ , so

\begin{aligned} P(X \le 50) &\approx P(Y \le 50.5) \\[0.4em] &= \Phi(0.1) \\[0.4em] &\approx 0.5398278 . \end{aligned}

The uncorrected value $P(Y \le 50) = 0.5$ cuts the bar at $50$ in half and misses the exact $0.5397946$ by about $0.040$ . For a single count the correction is essential, since $P(Y = 50) = 0$ while $P(49.5 \le Y \le 50.5) \approx 0.0796557$ , close to the exact $P(X = 50) \approx 0.0795892$ .

Success-failure condition

Definition The success-failure condition for a normal approximation to $X \sim \operatorname{Binomial}(n, p)$ asks that the expected numbers of successes and failures, $np$ and $n(1 - p)$ , both be at least a chosen threshold. A common threshold is $10$ , and some texts use $5$ or $15$ . The condition is a rule of thumb for when the binomial shape is close enough to symmetric for the approximation to be worth trying. It does not bound the error of any particular probability.

In plain words When one expected count is small and the other is large, the binomial probabilities pile up near $0$ or near $n$ and trail off in a long tail on the other side. A symmetric normal curve cannot follow that shape, and it puts noticeable area on counts below $0$ or above $n$ that cannot occur. Passing the condition usually makes the middle of the distribution close to normal, but extreme tails still need checking.

Example For $n = 100$ and $p = 0.5$ the expected counts are $50$ and $50$ , and the condition holds easily. For $n = 100$ and $p = 0.01$ they are $1$ and $99$ , and it fails. The exact probabilities of $0$ , $1$ , $2$ and $3$ successes are then about $0.366$ , $0.370$ , $0.185$ and $0.061$ , a shape skewed to the right. The approximating $Y \sim N(1, 0.99)$ has standard deviation about $0.995$ and gives $P(Y < -0.5) \approx 0.066$ , area that the continuity correction assigns to negative counts.

Approximation error

Definition The approximation error of a probability is the approximate value minus the exact value. Its absolute size is the absolute error, and the absolute error divided by a positive exact value is the relative error. Approximation error comes only from replacing the exact distribution with another one. It is separate from model error, the mismatch between the binomial model and the real process, so an approximation can agree closely with the exact binomial answer while the binomial model itself is wrong for the data.

In plain words The size of the error depends on the event as well as on $n$ and $p$ . A tiny absolute error can still be a large relative error when the exact probability is tiny, which matters for rare events. When the exact value can be computed, comparing with it tells you more than any rule of thumb.

Example For $X \sim \operatorname{Binomial}(100, 0.5)$ , the corrected approximation $0.5398278$ to $P(X \le 50)$ exceeds the exact $0.5397946$ by about $0.0000332$ . Now take $n = 100$ and $p = 0.01$ . A count cannot be negative, so the event $X = 0$ is the event $X \le 0$ , and the corrected cutoff is $0.5$ . The exact probability is $0.99^{100} \approx 0.36603$ , while

\begin{aligned} P(Y \le 0.5) &= \Phi\Bigl(\frac{0.5 - 1}{\sqrt{0.99}}\Bigr) \\[0.4em] &\approx 0.30765 . \end{aligned}

The error is about $-0.0584$ , roughly 16 percent of the exact value. The continuity correction aligns the cutoff with the event, but it cannot repair the mismatch in shape.

Check yourself

For $X \sim \operatorname{Binomial}(100, 0.5)$ , approximate $P(X \ge 55)$ with a continuity correction, then compare with the exact binomial value, which software gives as about $0.18410$ .

Show the answer

The event includes the bars at $55, 56, \dots, 100$ , whose outer edge is $54.5$ . With $Y \sim N(50, 25)$ the z-score is $(54.5 - 50)/5 = 0.9$ , so

\begin{aligned} P(X \ge 55) &\approx P(Y \ge 54.5) \\[0.4em] &= 1 - \Phi(0.9) \\[0.4em] &\approx 0.18406 . \end{aligned}

The exact value $0.18410$ is the sum of the binomial masses from $55$ to $100$ . The approximation is low by about $0.00004$ , a close agreement for this event.

A classmate approximates $P(X < 45)$ for $X \sim \operatorname{Binomial}(100, 0.5)$ by $P(Y \le 45.5)$ , where $Y$ is normal with mean $50$ and standard deviation $25$ . Find both mistakes and correct the answer.

Show the answer

The number $25$ is the variance, so the standard deviation is $\sqrt{25} = 5$ . The event $X < 45$ is the event $X \le 44$ , whose bars end at $44.5$ , so $45.5$ wrongly adds the bar at $45$ . The corrected approximation is

\begin{aligned} P(X \le 44) &\approx P(Y \le 44.5) \\[0.4em] &= \Phi(-1.1) \\[0.4em] &\approx 0.13567, \end{aligned}

close to the exact $P(X \le 44) \approx 0.13563$ . The classmate’s version gives $\Phi(-0.18) \approx 0.4286$ , more than three times too large.

For $X \sim \operatorname{Binomial}(100, 0.5)$ the success-failure condition holds with expected counts $50$ and $50$ . Approximate $P(X \ge 70)$ with a continuity correction and compare with the exact value, about $3.93 \times 10^{-5}$ . What does the comparison show?

Show the answer

The cutoff is $69.5$ , with z-score $(69.5 - 50)/5 = 3.9$ , so the approximation is $1 - \Phi(3.9) \approx 4.81 \times 10^{-5}$ . The absolute error, about $8.8 \times 10^{-6}$ , looks negligible, but the approximation is about 23 percent larger than the exact value. Meeting the success-failure condition did not make this extreme tail accurate in relative terms. When a probability this small matters, use the exact binomial calculation, which is available here.

Normal Approximations and Their Limits

Key terms

Exact binomial probability

Normal approximation

Continuity correction

Success-failure condition

Approximation error

Check yourself

How this lesson connects

Builds on

Used later in