One-Proportion Tests, statistics lesson

Sixteen of twenty independently tested components pass a quality check, a sample proportion of 0.8. Before testing, a pass probability of 0.5 was set as the benchmark, and the question was whether the true pass probability is higher. The count is a whole number from 0 to 20, so its distribution under the benchmark can be computed exactly, while the familiar normal approximation is a shortcut that needs conditions. This lesson computes both, explains why the hypothesized value goes into the standard error, and shows why p-values from counts move in steps.

Key terms

Exact binomial test

Definition Suppose the number $X$ of successes comes from $n$ independent trials that share one success probability $p$ , so that $X \sim \operatorname{Binomial}(n, p)$ . To test $H_0: p = p_0$ for a fixed $p_0$ with $0 < p_0 < 1$ , the exact binomial test uses the null distribution $\operatorname{Binomial}(n, p_0)$ itself, with probabilities

f_0(k) = \binom{n}{k} p_0^k (1 - p_0)^{n-k}

for $k = 0, 1, \dots, n$ . For an observed count $x$ , the p-values against $H_1: p > p_0$ and against $H_1: p < p_0$ are

\begin{gathered} P_0(X \ge x) = \sum_{k=x}^{n} f_0(k), \\[0.4em] P_0(X \le x) = \sum_{k=0}^{x} f_0(k), \end{gathered}

where $P_0$ denotes probability computed under $H_0$ . The word exact means that no approximating distribution is used. The result is only as trustworthy as the binomial model of independent trials with a common probability.

In plain words Under the null hypothesis the count has a known distribution, so the p-value is found by adding the probabilities of all counts at least as extreme as the observed one, the observed count included.

Example For $16$ passes in $20$ trials with $p_0 = 0.5$ and the upper alternative, every sequence of $20$ results has null probability $1/2^{20} = 1/1{,}048{,}576$ . The numbers of sequences with $16$ , $17$ , $18$ , $19$ and $20$ passes are $4845$ , $1140$ , $190$ , $20$ and $1$ , which add to $6196$ , so

\begin{aligned} P_0(X \ge 16) &= \frac{6196}{1{,}048{,}576} \\[0.4em] &\approx 0.005909 . \end{aligned}

The test rejects at the $0.05$ level and at the $0.01$ level. If the components came from batches with different pass probabilities, or if failures tended to occur together, the binomial null model would be wrong, and the exact p-value would be an exact answer for the wrong model.

Two-sided exact binomial p-values

Definition For the two-sided alternative $H_1: p \ne p_0$ , a rule must say which counts are at least as extreme as the observed $x$ in either direction. Three common rules give the following p-values.

Doubling the smaller of the two one-sided p-values $P_0(X \le x)$ and $P_0(X \ge x)$ , with the result capped at $1$ .
Adding $f_0(k)$ over all counts $k$ with $f_0(k) \le f_0(x)$ , the counts that are no more likely under the null than the observed one.
Adding $f_0(k)$ over all counts $k$ with $\lvert k - np_0 \rvert \ge \lvert x - np_0 \rvert$ , the counts at least as far from the null mean as the observed one.

When $p_0 = 0.5$ the null distribution is symmetric and the three rules agree. For other values of $p_0$ they can differ, so a report names the rule it used.

In plain words A two-sided test needs a rule for what counts as equally surprising on the other side, and when the null distribution is lopsided, reasonable rules disagree. Doubling a one-sided p-value is one convention among several, not the definition.

Example For $16$ of $20$ with $p_0 = 0.5$ , every rule gives

\begin{aligned} &P_0(X \le 4) + P_0(X \ge 16) \\[0.4em] &\quad \approx 0.011818 . \end{aligned}

Now take $n = 20$ , $p_0 = 0.3$ and $x = 10$ , where the null mean is $6$ . The upper tail is $P_0(X \ge 10) \approx 0.0480$ , so doubling gives about $0.0959$ . The counts no more likely than $10$ are $0$ , $1$ , $2$ and $10$ through $20$ , and their total null probability is about $0.0834$ . The distance rule picks the same counts here, since $\lvert k - 6 \rvert \ge 4$ means $k \le 2$ or $k \ge 10$ . Both values are legitimate two-sided p-values, but they are not interchangeable.

Discrete p-value

Definition A p-value computed from a discrete statistic, such as a count, can take only certain values, at most one for each possible observation. For the upper-tail binomial test the possible p-values are the tail sums $P_0(X \ge c)$ for $c = 0, 1, \dots, n$ . A test that rejects when $p \le \alpha$ then rejects exactly when $x \ge c^*$ , where $c^*$ is the smallest count with $P_0(X \ge c^*) \le \alpha$ , and its actual type I error probability is $P_0(X \ge c^*)$ , which is at most $\alpha$ and often well below it. If even $P_0(X = n) = p_0^n$ exceeds $\alpha$ , no count leads to rejection.

In plain words Moving the observed count from $16$ down to $15$ adds a whole bar of probability to the tail, so the p-value jumps instead of sliding. A test at the $5$ percent level may then use only part of its $5$ percent allowance for false rejections. Each p-value is still an exact tail probability, but only a limited set of values can occur.

Example For $n = 20$ and $p_0 = 0.5$ , the upper tails nearest $0.05$ are $P_0(X \ge 14) \approx 0.0577$ and $P_0(X \ge 15) \approx 0.0207$ . A one-sided test at $\alpha = 0.05$ therefore rejects exactly when $X \ge 15$ , and its actual type I error probability is about $0.0207$ , not $0.05$ . With only $n = 3$ trials and the same $p_0 = 0.5$ , even $3$ passes out of $3$ gives $P_0(X \ge 3) = 0.125$ , so a one-sided test at $0.05$ cannot reject whatever happens.

Null standard error

Definition For the sample proportion $\hat{p} = X/n$ from $n$ independent trials, under $H_0: p = p_0$ the mean of $\hat{p}$ is $p_0$ and its standard deviation is the null standard error

\operatorname{SE}_0(\hat{p}) = \sqrt{\frac{p_0(1 - p_0)}{n}},

computed from the hypothesized value rather than from the data. The estimated standard error $\sqrt{\hat{p}(1 - \hat{p})/n}$ , used in the Wald confidence interval for $p$ , estimates the standard deviation of $\hat{p}$ at the unknown true $p$ instead. The two are different quantities, and putting the estimated one into a test statistic gives a different test.

In plain words A test asks how the sample proportion would vary if the null hypothesis were true, and the null hypothesis already says how much that is. Nothing about the spread has to be estimated.

Example For $n = 20$ and $p_0 = 0.5$ , the null standard error is $\sqrt{0.25/20} \approx 0.1118$ . The estimate-based value $\sqrt{0.8 \times 0.2 / 20} \approx 0.0894$ is smaller, and dividing the difference $0.3$ by it would give $z \approx 3.354$ instead of the $2.683$ of the next card. If all $20$ components had passed, the estimate $\hat{p} = 1$ would make the estimate-based standard error $0$ and the ratio undefined, while the null standard error would stay $0.1118$ .

One-proportion z test

Definition The one-proportion z test, also called the score test for a proportion, tests $H_0: p = p_0$ with the statistic

z = \frac{\hat{p} - p_0}{\sqrt{p_0(1 - p_0)/n}}

and compares it with the standard normal distribution, using $P(Z \ge z)$ , $P(Z \le z)$ or $P(\lvert Z \rvert \ge \lvert z \rvert)$ for the upper, lower and two-sided alternatives. It approximates the exact binomial test and needs the same model of independent trials with a common probability. The approximation is usually judged adequate when the expected numbers of successes and failures under the null, $np_0$ and $n(1 - p_0)$ , are both at least about $10$ , a rule of thumb that does not bound the error. A continuity correction replaces the count $x$ by $x - 0.5$ for the upper tail and by $x + 0.5$ for the lower tail. It often brings the normal value closer to the exact one, with no guarantee.

In plain words The z test replaces the binomial bars with a smooth normal curve that has the null mean and the null standard error. It is a shortcut that works well when the binomial distribution is close to normal in shape, and the exact test remains available when it is not.

Example For $16$ of $20$ with $p_0 = 0.5$ , the expected counts $np_0 = 10$ and $n(1 - p_0) = 10$ meet the rule of thumb, and

\begin{aligned} z &= \frac{0.8 - 0.5}{\sqrt{0.5 \times 0.5 / 20}} \\[0.4em] &\approx 2.683 . \end{aligned}

The upper tail $P(Z \ge 2.683) \approx 0.00365$ is smaller than the exact $0.00591$ . With the continuity correction the count $16$ becomes $15.5$ , and since the null mean of $X$ is $10$ and its null standard deviation is $\sqrt{5}$ ,

\begin{aligned} z_c &= \frac{15.5 - 10}{\sqrt{5}} \\[0.4em] &\approx 2.460, \\[0.4em] P(Z \ge z_c) &\approx 0.00695, \end{aligned}

which is closer to the exact value in this case. The three numbers come from three methods, and a report says which one it gives.

Check yourself

In $10$ independent trials of a task, $9$ succeed. Test $H_0: p = 0.5$ against $H_1: p > 0.5$ , an alternative fixed in advance, with the exact binomial test. Does the test reject at the $0.05$ level, and at the $0.01$ level?

Show the answer

Under the null hypothesis each of the $2^{10} = 1024$ sequences of results has probability $1/1024$ . The counts at least as extreme as $9$ are $9$ and $10$ , reached by $\binom{10}{9} + \binom{10}{10} = 10 + 1 = 11$ sequences, so

\begin{aligned} P_0(X \ge 9) &= \frac{11}{1024} \\[0.4em] &\approx 0.0107 . \end{aligned}

The test rejects at the $0.05$ level but not at the $0.01$ level. The expected count $np_0 = 5$ is below the usual threshold of $10$ , which is one more reason to use the exact test here.

In $100$ independent trials, $60$ succeed, and the two-sided test of $H_0: p = 0.5$ was planned. A classmate computes the z statistic with the standard error $\sqrt{\hat{p}(1 - \hat{p})/n}$ . What is wrong, and what are the correct z statistic and p-value?

Show the answer

The z test measures the distance from $p_0$ in units of the null standard error, so the hypothesized value belongs under the square root. The expected counts $np_0 = 50$ and $n(1 - p_0) = 50$ support the normal approximation, and

\begin{aligned} \operatorname{SE}_0 &= \sqrt{\frac{0.5 \times 0.5}{100}} \\[0.4em] &= 0.05, \\[0.4em] z &= \frac{0.6 - 0.5}{0.05} = 2, \end{aligned}

with the two-sided p-value $P(\lvert Z \rvert \ge 2) \approx 0.0455$ . The classmate’s standard error $\sqrt{0.6 \times 0.4 / 100} \approx 0.0490$ gives $z \approx 2.041$ and $p \approx 0.0412$ , the result of a different test. The exact two-sided binomial p-value, on which all three rules agree because $p_0 = 0.5$ , is about $0.0569$ , and the continuity-corrected approximation gives about $0.0574$ . So near the $0.05$ threshold the decision depends on the method, which is why a report names the method and gives the p-value itself.

A process is supposed to fail with probability $0.01$ . In $20$ independent runs it fails twice. Test $H_0: p = 0.01$ against $H_1: p > 0.01$ , fixed in advance. Should you use the z test or the exact binomial test, and what does the resulting p-value describe?

Show the answer

The expected number of failures under the null is $20 \times 0.01 = 0.2$ , far below any usual threshold, so the null distribution is piled up at $0$ and strongly skewed, and the z test is not appropriate. The exact upper tail is

\begin{aligned} &P_0(X \ge 2) \\[0.4em] &\quad = 1 - 0.99^{20} \\[0.4em] &\qquad - 20(0.01)(0.99)^{19} \\[0.4em] &\quad \approx 0.01686 . \end{aligned}

The z statistic would be $(0.1 - 0.01)/\sqrt{0.01 \times 0.99 / 20} \approx 4.045$ , with a normal upper tail of about $0.000026$ , roughly $650$ times too small. The exact p-value is the probability, under a model of independent runs that each fail with probability $0.01$ , of at least $2$ failures in $20$ runs. It is not the probability that the failure rate is $0.01$ .

One-Proportion Tests

Key terms

Exact binomial test

Two-sided exact binomial p-values

Discrete p-value

Null standard error

One-proportion z test

Check yourself

How this lesson connects

Builds on

Used later in