Hypothesis Tests and P-Values

Example With $p$ the new procedure’s long-run success probability and the question whether it beats the benchmark,

The hypotheses are about $p$, not about the observed proportion $16/20 = 0.8$, which is known exactly once the data are in. If the null is written as $p \le 0.5$ instead, the calculations below are done at $p = 0.5$, because among the null values that boundary value makes large counts most likely, which gives the largest tail probability.

Example Let $X$ be the number of successes in $n = 20$ trials. If the trials are independent and each succeeds with probability $0.5$, then $X \sim \operatorname{Binomial}(20, 0.5)$ and

for $k = 0, 1, \dots, 20$. This null distribution has mean $10$. Large counts point toward $p > 0.5$, so here more extreme means larger. The observed value $16$ is compared with this distribution centered at $10$, not with a distribution built from the observed rate $0.8$, which would be centered at $16$. The independence and common probability assumptions are part of the model and are not checked by the count itself.

Example For $16$ successes and $H_1\colon p > 0.5$, the outcomes at least as extreme are the counts $16$ through $20$, so

If the benchmark model were correct, about 0.59 percent of repeated sets of 20 trials would give 16 or more successes. That is evidence against the benchmark model. The probability that $p = 0.5$ given these data is a different quantity, and computing it would need a prior distribution for $p$ that gives the value $0.5$ itself positive probability, as in Bayesian methods.

Example If the question were whether the rate differs from $0.5$ in either direction, then $H_1\colon p \ne 0.5$. Ordering outcomes by distance from the null mean $10$, the observed $16$ is $6$ away, so the counts at most $4$ and at least $16$ are at least as extreme. The null distribution is symmetric about $10$, so the two tails are equal and

Doubling is not automatic. For $X \sim \operatorname{Binomial}(10, 0.3)$ and an observed $6$, ordering by distance from the mean $3$ gives about $0.0756$, while doubling the upper tail $P(X \ge 6) \approx 0.0473$ gives about $0.0947$.

Example At $\alpha = 0.05$ the one-sided p-value $0.00591$ and the two-sided p-value $0.0118$ are both at most $0.05$, so each test, planned in advance, rejects $H_0$. The upper-tail test rejects exactly when $X \ge 15$, because $P(X \ge 15) \approx 0.0207$ is at most $0.05$ while $P(X \ge 14) \approx 0.0577$ is not. Its actual probability of rejecting a true null is therefore about $0.0207$, below the nominal $0.05$, because the attainable tail probabilities of a count jump in steps.

Example Suppose a test statistic $Z$ has a standard normal distribution under $H_0$. An analyst who looks at the sign of $Z$, picks the matching one-sided test and rejects when its p-value is at most $0.05$ is really using the rule that rejects when $\lvert Z \rvert \ge z_{0.95}$, where $z_{0.95} \approx 1.645$ is the $0.95$ quantile of the standard normal distribution. Under $H_0$ that happens with probability $2 \times 0.05 = 0.10$, twice the stated level.