Type I Error, Type II Error and Power, statistics lesson

Even a correctly run test can reach the wrong decision. It can reject a null hypothesis that is true, or fail to reject one that is false. A team planning an experiment wants to know how likely each mistake is before any data are collected, and above all whether the study has a fair chance of detecting an improvement large enough to be useful. This lesson defines the two errors, computes power for a simple test at two effect sizes, and turns the calculation around to choose a sample size.

Key terms

Type I error

Definition A type I error is rejecting $H_0$ when $H_0$ is true. A test has significance level $\alpha$ when, under the model, its probability of rejecting is at most $\alpha$ for every parameter value in $H_0$ . So the probability of a type I error is at most $\alpha$ , and it can be smaller, for example for a discrete test statistic.

In plain words A type I error is a false alarm, and $\alpha$ caps how often a true null sets one off. The level is a probability computed by assuming the null is true. It is not the probability that a particular significant result is a false alarm. That second probability also depends on how often the tested nulls are true and on the power against the real effects. Suppose $1000$ hypotheses are tested, $900$ of them true nulls whose tests reject with probability exactly $0.05$ and $100$ with real effects that the tests detect with probability $0.8$ . The expected numbers of rejections are $45$ among the true nulls and $80$ among the real effects, so in the long run about 36 percent of the rejections are type I errors.

Example The example test of this lesson uses $n = 25$ independent observations from a normal distribution with unknown mean $\mu$ and known standard deviation $\sigma = 10$ , and tests $H_0\colon \mu = 0$ against $H_1\colon \mu > 0$ at $\alpha = 0.05$ . The sample mean $\bar{X}$ has standard error $\sigma/\sqrt{n} = 2$ . With $z_q$ the $q$ quantile of the standard normal distribution, so that $z_{0.95} \approx 1.6449$ , the test rejects when the observed mean $\bar{x}$ exceeds

c = z_{0.95} \cdot 2 \approx 3.2897 .

If $\mu = 0$ , then $\bar{X} \sim N(0, 4)$ and $P(\bar{X} > c) = 0.05$ exactly, so this test makes a type I error with probability $0.05$ .

Type II error

Definition A type II error is failing to reject $H_0$ when $H_0$ is false. Its probability depends on which value in $H_1$ is true, so it is always computed at a specified alternative value $\theta_1$ . It is written $\beta(\theta_1)$ , the probability that the test fails to reject $H_0$ when $\theta_1$ is the true parameter value. A subscript such as $P_{\theta_1}$ marks a probability computed with that true value.

In plain words A type II error is a missed effect. A test does not have one type II error rate. It has one for each possible true effect, and a test like the example below misses small effects more often than large ones.

Example For the example test, if the true mean is $2$ , then $\bar{X} \sim N(2, 4)$ and

\begin{aligned} \beta(2) &= P_{2}(\bar{X} \le c) \\[0.4em] &= \Phi\Bigl(\frac{c - 2}{2}\Bigr) \\[0.4em] &\approx \Phi(0.6449) \\[0.4em] &\approx 0.7405 . \end{aligned}

If the true mean is $5$ , then $\beta(5) = \Phi\bigl((c - 5)/2\bigr) \approx \Phi(-0.8551) \approx 0.1962$ . The same test misses a 2-unit effect about 74 percent of the time and a 5-unit effect about 20 percent of the time.

Power

Definition The power of a test at an alternative value $\theta_1$ is the probability of rejecting $H_0$ when $\theta_1$ is the true value, which is $1 - \beta(\theta_1)$ . The probability of rejecting, viewed as a function of the true parameter value, is the power function of the test. At values in $H_0$ it gives the type I error probability. Power depends on the true effect, the variability of the data, the sample size, the level $\alpha$ , whether the test is one-sided or two-sided, and the design and model. For the one-sided z test of $H_0\colon \mu = 0$ at level $0.05$ with $n$ observations, as in the example, the power at a true mean $\mu$ is

1 - \Phi\Bigl(z_{0.95} - \frac{\mu\sqrt{n}}{\sigma}\Bigr).

In plain words Power is the chance that the study detects an effect of a stated size if that effect is real. It always refers to a specific effect. For the example test at a true mean above $0$ , a larger sample, less noisy measurements, a larger true effect or a larger $\alpha$ each raise it, though a larger $\alpha$ also means more false alarms. Its power function equals $\alpha$ at $\mu = 0$ and drops below $\alpha$ when $\mu < 0$ , since a one-sided test is not looking in that direction.

Example For the example test the power is $1 - 0.7405 \approx 0.2595$ at $\mu = 2$ and $1 - 0.1962 \approx 0.8038$ at $\mu = 5$ , with the same rejection rule, sample size and noise. A simulation that draws 100,000 sample means from each alternative and applies the rule gives rejection frequencies close to these values, with Monte Carlo standard errors of about $0.0014$ and $0.0013$ . The simulation checks the arithmetic of the model. It is not evidence that a real effect exists.

Power analysis

Definition A power analysis, done when a study is planned, relates the significance level, sample size, variability, effect size and power for a stated test and model. It is most often used to find the sample size that gives a target power $1 - \beta$ at an effect size $\delta$ chosen because it matters in practice. For the one-sided test of $H_0\colon \mu = \mu_0$ against $H_1\colon \mu > \mu_0$ with independent normal observations and known $\sigma$ , and a target power $1 - \beta$ larger than $\alpha$ , the power at $\mu_0 + \delta$ with $\delta > 0$ is at least $1 - \beta$ exactly when $n$ is at least

\biggl(\frac{\sigma\,(z_{1-\alpha} + z_{1-\beta})}{\delta}\biggr)^{2}.

The planned sample size is this number rounded up to a whole number.

In plain words Decide in advance which effect would be worth detecting, then choose a sample large enough to detect it with high probability. The formula belongs to this one test. Unknown variance, a two-sided alternative, unequal groups, clustering or dropouts each call for a different calculation. Power computed after the study by plugging in the observed estimate, often called observed power, adds no information beyond the p-value. For the one-sided test above it is a decreasing function of the p-value alone and equals $0.5$ exactly when the p-value equals $\alpha$ . After the study, a confidence interval for the effect is the useful summary.

Example Take $\delta = 2$ , $\sigma = 10$ , $\alpha = 0.05$ and target power $0.80$ . With $z_{0.80} \approx 0.84162$ the quantile sum is $z_{0.95} + z_{0.80} \approx 2.48647$ , and

\begin{aligned} n &\approx \Bigl(\frac{10 \cdot 2.48647}{2}\Bigr)^{2} \\[0.4em] &\approx 154.56 , \end{aligned}

so the plan calls for $155$ observations. With $155$ observations the power at $\delta = 2$ is about $0.801$ , while $154$ would give about $0.799$ .

Minimum detectable effect

Definition For a fixed design, sample size, level $\alpha$ and target power $1 - \beta$ , the minimum detectable effect is the smallest effect size at which the power of the test reaches $1 - \beta$ . For the one-sided test with known $\sigma$ described above,

\delta_{\text{MDE}} = \frac{(z_{1-\alpha} + z_{1-\beta})\,\sigma}{\sqrt{n}} .

In plain words The minimum detectable effect describes the design, not the data. Effects smaller than it can still produce a rejection, only with probability below the target. It is also a different idea from the smallest effect that would matter in practice, which comes from the subject, not from the sample size. A good plan makes the minimum detectable effect no larger than that practical threshold.

Example With $n = 25$ , $\sigma = 10$ , $\alpha = 0.05$ and target power $0.80$ ,

\begin{aligned} \delta_{\text{MDE}} &\approx 2.48647 \cdot 2 \\[0.4em] &\approx 4.97 \text{ units}. \end{aligned}

A 2-unit effect is below this, and the power there is only about $0.26$ . Because the standard error shrinks like $1/\sqrt{n}$ , quadrupling the sample to $100$ halves the minimum detectable effect to about $2.49$ units.

Optional stopping

Definition Optional stopping means letting the results seen so far decide when data collection ends, for example by testing after each batch of data and stopping at the first significant result, and then analyzing the data as if the sample size had been fixed in advance. When every look uses a z test at the same level, the probability under $H_0$ that at least one of several looks rejects exceeds that level, and it grows as looks are added. With independent, identically distributed observations and a z test at every look, a true null is eventually rejected with probability $1$ if the number of looks is unlimited.

In plain words Each look is another chance for noise to cross the line, and stopping as soon as it does keeps the false alarm and never collects the later data that might have reversed it. The looks are strongly dependent because they share data, so the inflation cannot be found by treating them as independent tests. Sequential designs that plan the looks in advance, with stricter boundaries set so that the overall type I error probability stays at most $\alpha$ , avoid the problem.

Example Apply the example test with $\alpha = 0.05$ after $25$ observations and, if it does not reject, again after $50$ . When the null is true, the probability that at least one of the two looks rejects is about $0.080$ . With looks after $25$ , $50$ , $75$ , $100$ and $125$ observations it is about $0.13$ , well above the $0.05$ the analysis would report.

Check yourself

For the example test ( $n = 25$ , $\sigma = 10$ , $\alpha = 0.05$ , rejection when $\bar{x} > 3.2897$ ), find the power and the type II error probability at a true mean of $4$ .

Show the answer

If $\mu = 4$ , then $\bar{X} \sim N(4, 4)$ , and in standard units the boundary is $(3.2897 - 4)/2 \approx -0.3551$ . The power is

1 - \Phi(-0.3551) \approx 0.639,

and the type II error probability is $\beta(4) \approx 0.361$ .

A study uses the example test, which has power about $0.26$ against a 2-unit effect, and the result is not significant. Does the study rule out a 2-unit effect? Name two design changes that would raise the power at that effect.

Show the answer

No. If the true effect were 2 units, the test would fail to reject about 74 percent of the time, so a nonsignificant result is entirely ordinary under that effect. An interval estimate shows which effect sizes remain compatible with the data. More independent observations raise the power, and with $n = 100$ the power at $2$ units is about $0.639$ . More precise measurement also raises it, since halving $\sigma$ to $5$ at $n = 25$ gives the same $0.639$ . A more efficient design such as pairing can help as well. Raising $\alpha$ would increase power too, but only by accepting more false alarms.

Under the same model ( $\sigma = 10$ , one-sided, $\alpha = 0.05$ ), how many observations give power at least $0.90$ against a 5-unit effect? Use $z_{0.90} \approx 1.28155$ .

Show the answer

The quantile sum is $z_{0.95} + z_{0.90} \approx 1.64485 + 1.28155 = 2.92640$ , so the planning formula gives

\begin{aligned} n &\approx \Bigl(\frac{10 \cdot 2.92640}{5}\Bigr)^{2} \\[0.4em] &\approx 34.26 , \end{aligned}

so the plan needs $35$ observations. A check confirms it, since the power at $\delta = 5$ is about $0.905$ with $35$ observations and about $0.898$ with $34$ . Rounding down would miss the target.

Type I Error, Type II Error and Power

Key terms

Type I error

Type II error

Power

Power analysis

Minimum detectable effect

Optional stopping

Check yourself

How this lesson connects

Builds on

Used later in