Planned Contrasts, Post-Hoc Comparisons and Multiple Testing

Definition For population means $\mu_1, \dots, \mu_k$, a contrast is a combination

with coefficients $c_j$ that are not all zero. It is estimated by $\hat{L} = \sum_j c_j\bar{y}_j$. Under the one-way ANOVA model, with independent normal errors of common variance, the estimated standard error and the test statistic for $H_0\colon L = 0$ are

and $t$ has a t distribution with $N - k$ degrees of freedom when $L = 0$. A planned contrast is chosen from the research question before the data are seen and is recorded in the analysis plan. A post-hoc comparison is chosen after looking at the results, for example the pair of groups whose means differ most. An ordinary p-value or interval for a post-hoc comparison ignores how it was selected. A valid post-hoc analysis uses a method that covers every comparison that could have been selected, and otherwise the result is reported as exploratory.

Definition A family is the set of hypotheses whose conclusions are reported and interpreted together, fixed before the results are seen. If $V$ is the number of true null hypotheses in the family that are rejected, the family-wise error rate is

the probability of at least one false rejection. A procedure controls the FWER at level $\alpha$ if this probability is at most $\alpha$ whichever of the null hypotheses are true. When $m$ true null hypotheses are each tested with rejection probability exactly $\alpha$ and the tests are independent, the FWER is $1 - (1 - \alpha)^m$. Without independence the exact value depends on the dependence, and it is at most $\min(1, m\alpha)$.

Definition For a family of $m$ hypotheses with p-values $p_1, \dots, p_m$, the Bonferroni correction rejects $H_i$ when $p_i \le \alpha/m$, or equivalently when the adjusted p-value $\min(1, mp_i)$ is at most $\alpha$. Suppose each p-value is valid, meaning that $P(p_i \le u) \le u$ for every $u$ in $[0, 1]$ when $H_i$ is true. If $I_0$ is the set of the $m_0$ true null hypotheses, the union bound gives

with no assumption about dependence between the tests. In the same way, $m$ confidence intervals, each with confidence level $1 - \alpha/m$, all cover their parameters simultaneously with probability at least $1 - \alpha$.

Definition Order the p-values as $p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}$, with matching hypotheses $H_{(1)}, \dots, H_{(m)}$. Holm’s procedure compares $p_{(i)}$ with

starting at $i = 1$. It rejects $H_{(1)}, H_{(2)}, \dots$ in turn while each p-value is at or below its threshold, and stops at the first one that is above. That hypothesis and all later ones are not rejected, and if no p-value fails, all are rejected. With valid p-values and any dependence, Holm controls the FWER at $\alpha$, and it rejects every hypothesis that Bonferroni rejects, sometimes more. The Holm-adjusted p-values are the running maxima of $\min\bigl(1, (m - i + 1)p_{(i)}\bigr)$ taken in order from $i = 1$, and a hypothesis is rejected when its adjusted value is at most $\alpha$.

Definition Let $R$ be the number of rejected hypotheses and $V$ the number of rejected null hypotheses that are actually true. The false discovery proportion is $V/\max(R, 1)$, which is $0$ when nothing is rejected, and the false discovery rate is its expectation

The Benjamini-Hochberg procedure at level $q$ orders the $m$ p-values, finds the largest $i$ with

and rejects $H_{(1)}, \dots, H_{(i)}$, even if some of the smaller p-values were above their own thresholds. If no such $i$ exists, nothing is rejected. When the p-values of the true null hypotheses are independent of each other and of the remaining p-values, or under a certain kind of positive dependence, the procedure keeps the FDR at most $qm_0/m \le q$. Under arbitrary dependence that guarantee can fail, and the Benjamini-Yekutieli variant, which replaces $q$ by $q/(1 + \tfrac12 + \dots + \tfrac1m)$, restores it. When every null hypothesis is true, every rejection is false, so the FDR equals the FWER.

Definition Consider the family of all $k(k - 1)/2$ pairwise differences $\mu_i - \mu_j$ in a one-way design with independent normal errors, a common variance and equal group sizes $n$. Tukey’s procedure gives the intervals

where $\nu = N - k$ and $q_{\alpha;\,k,\,\nu}$ is the upper $\alpha$ point of the studentized range distribution. That is the distribution of the range of $k$ independent standard normal variables divided by an independent $\sqrt{\chi^2_\nu/\nu}$. All the intervals cover their differences simultaneously with probability $1 - \alpha$, so declaring a pair different when its interval excludes $0$ controls the FWER at $\alpha$ for the family of all pairs. With unequal group sizes the Tukey-Kramer version replaces $\sqrt{\text{MS}_{\text{W}}/n}$ by $\sqrt{(\text{MS}_{\text{W}}/2)(1/n_i + 1/n_j)}$, and its joint coverage is at least $1 - \alpha$. The procedure does not need a significant F test first. It covers only the pairwise family, not other contrasts, other outcomes or subgroups, and unequal variances call for a different pairwise method.