Sparse Tables, Fisher's Exact Test and Paired Binary Data, statistics lesson

Two small studies both end in a 2 by 2 table. In the first, all four people in group A have an outcome and none of the four people in group B do, so every expected count from the margins is only 2 and a chi-square approximation is doubtful. In the second, 20 people answer the same yes or no question before and after a change, and their 40 answers are not 40 independent observations. This lesson builds Fisher’s exact test for the first design and McNemar’s test for the second, and it shows why the design, not the shape of the table, decides which test applies.

Key terms

Fixed margins and conditional inference

Definition The margins of a 2 by 2 table are its row totals and column totals. Suppose the rows are two groups of independent units with $n_A$ and $n_B$ members, $N = n_A + n_B$ , the columns record whether an outcome occurred, and $m$ units in total have the outcome. Fixing the margins means comparing the observed table only with the tables that have the same row totals and the same column totals. With the margins fixed, the count $X$ of outcomes in group A determines the other three cells, and its possible values are the integers $x$ with

\begin{gathered} x \ge \max(0,\ n_A + m - N), \\[0.4em] x \le \min(n_A, m). \end{gathered}

Conditional inference bases a test or an interval on the distribution of the data given the observed value of some statistic, here the margins. Suppose each unit in group A has the outcome with probability $p_A$ and each unit in group B with probability $p_B$ , all independently, and the null hypothesis is $H_0\colon p_A = p_B$ . Under $H_0$ , given the margins,

\begin{aligned} &P(X = x \mid \text{margins}) \\[0.4em] &\quad = \frac{\binom{m}{x}\binom{N - m}{n_A - x}}{\binom{N}{n_A}}, \end{aligned}

which is the hypergeometric distribution with population size $N$ , $m$ marked items and sample size $n_A$ . The unknown common probability cancels, so this null distribution contains no unknown parameter.

In plain words The margins say how many people were in each group and how many had the outcome overall. Conditioning on them sets those numbers aside and asks only how the outcomes were split between the groups. If the groups do not differ, then once we know that $m$ of the $N$ people had the outcome, every set of $m$ people is equally likely to be that set, so group A is like a random sample of $n_A$ people from the $N$ . In a two-group study the group sizes are usually fixed by the design while the outcome total is random, and the test conditions on that total anyway. That choice is what makes the inference conditional.

Example In the first study $n_A = n_B = 4$ , $m = 4$ and $N = 8$ , so $X$ can be any integer from $0$ to $4$ . There are $\binom{8}{4} = 70$ sets of four people that could carry the outcome, all equally likely under $H_0$ , and the set gives $X = x$ in $\binom{4}{x}\binom{4}{4 - x}$ of them.

$x$	$0$	$1$	$2$	$3$	$4$
Count of sets	$1$	$16$	$36$	$16$	$1$
$P(X = x \mid \text{margins})$	$\tfrac{1}{70}$	$\tfrac{16}{70}$	$\tfrac{36}{70}$	$\tfrac{16}{70}$	$\tfrac{1}{70}$

The probabilities add to $\tfrac{70}{70} = 1$ . The observed table has $x = 4$ , and its conditional probability under $H_0$ is $\tfrac{1}{70}$ .

Fisher’s exact test

Definition Fisher’s exact test applies to a table of counts from independent units, and this lesson treats the 2 by 2 case. Its null hypothesis depends on the design. For two independent groups it states that both groups have the same outcome probability, and for a single sample classified in two ways it states that the two classifications are independent. Under either null, the count $X$ has the hypergeometric distribution of the previous card given the margins. In a randomized experiment that assigns $n_A$ of the $N$ units to group A completely at random, the same distribution follows from the null hypothesis that the treatment changes no unit’s outcome, and then the randomness comes from the assignment. The p-value is the conditional probability under $H_0$ of the tables with the observed margins that are at least as extreme as the observed table. Against the one-sided alternative that group A has the higher outcome probability it is

p = P(X \ge x_{\text{obs}} \mid \text{margins}),

and the next card gives the two-sided rules. The word exact means that the null distribution is computed exactly instead of being approximated by a large-sample distribution. A test that rejects $H_0$ when $p \le \alpha$ then has a probability of a type I error of at most $\alpha$ , and because the distribution is discrete this probability is often below $\alpha$ .

In plain words The test asks how surprising the observed split of outcomes would be if group membership had nothing to do with the outcome. Exact describes the arithmetic, not the study. The test still needs independent units, a design that matches the stated null and a stated rule for two-sided p-values, and it cannot repair a biased sample. It is not the only exact method either. Unconditional exact tests such as Barnard’s test do not condition on the outcome total and can give different p-values for the same table.

Example In the first study the observed count is $x_{\text{obs}} = 4$ . The one-sided p-value is $P(X \ge 4 \mid \text{margins}) = \tfrac{1}{70} \approx 0.0143$ . The two-sided p-value also counts the equally improbable table $x = 0$ in the other direction, giving $\tfrac{2}{70} = \tfrac{1}{35} \approx 0.0286$ . For comparison, every expected count is $4 \cdot 4 / 8 = 2$ , the Pearson chi-square statistic equals $8$ , and the $\chi^2_1$ approximation gives a p-value of about $0.0047$ , roughly six times smaller than the exact value. The sample odds ratio $(4 \cdot 4)/(0 \cdot 0)$ has a zero denominator and is infinite. The finite p-value measures how surprising the table is under $H_0$ , and it does not say how large the difference between the groups is.

Two-sided exact p-value

Definition For a discrete null distribution there is more than one reasonable two-sided p-value, so a report names the rule it uses. Write $f(x) = P(X = x \mid \text{margins})$ under $H_0$ . The probability-ordering rule, a widely used rule for Fisher’s test, adds the null probabilities of every table with the same margins that is no more probable than the observed table,

p = \sum_{x \,:\, f(x) \le f(x_{\text{obs}})} f(x).

The doubling rule takes twice the smaller one-sided tail probability and caps the result at $1$ ,

\begin{aligned} p &= \min\bigl(1,\ 2\min(L, U)\bigr), \\[0.4em] L &= P(X \le x_{\text{obs}}), \\[0.4em] U &= P(X \ge x_{\text{obs}}), \end{aligned}

with both tails computed from the same conditional null distribution. When that distribution is symmetric, as in the first study, the two rules give the same value. In general they can differ. Whether to use a one-sided or a two-sided p-value is decided before looking at the data.

In plain words Two-sided means that a lopsided split in either direction counts as evidence. The rules differ in how they decide which tables on the other side are at least as extreme. The probability-ordering rule uses the null probability itself as the measure of extremeness, while the doubling rule treats the other side as if it carried as much probability as the observed tail. Picking whichever rule gives the smaller p-value after seeing the data can make the reported p-value too small.

Example All three people in group A have the outcome and none of the four people in group B do, so $n_A = 3$ , $n_B = 4$ , $m = 3$ and $x_{\text{obs}} = 3$ . There are $\binom{7}{3} = 35$ sets of three people.

$x$	$0$	$1$	$2$	$3$
$f(x)$	$\tfrac{4}{35}$	$\tfrac{18}{35}$	$\tfrac{12}{35}$	$\tfrac{1}{35}$

Only the observed table has probability at most $\tfrac{1}{35}$ . The most lopsided table in the other direction, with all three outcomes in group B, has probability $\tfrac{4}{35}$ , so the probability-ordering rule adds nothing from that side and gives $p = \tfrac{1}{35} \approx 0.0286$ . The doubling rule gives $2 \cdot \tfrac{1}{35} = \tfrac{2}{35} \approx 0.0571$ . At the $0.05$ level the two rules reach different conclusions for the same table, which is why the rule is fixed in advance and reported.

Paired binary data and discordant pairs

Definition Paired binary data consist of $n$ independent units, each giving two binary outcomes that belong together, such as one person’s answer before and after a change or the outcomes of the two members of a matched pair. The data are summarized by classifying each pair by both outcomes, so the table counts pairs, not single answers.

	Second yes	Second no	Total
First yes	$a$	$b$	$a + b$
First no	$c$	$d$	$c + d$
Total	$a + c$	$b + d$	$n$

A pair is concordant if its two outcomes agree, as in cells $a$ and $d$ , and discordant if they differ, as in cells $b$ and $c$ . The sample proportions of yes at the two measurements are $\hat{p}_1 = (a + b)/n$ and $\hat{p}_2 = (a + c)/n$ , so

\hat{p}_2 - \hat{p}_1 = \frac{c - b}{n}.

In plain words The unit is the pair. Twenty people measured twice give twenty independent pairs, not forty independent observations. The two proportions come from the same people, so the design does not provide two independent samples, and the standard error for two independent proportions ignores the pairing. A concordant pair adds the same amount to both proportions, so the concordant cells drop out of the difference, although they still count in $n$ .

Example In the second study the rows are the answers before the change and the columns the answers after it.

	After yes	After no	Total
Before yes	$8$	$1$	$9$
Before no	$7$	$4$	$11$
Total	$15$	$5$	$20$

The independent units are the 20 people. There are $8 + 4 = 12$ concordant pairs and $1 + 7 = 8$ discordant pairs. The yes rate is $\tfrac{9}{20} = 0.45$ before and $\tfrac{15}{20} = 0.75$ after, and the difference $(7 - 1)/20 = 0.30$ comes entirely from the discordant pairs.

Marginal homogeneity

Definition For paired binary data, let $p_{11}$ , $p_{12}$ , $p_{21}$ and $p_{22}$ be the population probabilities that a pair falls in each cell, where the first index refers to the first measurement, the second index to the second measurement, and index 1 means yes. Marginal homogeneity means that the probability of yes is the same at both measurements,

p_{11} + p_{12} = p_{11} + p_{21}.

In a 2 by 2 table this holds exactly when $p_{12} = p_{21}$ , so the two kinds of discordant pair are equally likely. It is a statement about the margins and is a different property from independence of the two outcomes within a pair.

In plain words Marginal homogeneity means no net change. Some units may switch from no to yes and others from yes to no, but in the population the two kinds of switch are equally common, so the overall yes rate stays the same. Paired tables often show association, because a person who says yes once often says yes again, and that association is a separate question.

Example A population with $p_{11} = 0.4$ , $p_{12} = 0.1$ , $p_{21} = 0.1$ and $p_{22} = 0.4$ has a yes rate of $0.5$ at both measurements, so marginal homogeneity holds. Its outcomes are strongly associated, since $P(\text{second yes} \mid \text{first yes}) = 0.8$ while $P(\text{second yes} \mid \text{first no}) = 0.2$ . A population with $p_{11} = 0.4$ , $p_{12} = 0.1$ , $p_{21} = 0.4$ and $p_{22} = 0.1$ is the opposite case. Each cell equals the product of its margins, so the two outcomes are independent, yet the yes rate rises from $0.5$ to $0.8$ . For the 20 people above, Fisher’s exact test applied to the paired table asks the association question and gives a two-sided p-value of about $0.32$ under the probability-ordering rule. That test does not address whether the population yes rate changed, which is the marginal homogeneity question. Fisher’s test conditions on both margins, so it takes the observed yes rates $0.45$ and $0.75$ as given.

McNemar test

Definition McNemar’s test uses $n$ independent pairs, each falling in the four cells with the same probabilities $p_{ij}$ , to test $H_0\colon p_{12} = p_{21}$ , which is marginal homogeneity. Let $B$ and $C$ be the random counts in the yes then no cell and the no then yes cell, with observed values $b$ and $c$ , and let $n_d = b + c$ be the number of discordant pairs. Given $n_d \ge 1$ , each discordant pair is independently of the yes then no kind with probability $p_{12}/(p_{12} + p_{21})$ , which equals $\tfrac12$ under $H_0$ . So under $H_0$

\begin{aligned} &B \mid (B + C = n_d) \\[0.4em] &\quad \sim \operatorname{Binomial}(n_d, \tfrac12), \end{aligned}

and the exact two-sided p-value is

\begin{aligned} p &= \min\bigl(1,\ 2P(B \le k)\bigr), \\[0.4em] k &= \min(b, c), \end{aligned}

with the probability computed from this binomial distribution. Because this distribution is symmetric, the doubling rule and the probability-ordering rule give the same value. The concordant counts $a$ and $d$ do not enter, and with no discordant pairs the p-value is $1$ . A large-sample version compares

X^2 = \frac{(b - c)^2}{b + c}

with the $\chi^2_1$ distribution, and some versions first subtract $1$ from $\lvert b - c \rvert$ as a continuity correction. That approximation needs many discordant pairs.

In plain words Only the units that changed carry information about the direction of change. Under no net change, each unit that changed was equally likely to change either way, like one fair coin toss per changer, and the test asks whether the split of the changers is too lopsided for fair coins. The units that did not change affect the size of the difference in proportions but not the test.

Example For the 20 people, $b = 1$ and $c = 7$ , so $n_d = 8$ . Under $H_0$ the count $B$ is $\operatorname{Binomial}(8, \tfrac12)$ given the eight discordant pairs, and $P(B \le 1) = (1 + 8)/2^8 = \tfrac{9}{256}$ , so

\begin{aligned} p &= 2 \cdot \frac{9}{256} = \frac{18}{256} \\[0.4em] &= 0.0703125. \end{aligned}

At a 5 percent level chosen in advance, the test does not reject marginal homogeneity. That is not evidence of no change. The observed yes rate rose by $0.30$ , and only eight pairs carry information about the direction. The large-sample statistic $X^2 = (1 - 7)^2/8 = 4.5$ would give a p-value of about $0.034$ , and the continuity-corrected value $(6 - 1)^2/8 = 3.125$ gives about $0.077$ . With eight discordant pairs the approximations disagree with each other and with the exact value, so the exact value is the one to report.

Two changes show what the test responds to. If the eight discordant pairs split 2 and 6 instead of 1 and 7, the exact p-value becomes $2(1 + 8 + 28)/256 = \tfrac{74}{256} \approx 0.289$ . If instead the original table gains 100 more people who say yes both times, the exact p-value stays $0.0703125$ , while the difference in yes rates falls to $\tfrac{6}{120} = 0.05$ .

Check yourself

All three people in group A have an outcome and none of the three people in group B do. Find the conditional distribution of the count $X$ in group A given the margins, the one-sided and two-sided Fisher p-values, and the smallest two-sided p-value that any table with these margins could give.

Show the answer

The margins are $n_A = n_B = 3$ and $m = 3$ , so $N = 6$ and there are $\binom{6}{3} = 20$ equally likely sets of three people to carry the outcome under $H_0$ . The count $X = x$ arises in $\binom{3}{x}\binom{3}{3 - x}$ of them, which gives $1$ , $9$ , $9$ and $1$ sets for $x = 0, 1, 2, 3$ . The conditional probabilities are $\tfrac{1}{20}$ , $\tfrac{9}{20}$ , $\tfrac{9}{20}$ and $\tfrac{1}{20}$ .

The observed table has $x = 3$ . The one-sided p-value is $P(X \ge 3) = \tfrac{1}{20} = 0.05$ . The tables no more probable than the observed one are $x = 0$ and $x = 3$ , so the two-sided p-value is $\tfrac{2}{20} = 0.10$ , and doubling gives the same value because the distribution is symmetric. The observed table is already the most extreme one, so no table with these margins gives a two-sided p-value below $0.10$ . A two-sided test at the $0.05$ level can never reject with these margins, and a nonsignificant result here says very little about whether the groups differ.

Thirty people answer a yes or no question before and after a training session. Twelve say yes both times, 6 say no both times, 10 change from no to yes and 2 change from yes to no. Which counts enter the exact McNemar test? Find the two yes rates, their difference and the exact two-sided p-value.

Show the answer

The 30 people are the independent units. Only the $2 + 10 = 12$ discordant pairs enter the test. The yes rate is $\tfrac{14}{30} \approx 0.467$ before and $\tfrac{22}{30} \approx 0.733$ after, and the difference is $(10 - 2)/30 = \tfrac{8}{30} \approx 0.267$ . Under $H_0$ the count $B$ of yes then no pairs is $\operatorname{Binomial}(12, \tfrac12)$ given the 12 discordant pairs, and the smaller count is $2$ , so

\begin{aligned} p &= 2 \cdot \frac{1 + 12 + 66}{2^{12}} \\[0.4em] &= \frac{79}{2048} \approx 0.0386. \end{aligned}

At a 5 percent level chosen in advance, the test rejects marginal homogeneity. The 12 people who said yes both times and the 6 who said no both times change the yes rates and $n$ , but not the p-value. The test is about whether the population yes rate changed. A before and after design without a comparison group cannot separate the effect of the training from anything else that happened between the two answers.

A report on 20 people gives only the before and after yes counts, 9 and 15, and analyzes them with Fisher’s exact test as two independent groups of 20. What is wrong, and can McNemar’s test be computed from these counts?

Show the answer

Each of the 20 people appears in both groups, so the 40 answers are 20 pairs. Fisher’s hypergeometric distribution assumes 40 independent units, and the two-sided p-value of about $0.105$ that this analysis produces belongs to a design that did not happen.

McNemar’s test needs the discordant counts, and the yes counts do not determine them. With $a + b = 9$ , $a + c = 15$ and $a + b + c + d = 20$ , the counts satisfy $c = b + 6$ , and every value of $b$ from $0$ to $5$ fits.

$b$	$0$	$1$	$2$	$3$	$4$	$5$
$c$	$6$	$7$	$8$	$9$	$10$	$11$
Exact $p$	$0.031$	$0.070$	$0.109$	$0.146$	$0.180$	$0.210$

The same yes counts are compatible with exact p-values from about $0.031$ to about $0.210$ . The analysis needs the paired table, which classifies each person by both answers.

Sparse Tables, Fisher's Exact Test and Paired Binary Data

Key terms

Fixed margins and conditional inference

Fisher’s exact test

Two-sided exact p-value

Paired binary data and discordant pairs

Marginal homogeneity

McNemar test

Check yourself

How this lesson connects

Builds on

Used later in