Simpson's Paradox, math that surprises you lesson

In an invented video game, two players attempt easy levels and hard levels. Player A has the higher success rate on the easy levels and also on the hard levels, yet when every attempt is counted together, player B has the higher success rate by a wide margin. No count in the table is wrong. This page shows that a combined rate is a weighted average of the group rates, explains how unequal weights produce the reversal, and separates what each comparison describes from what it would take to say why the players differ.

Key terms

Simpson’s paradox

Definition Simpson’s paradox occurs when two options, such as two players or two methods, are compared on a success rate, one of them is ahead within every group of the data, and the other is ahead once the groups are combined. In the setting of this page, every attempt by option A or option B falls in exactly one of the groups $1, \dots, K$ , and each option has at least one attempt in each group. The rate of an option in a group is its successes there divided by its attempts there, and its overall rate is its total successes divided by its total attempts. The paradox is that A has the higher rate in every group while B has the higher overall rate, and in a weaker form the two overall rates are merely equal. There is no contradiction. The overall rate of each option is a weighted average of its own group rates, weighted by how its attempts are spread over the groups, so when the two options spread their attempts differently, the overall comparison also reflects that difference in spread. If both options spread their attempts over the groups in the same proportions, the reversal cannot happen. The effect is named after the statistician Edward H. Simpson, who described it in 1951, although Karl Pearson and Udny Yule had pointed it out around 1900.

In plain words A player can be better at easy levels and better at hard levels and still have the worse combined record, if most of that player’s attempts were on hard levels. The combined record mixes how well each player did with which levels each player tried.

Example Here are the invented records, with successes over attempts.

Levels	Player A	Player B
Easy	$9/10 = 90\%$	$80/100 = 80\%$
Hard	$30/100 = 30\%$	$2/10 = 20\%$
All	$39/110 \approx 35.5\%$	$82/110 \approx 74.5\%$

Player A is 10 percentage points ahead on easy levels and 10 percentage points ahead on hard levels. Both players made 110 attempts, and adding the successes gives $9 + 30 = 39$ for A and $80 + 2 = 82$ for B, so B is about 39 percentage points ahead overall. The difference lies in the mix. Of the 110 attempts by A, 100 were on hard levels, while 100 of the 110 attempts by B were on easy levels, and both players succeed far more often on easy levels.

Weighted average

Definition Let $x_1, \dots, x_K$ be numbers and let $w_1, \dots, w_K$ be weights with every $w_k \ge 0$ and a positive total. The weighted average of the numbers with these weights is

\bar{x} = \frac{w_1 x_1 + \cdots + w_K x_K}{w_1 + \cdots + w_K}.

It lies between the smallest and the largest of the numbers that have positive weight. With equal weights it is the ordinary average, and with two different numbers and unequal weights it lies closer to the number with the larger weight. An overall success rate is the weighted average of the group rates with the attempt counts as weights, because a group with $s_k$ successes in $n_k$ attempts has rate $s_k / n_k$ and contributes $n_k \cdot (s_k / n_k) = s_k$ to the numerator.

In plain words Each number counts as often as its weight says. A rate built from 100 attempts should count ten times as much as a rate built from 10 attempts.

Example For player A the weights are 10 easy attempts and 100 hard attempts, so the overall rate is

\frac{10 \cdot 0.9 + 100 \cdot 0.3}{110} = \frac{39}{110},

about $0.355$ , close to the hard rate $0.3$ . The ordinary average $(0.9 + 0.3)/2 = 0.6$ is not an error, but it answers a different question, namely what rate A would have had with equal numbers of easy and hard attempts. If each player made 100 easy and 100 hard attempts at the same rates, A would have $90 + 30 = 120$ successes out of 200, which is 60 percent, and B would have $80 + 20 = 100$ out of 200, which is 50 percent. This is a calculation about a mix that never happened, not a new record of play. More generally, if both players gave the same share $w$ of their attempts to easy levels, with $0 \le w \le 1$ , their overall rates would be

\begin{aligned} r_A &= 0.9w + 0.3(1 - w) \\[0.4em] &= 0.3 + 0.6w, \\[0.4em] r_B &= 0.8w + 0.2(1 - w) \\[0.4em] &= 0.2 + 0.6w. \end{aligned}

The difference $r_A - r_B = 0.1$ is the same for every shared mix, so with equal mixes A always leads by 10 percentage points.

Aggregated data

Definition Aggregated data are counts that have been added over the categories of some variable, so that the variable no longer appears in the table. To aggregate success data, add the successes and add the attempts across the categories, and divide only at the end. Adding or averaging the category percentages does not give the aggregated rate in general. The aggregated rate of each option is the weighted average of its category rates, with that option’s own attempt counts as the weights. An aggregated table describes the totals correctly, but it no longer shows how each option’s attempts were spread over the categories.

In plain words Aggregating throws away the group labels and keeps only the totals. The totals are true, but they cannot tell you whether two options faced the same mix of easy and hard work.

Example After adding over difficulty, the game records shrink to two rows.

Player	Successes	Failures	Attempts
A	$39$	$71$	$110$
B	$82$	$28$	$110$

These totals correctly say that B succeeded on about 74.5 percent of the attempts B actually made, and A on about 35.5 percent of the attempts A made. What they hide is that 100 of the 110 attempts by A were hard and 100 of the 110 attempts by B were easy. That is what pulls the overall rate of A toward its hard rate of 30 percent and the overall rate of B toward its easy rate of 80 percent.

Confounding variable

Definition Suppose two options are compared on an outcome. A confounding variable is a third variable that is related to which option a record belongs to and that also affects the outcome, without itself being a result of the option. When one is present, the aggregated comparison mixes any difference between the options with the effect of the confounding variable. Comparing the options within each category of a measured confounding variable holds that variable fixed, but other confounding variables may remain, and splitting the data by a variable that the option itself affects can create a misleading comparison. So neither always combining nor always splitting is a safe rule. A conclusion about cause needs a study design that supports it, such as assigning the options at random, or clearly stated assumptions about how the data arose.

In plain words A confounding variable is a third factor that pushes on both sides of the comparison at once. Here it influences which levels a player tends to try and also how often an attempt succeeds.

Example In the game, difficulty is related to the player, since about 91 percent of the attempts by A were hard against about 9 percent of the attempts by B. Difficulty also affects success, since both players succeed far more often on easy levels. So difficulty confounds the overall comparison of the players. For a question such as which player did better on levels of the same difficulty, the grouped table answers it, and A is 10 percentage points ahead in each group. For the question which player succeeded on a larger share of the attempts actually made, the aggregated table answers it, and B is ahead. Neither table says why. The players chose their own levels, and the hard levels that A tried may differ from those that B tried, so a claim that A is the more skillful player would need more information, for example from a test in which both players are given the same randomly chosen levels.

Check yourself

Suppose each player made 40 easy and 60 hard attempts, with the same success rates as in the table. Find each player’s number of successes and overall rate. Who is ahead?

Show the answer

Player A would have $0.9 \cdot 40 = 36$ easy successes and $0.3 \cdot 60 = 18$ hard successes, so $54$ out of $100$ , which is 54 percent. Player B would have $0.8 \cdot 40 = 32$ and $0.2 \cdot 60 = 12$ , so $44$ out of $100$ , which is 44 percent.

Player A is ahead by 10 percentage points, the same lead as within each group. With the same mix for both players there is no reversal, which agrees with the formulas $r_A = 0.3 + 0.6w$ and $r_B = 0.2 + 0.6w$ at $w = 0.4$ .

A friend averages each player’s two percentages and gets 60 percent for A and 50 percent for B. Since A wins this way, the friend says the overall rates of 35.5 and 74.5 percent must contain a mistake. What is wrong with the reasoning, and when does the plain average of the two percentages equal a player’s overall rate?

Show the answer

The overall rate weights each group rate by the number of attempts behind it. For A the weights are 10 easy and 100 hard attempts, so the overall rate is $39/110 \approx 35.5\%$ , close to the hard rate. The plain average of 60 percent gives the two groups equal weight, so it is the rate A would have had with equal numbers of easy and hard attempts. Both numbers are correct, and they answer different questions, so there is no mistake to find.

Suppose a player puts a share $w$ of the attempts in a group with rate $x_1$ and the rest in a group with rate $x_2$ . The overall rate minus the plain average is

\begin{aligned} &w x_1 + (1 - w) x_2 - \frac{x_1 + x_2}{2} \\[0.4em] &\quad = \left(w - \tfrac12\right)(x_1 - x_2). \end{aligned}

So the two agree exactly when $w = \tfrac12$ , meaning equal numbers of attempts in the two groups, or when $x_1 = x_2$ . For A the easy share is $w = 10/110 = 1/11$ , and the formula gives $0.6 + (1/11 - 1/2)(0.6) = 39/110$ , as it should.

Keep the four success rates of the table and keep the attempts of A as they are. Suppose B puts a share $e$ of its attempts on easy levels and the rest on hard levels. How large must $e$ be for B to have the higher overall rate, even though B is behind in both groups?

Show the answer

The overall rate of B is $0.8e + 0.2(1 - e) = 0.2 + 0.6e$ , and the overall rate of A is $39/110$ . Player B is ahead overall when

0.2 + 0.6e > \frac{39}{110},

that is, when $0.6e > 17/110$ , or $e > 17/66 \approx 0.258$ . An easy share of 25 percent gives B a rate of $0.35$ , just below $39/110 \approx 0.3545$ , while 30 percent gives $0.38$ , already ahead.

The easy share of A is $f = 1/11$ , and its overall rate is $0.3 + 0.6f$ . Comparing the two formulas, B is ahead overall exactly when $0.6(e - f) > 0.1$ , which means $e - f > 1/6$ . The reversal needs no extreme mix. It happens as soon as the easy share of B exceeds that of A by more than one sixth.

Simpson's Paradox

Video 1, chapter 10

Key terms