Unit 2, lesson 12

Two-Way Tables and Simpson's Paradox

How marginal and conditional percentages in a two-way table can disagree, why aggregation can reverse an association in Simpson's paradox, and how a standardized rate compares groups over a common mixture.

  • Video 12
  • Comparisons and Relationships

Two constructed support services handle cases that are either easy or hard. Overall, service B succeeded on 74 percent of its cases and service A on only 36 percent. Yet among easy cases A did better than B, and among hard cases A did better as well. Both statements describe the same 200 records, and neither contains an arithmetic error. This lesson explains how combining groups can reverse a comparison, and how to choose the comparison that matches the question being asked.

Key terms

Marginal percentage

Definition In a two-way table, a marginal percentage is computed from the row or column totals, which add counts over the categories of the other variable. Examples are the percentage of all cases that succeeded and the percentage of all cases handled by service A. When the records are also classified by a third variable, such as case difficulty, a percentage computed after adding over that variable is marginal with respect to it. The overall success percentage of service A, which adds its easy and hard cases, is of this kind. A conditional percentage, by contrast, restricts the denominator to the records of one group.

In plain words Marginal percentages come from totals and ignore whatever was added over. They describe the totals correctly, but they hide any difference in how the groups are made up.

Example After adding over difficulty, the 200 constructed cases give this table of service and outcome.

ServiceSuccessFailureTotal
A3664100
B7426100
Total11090200

The marginal percentage of successes is 110/200=55%110/200 = 55\%, and half of all cases went to A. Within the rows, A succeeded on 36 percent of its cases and B on 74 percent. These two percentages are conditional on the service and marginal with respect to difficulty.

Conditional association

Definition The conditional association of two variables given a third variable is their association within each fixed category of the third variable, computed only from the records in that category. It can differ from one category to another, and it can differ from the marginal association in the table that adds over the third variable.

In plain words Compare like with like. Hold the third variable fixed, compare the groups inside each category, and only then look at how the categories combine. A conditional comparison is still a description of the data. It is not a causal effect unless the design supports one, for example through random assignment within each category, and the third variable need not capture every relevant difference between the groups.

Example Split the same 200 cases by difficulty.

CasesService AService B
Easy9/10=90%9/10 = 90\%72/90=80%72/90 = 80\%
Hard27/90=30%27/90 = 30\%2/10=20%2/10 = 20\%
All36/100=36%36/100 = 36\%74/100=74%74/100 = 74\%

Given difficulty, the success percentage of A is 10 percentage points higher among easy cases and 10 percentage points higher among hard cases. In the aggregated table B is 38 percentage points higher.

Aggregation

Definition Aggregation combines records from several subgroups into one table by adding their counts. Suppose a group has nk≥1n_k \ge 1 records in subgroup kk for k=1,…,Kk = 1, \dots, K, with observed success proportion p^k\hat{p}_k, and n=n1+⋯+nKn = n_1 + \dots + n_K records in all. Its aggregated success proportion is the weighted average

p^=∑k=1Knkn p^k.\hat{p} = \sum_{k=1}^{K} \frac{n_k}{n}\, \hat{p}_k .

The weights nk/nn_k / n describe how the records of the group are spread over the subgroups. The aggregated proportion therefore lies between the smallest and the largest subgroup proportion, and a subgroup that holds more of the records pulls it more strongly toward its own proportion.

In plain words An overall rate is not a separate fact about a group. It is the subgroup rates of that group, mixed in the proportions in which the group happened to meet the subgroups. Aggregating does not change a single record, only which records share a denominator.

Example Service A handled 10 easy and 90 hard cases, and service B handled 90 easy and 10 hard cases. Their overall success proportions are

p^A=0.1(0.9)+0.9(0.3)=0.36\begin{aligned} \hat{p}_A &= 0.1(0.9) + 0.9(0.3) \\[0.4em] &= 0.36 \end{aligned}

and

p^B=0.9(0.8)+0.1(0.2)=0.74.\begin{aligned} \hat{p}_B &= 0.9(0.8) + 0.1(0.2) \\[0.4em] &= 0.74 . \end{aligned}

Most cases of A were hard, so its overall rate sits close to its hard-case rate of 30 percent. Most cases of B were easy, so its overall rate sits close to its easy-case rate of 80 percent.

Simpson’s paradox

Definition Simpson’s paradox occurs when an association between two variables has the same direction within every category of a third variable but is reversed, or disappears, once the categories are aggregated. For success proportions with group A ahead in every subgroup, so that p^A,k>p^B,k\hat{p}_{A,k} > \hat{p}_{B,k} for every kk, it means that p^A≤p^B\hat{p}_A \le \hat{p}_B overall. It can happen only when the groups compared are spread differently over the subgroups. Write dk=p^A,k−p^B,kd_k = \hat{p}_{A,k} - \hat{p}_{B,k}. If both groups had the same weights wkw_k, the overall difference would be

p^A−p^B=∑kwkdk,\hat{p}_A - \hat{p}_B = \sum_{k} w_k d_k ,

which is positive whenever every dkd_k is positive.

In plain words The paradox is not a contradiction. Each statement is true about its own denominators, and the totals mix a comparison of the groups with a comparison of their case mixes. Which comparison answers a question depends on the question and on how the data arose. If difficulty is fixed before a case reaches a service and affects both which service takes it and whether it succeeds, difficulty confounds the overall comparison, and the within-difficulty comparison is the more relevant description of how the services do on similar cases. Adjusting for every available variable is not a safe rule, though. Adjusting for a variable that the service itself affects, or one that reflects how records were selected, can create a misleading association.

Example In the service data, A leads by 10 percentage points among easy cases and by 10 among hard cases, yet B leads by 38 percentage points overall. The reversal comes entirely from the weights. Of the cases of A, 90 percent were hard, while 90 percent of the cases of B were easy, and hard cases succeed far less often at both services. Before recommending a service, we would need to know how cases were assigned, whether difficulty was recorded before the assignment, and whether easy and hard cases differ in other ways that affect success.

Standardized rate

Definition A standardized rate applies one fixed set of subgroup weights, the standard mixture w1,…,wKw_1, \dots, w_K with each wk≥0w_k \ge 0 and w1+⋯+wK=1w_1 + \dots + w_K = 1, to the subgroup rates of every group being compared. For service A it is

p^A std=∑k=1Kwk p^A,k,\hat{p}_A^{\,\text{std}} = \sum_{k=1}^{K} w_k\, \hat{p}_{A,k},

and likewise for B. Every subgroup with positive weight must contain records from each group. Because the weights are shared, the difference of two standardized rates is ∑kwkdk\sum_k w_k d_k, so a group that is ahead in every subgroup is ahead under every standard mixture. A group’s standardized rate describes a hypothetical mixture, not a rate that was observed, unless the standard mixture happens to be that group’s own mixture.

In plain words Standardizing asks how the groups would compare if each kept its own subgroup rates but faced the same mix of cases. The standard mixture should come from the question, such as the mix of cases expected next year, and should not be chosen because it produces a preferred ranking. Standardizing sample rates does not by itself show what would happen if a service were assigned to new cases.

Example With weights of one half for easy and for hard cases,

p^A std=0.5(0.9)+0.5(0.3)=0.60\begin{aligned} \hat{p}_A^{\,\text{std}} &= 0.5(0.9) + 0.5(0.3) \\[0.4em] &= 0.60 \end{aligned}

and

p^B std=0.5(0.8)+0.5(0.2)=0.50.\begin{aligned} \hat{p}_B^{\,\text{std}} &= 0.5(0.8) + 0.5(0.2) \\[0.4em] &= 0.50 . \end{aligned}

A standard mixture with 20 percent easy cases gives 0.42 for A and 0.32 for B. The difference is 10 percentage points both times, because it is 10 points in each subgroup. The observed overall rates of 36 and 74 percent stay as they were. They answer a different question, about the cases each service actually handled.

Check yourself

  1. Two constructed help desks resolve tickets on first contact. Desk X resolved 45 of 50 simple tickets and 60 of 150 complex ones. Desk Y resolved 136 of 160 simple tickets and 14 of 40 complex ones. Compute the four conditional percentages and the overall percentage for each desk. Does the table show Simpson’s paradox?
Show the answer

Desk X resolved 45/50=90%45/50 = 90\% of its simple tickets and 60/150=40%60/150 = 40\% of its complex ones. Desk Y resolved 136/160=85%136/160 = 85\% and 14/40=35%14/40 = 35\%. So X is 5 percentage points ahead within each subgroup.

Overall, X resolved 105/200=52.5%105/200 = 52.5\% and Y resolved 150/200=75%150/200 = 75\%, so Y is 22.5 percentage points ahead after aggregation. The direction of the comparison reverses, which is Simpson’s paradox.

  1. The tickets of desk X were 25 percent simple. What is the standardized rate of desk Y with that mixture as the standard, and why can no standard mixture put Y ahead of X?
Show the answer

The standardized rate of Y is 0.25(0.85)+0.75(0.35)=0.4750.25(0.85) + 0.75(0.35) = 0.475, or 47.5 percent, against the observed 52.5 percent of X, which already has this mixture. For any standard mixture with weights ww for simple and 1−w1 - w for complex tickets, the difference X minus Y is w(0.05)+(1−w)(0.05)=0.05w(0.05) + (1 - w)(0.05) = 0.05. So X stays exactly 5 percentage points ahead. The lead of Y in the totals comes from its mixture, 80 percent simple tickets against 25 percent at X, not from higher rates within either subgroup.

  1. A manager reads only the totals and proposes sending every ticket to desk Y. Explain the reversal in words, and say what further information about the design you would need before recommending either desk.
Show the answer

Complex tickets are resolved far less often at both desks, and X received mostly complex tickets while Y received mostly simple ones. The total of X is therefore pulled toward its complex-ticket rate and the total of Y toward its simple-ticket rate, which hides the lead of X within each subgroup. The totals compare the mixes of tickets as much as the desks.

Before recommending a desk we would need to know how tickets were routed, whether complexity was recorded before routing rather than judged afterward, whether the tickets differ between desks in other ways that affect resolution, and which mix of tickets the recommendation is meant for. A complexity label assigned after a desk had worked on a ticket could itself be affected by the desk, and adjusting for it could mislead. A design that assigns tickets to desks at random within each level of complexity would let a difference in resolution rates be attributed to the desks, apart from chance variation.

How this lesson connects