Fairness and Subgroup Evaluation, machine learning lesson

A classifier that sends cases to an extra review is right for 80 percent of the cases in each of two groups. That one number does not say whether both groups are referred equally often, whether the cases that need review are caught equally often, or whether a referral is equally likely to be needed in both groups. Each of these questions compares a different rate with a different denominator, and when the groups differ in how often review is needed, the rates cannot all be equal at once. This lesson computes subgroup metrics with their denominators, states the main statistical fairness criteria precisely, and shows how to make the tradeoff between two decision policies explicit.

Key terms

Algorithmic fairness

Definition Algorithmic fairness is the study of whether the decisions made with a model, and the benefits and harms that follow from them, are distributed acceptably among the people and groups they affect. An analysis starts from a stated decision, a target label, the people affected and the consequences of each kind of error. A statistical fairness criterion is a precise condition on the joint distribution of group membership $G$ , the actual label $Y$ and the decision $\hat{Y}$ (or a score), for example equal selection rates or equal error rates across groups. Each criterion answers one question. Meeting it does not show that the target, the data collection or the use of the model is justified, and whether group membership may be recorded or used at all depends on the law, the consent given and the setting, which no table of rates can settle.

In plain words Fairness is not a single number that a model has or lacks. A criterion makes one comparison exact, and deciding which comparison matters is a judgment about the application and about whose outcomes are at stake. Different criteria can disagree about the same table of decisions.

Example In a constructed review task, the decision $\hat{Y} = 1$ refers a case for an extra review, and the label $Y = 1$ means that the case needs that review under a constructed labeling rule. Groups A and B have 100 cases each. They are abstract teaching groups, not real populations, and the counts are constructed.

group	TP	FP	FN	TN
A	16	16	4	64
B	40	10	10	40

Both groups have accuracy $80/100 = 0.8$ , and so does the pooled table of 200 cases. Yet a referral in group A is needed in 16 of 32 cases, while in group B it is needed in 40 of 50. The accuracy hides this difference, and whether the difference matters depends on what an unnecessary referral costs the person referred and what a missed review costs the person who needed it.

Subgroup metric

Definition A subgroup metric is an evaluation metric computed only on the cases of one group $g$ and reported with its denominator. From the confusion counts of group $g$ ,

\begin{aligned} \text{TPR}_g &= \frac{\text{TP}_g}{\text{TP}_g + \text{FN}_g}, \\[0.4em] \text{FPR}_g &= \frac{\text{FP}_g}{\text{FP}_g + \text{TN}_g}, \\[0.4em] \text{PPV}_g &= \frac{\text{TP}_g}{\text{TP}_g + \text{FP}_g}. \end{aligned}

The true positive rate, also called recall, divides by the actual positives of the group. The false positive rate divides by its actual negatives, and the precision, or positive predictive value, divides by its positive decisions. A rate is undefined when its denominator is zero. When the cases of a group are a random sample from a population, each rate estimates a conditional probability, for example $\text{TPR}_g$ estimates $P(\hat{Y} = 1 \mid Y = 1, G = g)$ , and its sampling error grows as its denominator shrinks.

In plain words Every subgroup rate first fixes which cases it counts. Recall looks only at the cases of the group that need review, the false positive rate only at those that do not, and precision only at those that were referred. A rate built on a small denominator is noisy. Searching many slices of the data, including intersections of groups, and reporting the largest gap tends to find a large gap by chance, so a reported gap needs its counts, an uncertainty statement that fits how the data were sampled, and a note of whether the slice was chosen before looking.

Example In both groups the true positive rate is $16/20 = 40/50 = 0.8$ and the false positive rate is $16/80 = 10/50 = 0.2$ . The precisions differ, $16/32 = 0.5$ in group A and $40/50 = 0.8$ in group B. Group A’s recall rests on only 20 actual positives, so moving one of them from caught to missed changes it by $1/20 = 0.05$ , against $1/50 = 0.02$ in group B. If the positives of each group are treated as independent draws, the estimated standard errors of the two recalls are

\begin{aligned} \sqrt{0.8 \cdot 0.2 / 20} &\approx 0.089, \\[0.4em] \sqrt{0.8 \cdot 0.2 / 50} &\approx 0.057, \end{aligned}

and 95 percent Wilson intervals run from about 0.58 to 0.92 in group A and from about 0.67 to 0.89 in group B. Equal sample recalls of 0.8 therefore do not show equal recalls in the populations the groups were drawn from. If the cases are clustered by person, site or time, these independent-draw formulas can understate the uncertainty.

Group base rate

Definition The base rate of group $g$ is the share of its cases with the positive label,

\pi_g = P(Y = 1 \mid G = g),

estimated by $(\text{TP}_g + \text{FN}_g)/n_g$ for a group with $n_g$ cases. It is a property of the labels and the population, not of the model. When the label is a recorded stand-in for the outcome of interest, such as a past decision or a measurement taken more often in one group, the base rate describes that stand-in, and a difference in base rates may come partly from how the labels were produced.

In plain words The base rate says how common the positive label is in a group before any model is involved. Every comparison of rates across groups interacts with it. In particular, precision depends on the base rate even when the decision rule catches positives and refers negatives at the same rates in every group.

Example Group A has base rate $20/100 = 0.2$ and group B has $50/100 = 0.5$ . Consider a group with base rate $\pi$ in which the rule has true positive rate $t$ and false positive rate $f$ . The selected positives make up the fraction $t\pi$ of the group and the selected negatives the fraction $f(1 - \pi)$ , so

\text{PPV} = \frac{t\pi}{t\pi + f(1 - \pi)}

whenever the denominator is positive. With $t = 0.8$ and $f = 0.2$ , the base rate 0.2 gives $0.16/0.32 = 0.5$ and the base rate 0.5 gives $0.4/0.5 = 0.8$ , exactly the two precisions in the table. With the error rates equal, the difference in precision comes entirely from the base rates.

Selection rate and demographic parity

Definition The selection rate of group $g$ is the share of its cases that receive the positive decision,

s_g = P(\hat{Y} = 1 \mid G = g),

estimated by $(\text{TP}_g + \text{FP}_g)/n_g$ . A decision rule satisfies demographic parity, also called statistical parity, when all groups have the same selection rate, which means that $\hat{Y}$ is independent of $G$ . The criterion does not condition on the label. In terms of the group’s error rates $t_g$ and $f_g$ ,

s_g = t_g \pi_g + f_g (1 - \pi_g),

so the selection rate mixes the rate at which positives are selected and the rate at which negatives are selected, in proportion to the base rate.

In plain words Demographic parity asks whether each group receives the decision equally often, whoever needed it. It fits questions about how an opportunity or a burden itself is shared. Because it ignores the labels, it can hold for a rule that catches the positives of one group far less often than those of another, and it can fail for a rule with identical error rates in every group.

Example Group A has 32 referrals among 100 cases and group B has 50, so the selection rates are 0.32 and 0.5 and demographic parity fails, although the two groups have the same true and false positive rates. The formula gives the same numbers, $0.8 \cdot 0.2 + 0.2 \cdot 0.8 = 0.32$ and $0.8 \cdot 0.5 + 0.2 \cdot 0.5 = 0.5$ . To reach parity, the rules would have to refer more cases in group A, fewer in group B, or both, and any change in the number referred in a group changes its true positive rate, its false positive rate or both.

Equalized odds and equality of opportunity

Definition A binary decision satisfies equalized odds across groups when the decision is independent of group membership given the actual label, that is, when

P(\hat{Y} = 1 \mid Y = y, G = g)

takes the same value for every group $g$ , both for $y = 1$ and for $y = 0$ . The groups then share one true positive rate and one false positive rate. Equality of opportunity keeps only the condition for the label that marks the cases that should receive the decision, usually $y = 1$ , so it asks for equal true positive rates alone. Both criteria treat the label as correct, and both can conflict with equal precision. Suppose two groups have base rates strictly between 0 and 1 that differ, and share a true positive rate $t > 0$ and a false positive rate $f > 0$ . Then

\frac{\text{PPV}_g}{1 - \text{PPV}_g} = \frac{t}{f} \cdot \frac{\pi_g}{1 - \pi_g},

so the precision odds are proportional to the base-rate odds and the two precisions differ. Equal error rates and equal precision can hold together across such groups only in degenerate cases, a rule with no false positives in either group, which includes perfect prediction, or a rule with no true positives.

In plain words Equalized odds asks that the cases that need review are caught at the same rate in every group and that the cases that do not need it are referred at the same rate. Equality of opportunity keeps only the first half. Neither says that a referral is equally likely to be needed in each group. When base rates differ, a rule that produces some true positives and some false positives cannot have equal error rates and equal precision at the same time, so choosing one of these criteria is a choice about which question matters.

Example The constructed table satisfies equalized odds exactly in this sample, with true positive rate 0.8 and false positive rate 0.2 in both groups, so it satisfies equality of opportunity too. The ratio $t/f$ is 4. The base-rate odds are $0.2/0.8 = 0.25$ in group A and $0.5/0.5 = 1$ in group B, so the precision odds are 1 and 4, which are the precisions 0.5 and 0.8. Because these are sample rates with the standard errors found above, the table is consistent with equalized odds in the populations but does not establish it.

Decision tradeoff

Definition A decision tradeoff is the exchange between kinds of error, between groups or between fairness criteria that follows from changing a decision rule, for example a threshold. With a fixed score, raising the threshold in a group can only lower or keep its false positives and can only raise or keep its false negatives. One way to compare policies openly is to choose a cost for each kind of error, $c_{\text{FN}}$ for a miss and $c_{\text{FP}}$ for an unnecessary positive decision, and compute

C = c_{\text{FN}}\,\text{FN} + c_{\text{FP}}\,\text{FP}

for each group and in total. The costs are judgments about consequences, not facts supplied by the data, and a single total can hide who bears the errors.

In plain words A new threshold does not remove errors. It moves them between misses and false alarms and between groups, and it can bring one fairness criterion closer while pushing another away. A useful comparison of two policies shows what each one changes, for whom, and at which stated price, so that the people deciding can see the tradeoff instead of a single verdict.

Example A second policy keeps group A as it was and applies a stricter rule to group B, which now has 30 true positives, 20 false negatives, 5 false positives and 45 true negatives. Group B’s selection rate falls from 0.5 to 0.35, its false positive rate from 0.2 to 0.1 and its true positive rate from 0.8 to 0.6, while its precision rises from 0.8 to $30/35 \approx 0.857$ . Equalized odds no longer holds, because group A keeps 0.8 and 0.2, but the gap in selection rates shrinks from 0.18 to 0.03. With constructed costs of 5 units per missed review and 1 unit per unnecessary review, the first policy costs

\begin{aligned} C_{\text{A}} &= 5 \cdot 4 + 16 = 36, \\[0.4em] C_{\text{B}} &= 5 \cdot 10 + 10 = 60, \end{aligned}

which is 96 in total, and the second costs $36 + (5 \cdot 20 + 5) = 141$ . The stricter rule spares 5 cases an unnecessary review and misses 10 more cases that need one. These weights are illustrations, not measured harms, and they leave out how the labels were produced and whether group-specific rules are acceptable in the setting.

Check yourself

Two more constructed groups have 50 cases each. Group C has 10 actual positives, 8 of them referred, and 40 actual negatives, 4 of them referred. Group D has 20 actual positives, 16 of them referred, and 30 actual negatives, 6 of them referred. Find each group’s base rate, selection rate, true positive rate, false positive rate and precision, and say which of demographic parity, equality of opportunity and equalized odds hold in these tables.

Show the answer

Group C has base rate $10/50 = 0.2$ and 12 referrals, so its selection rate is $12/50 = 0.24$ . Its true positive rate is $8/10 = 0.8$ , its false positive rate is $4/40 = 0.1$ and its precision is $8/12 = 2/3$ .

Group D has base rate $20/50 = 0.4$ and 22 referrals, a selection rate of $22/50 = 0.44$ . Its true positive rate is $16/20 = 0.8$ , its false positive rate is $6/30 = 0.2$ and its precision is $16/22 = 8/11 \approx 0.727$ .

Equality of opportunity holds in these tables, since both true positive rates are 0.8. Equalized odds fails, because the false positive rates 0.1 and 0.2 differ, so equal recall alone does not establish it. Demographic parity fails too, with selection rates 0.24 and 0.44. These are descriptions of two small constructed tables. Group C’s recall rests on 10 positives, so a single case moves it by 0.1, and the tables say nothing about why the rates differ or which policy to adopt.

Two groups have different base rates, and a rule has the same true positive rate $t$ and false positive rate $f$ in both, so it satisfies equalized odds. Show that its selection rates are equal only if $t = f$ . What does $t = f$ say about the rule, and what is its precision in each group when $t > 0$ ?

Show the answer

The selection rate of group $g$ is $s_g = t\pi_g + f(1 - \pi_g) = f + (t - f)\pi_g$ . Subtracting the two groups gives

s_{\text{A}} - s_{\text{B}} = (t - f)(\pi_{\text{A}} - \pi_{\text{B}}),

which is zero with $\pi_{\text{A}} \ne \pi_{\text{B}}$ exactly when $t = f$ .

A rule with $t = f$ selects positives and negatives at the same rate within each group, so within a group the decision carries no information about the label, like a lottery with a fixed selection probability. Its precision is

\frac{t\pi_g}{t\pi_g + t(1 - \pi_g)} = \pi_g ,

the base rate itself. For example, with base rates 0.2 and 0.5 and $t = f = 0.3$ , both groups have selection rate 0.3, and the precisions are 0.2 and 0.5. So when base rates differ, demographic parity and equalized odds can hold together only for a rule whose decisions are no better than chance at telling positives from negatives. Adjusting thresholds cannot get around this.

In group B the stricter policy changes the false negatives from 10 to 20 and the false positives from 10 to 5, with group A unchanged. For which ratios of the cost of a miss to the cost of an unnecessary review is the stricter policy cheaper? A colleague says the stricter policy is simply fairer because it nearly equalizes the selection rates and raises group B’s precision. What does that claim leave out?

Show the answer

Group A’s cost is the same under both policies, so only group B matters. The change in cost is

\Delta C = 10\,c_{\text{FN}} - 5\,c_{\text{FP}},

which is negative exactly when $c_{\text{FN}}/c_{\text{FP}} < 1/2$ . The stricter policy is cheaper only if a missed review costs less than half as much as an unnecessary one. With the constructed costs 5 and 1 it costs $50 - 5 = 45$ units more, 141 against 96.

The claim picks one criterion and ignores the others. The selection-rate gap does shrink from 0.18 to 0.03, but equalized odds now fails, with true positive rates 0.8 and 0.6 and false positive rates 0.2 and 0.1, and 10 more cases in group B that need review are missed. A fuller comparison would also ask what a missed review and an unnecessary one cost the people involved, whether the labels measure need equally well in both groups, how much the rates could move by chance with 50 positives in group B, and whether a rule specific to one group is acceptable in this setting. The numbers make the tradeoff visible. They do not decide it.

Fairness and Subgroup Evaluation

Key terms

Algorithmic fairness

Subgroup metric

Group base rate

Selection rate and demographic parity

Equalized odds and equality of opportunity

Decision tradeoff

Check yourself

How this lesson connects

Builds on

Used later in