Unit 15, lesson 90

Rare-Event Classification Project

How a rare-event classification project links an event definition, a calibration sample, a cost-based threshold under review capacity, a frozen procedure and a subgroup audit to a final report with honest limits.

  • Video 90
  • Regression and Classification Projects

A parcel service wants to flag, before dispatch, the parcels likely to arrive more than 15 minutes late, so that someone can review them. When only about 8 percent of parcels are late, a rule that never flags anything is right about 92 percent of the time and finds none of the late parcels. A useful project has to connect the label, the probabilities, the decision and its consequences, and then test the whole frozen procedure on parcels that played no part in building it. This lesson follows one constructed project through those steps and shows what its final report can and cannot claim.

Key terms

Rare-event classification

Definition Rare-event classification is binary classification in which the positive class, the event, has a low prevalence π=P(Y=1)\pi = P(Y = 1), often a few percent or less. With n1n_1 events among nn cases the sample prevalence is n1/nn_1/n. The rule that never predicts the event has accuracy 1−n1/n1 - n_1/n and recall 0, and its precision is undefined because it makes no positive predictions. A recall is estimated from the n1n_1 events alone, so its uncertainty depends on the number of events rather than on nn. Precision depends on the prevalence, and a score that is the same for every case has average precision equal to the prevalence when tied scores are grouped, which makes the prevalence the reference value for that metric. Any reweighting or resampling of classes is done inside the development data, and evaluation data keep the prevalence of the population the report describes.

In plain words When events are rare, accuracy mostly counts the easy negatives. A useful report gives the prevalence with its counts and uses metrics that look at the events, and it remembers that a few dozen events give only rough estimates even inside thousands of rows. Enriching an evaluation set with extra events changes precision and other prevalence-dependent numbers unless the sampling is accounted for.

Example In the constructed data, two indices from 0 to 1 with no physical unit, queue load x1x_1 and route complexity x2x_2, are known at dispatch, and each parcel belongs to depot A or B, with B chosen with probability 0.2. The indices are independent and uniform on [0,1][0, 1], and the event probability is σ(z)\sigma(z), with the logistic sigmoid σ\sigma and

z=−5+2x1+1.5x2+2h+0.5g,\begin{aligned} z &= -5 + 2x_1 + 1.5x_2 \\[0.4em] &\quad + 2h + 0.5g , \end{aligned}

where h=1h = 1 when both x1>0.7x_1 > 0.7 and x2>0.7x_2 > 0.7 and h=0h = 0 otherwise, and g=1g = 1 for depot B and g=0g = 0 for depot A. About 8.7 percent of parcels in this population are events. The 1000 training, 600 calibration, 600 validation and 1200 test parcels contain 87, 50, 45 and 96 events. On validation the never-review rule is right for the 555 negatives, an accuracy of 92.5 percent, while its recall is 0 out of 45 and it has no precision at all. The candidates are a constant baseline that gives every parcel the training event fraction 87/1000=0.08787/1000 = 0.087, a logistic regression on the standardized indices with a small penalty on its slopes, and a random forest of 41 trees of depth at most 4, whose leaves predict (k+1)/(m+2)(k + 1)/(m + 2) for kk events among mm sampled rows. Both models are trained without class weights or oversampling.

Event definition

Definition An event definition is the exact rule that sets the label YY of each unit, with its measurement, threshold, units and time window, the handling of borderline and missing cases, and the time at which the label becomes known. A rare-event project ties it to a prediction time, a decision taken at that time and the action that follows a positive decision. Every input must be recorded before the prediction time, and anything known only afterward, including the outcome and fields derived from it, is excluded. Fields kept only for evaluation, such as a group label for an audit, are named as such.

In plain words The label is a measurement choice, not a fact of nature. Changing “more than 15 minutes late” to “more than 10 minutes late” changes the prevalence, the costs and every metric. A classifier that uses information arriving after the decision can look accurate in hindsight and still be useless when the decision is made.

Example A parcel is an event, Y=1Y = 1, when it arrives more than 15 minutes late, so a parcel exactly 15 minutes late is not an event. The label is known only after delivery. The decision, made before dispatch, is whether to send the parcel for manual review. The inputs are the two indices known at dispatch, and the depot label stays out of the inputs and is used only for the subgroup audit. Review is an action defined for this exercise. Nobody has measured whether review prevents delays, so the costs used later state assumed consequences, not savings shown by an intervention study.

Calibration sample

Definition A calibration sample is a set of labeled cases, separate from the rows used to fit the base model and from the validation and test data, used only to fit a map from model scores to probabilities. It is part of the development data and is used before the final evaluation. The sigmoid map used here clips a raw probability pp into [10−6,1−10−6][10^{-6}, 1 - 10^{-6}], giving p′p', forms the log odds s=log⁡(p′/(1−p′))s = \log\bigl(p'/(1 - p')\bigr) and returns

pcal=11+e−(as+c),p_{\text{cal}} = \frac{1}{1 + e^{-(as + c)}} ,

where aa and cc minimize the mean cross-entropy on the calibration sample plus the small penalty 10−4a2/210^{-4}a^2/2. When a>0a > 0 the map is increasing, so on scores inside the clipping range it keeps the order of the cases and leaves ranking metrics such as ROC AUC and average precision unchanged. It can still change which cases pass a fixed threshold. Whether the calibrated probabilities are trustworthy has to be checked on data that did not fit the map.

In plain words A model can rank parcels well and still say 30 percent where the true rate is 15 percent. Calibration rescales the scores into probabilities using cases the model never trained on. Fitting the map on the training rows would calibrate the scores the model gives to cases it has already fit, which are usually more extreme than its scores on new cases, and fitting it on the test rows would use up the final evaluation.

Example The 600 calibration parcels, with 50 events, gave a≈1.281a \approx 1.281 and c≈0.435c \approx 0.435 for the logistic model and a≈1.084a \approx 1.084 and c≈0.053c \approx 0.053 for the forest. On validation both models keep their ROC AUC, about 0.865 and 0.851, while the forest’s Brier score moves from 0.04946 to 0.04920, a small gain that does not show the probabilities are right. At the threshold 0.1, the raw forest scores would flag 97 of the 600 validation parcels and the calibrated scores flag 68. Later, on the final test data, the highest probability bin declared in advance, from 0.4 to 1, holds 67 parcels with a mean predicted probability of 54.3 percent but only 26 events, an observed rate of 38.8 percent, so the calibrated forest still overpredicts there. See Probability Calibration for the methods in more detail.

Cost-based threshold and review capacity

Definition Suppose an unnecessary review of a negative case costs cFP>0c_{\text{FP}} > 0, a missed event costs cFN>0c_{\text{FN}} > 0 and correct decisions cost nothing. If p=P(Y=1∣x)p = P(Y = 1 \mid \mathbf{x}), reviewing has expected cost cFP(1−p)c_{\text{FP}}(1 - p) and not reviewing has expected cost cFN pc_{\text{FN}} \, p, so reviewing costs no more exactly when

p≥cFPcFP+cFN.p \ge \frac{c_{\text{FP}}}{c_{\text{FP}} + c_{\text{FN}}} .

This cost-based threshold minimizes the expected cost of each case decided on its own when the probabilities are the true conditional probabilities, the costs are known and fixed, and any number of cases can be reviewed. Review capacity is a limit on how many cases can be reviewed, for example a maximum fraction of all cases. The expected cost saved by reviewing a case is (cFP+cFN)p−cFP(c_{\text{FP}} + c_{\text{FN}})p - c_{\text{FP}}, which grows with pp, so under a binding limit the cheapest choice reviews the cases with the highest probabilities among those above the threshold, up to the limit. A fixed threshold does not enforce a future quota, because the number of cases above it changes with the scores that arrive.

In plain words The threshold comes from what the mistakes cost, not from 0.5. Cheap reviews and expensive misses push it down, and a cap on reviews can force the operating point above it. Because estimated probabilities are never exact, candidate thresholds are compared by their observed cost on validation data.

Example With cFP=2c_{\text{FP}} = 2 and cFN=20c_{\text{FN}} = 20 the threshold is 2/22=1/11≈0.09092/22 = 1/11 \approx 0.0909, and a parcel is reviewed when its probability is at least the threshold. The project compared thresholds 0.05, 0.1, 0.15, 0.2, 0.3, 0.5 and 1 on validation, allowing at most 150 of the 600 parcels to be reviewed. The rule with the lowest total cost wins, with ties broken by fewer reviews, then the declared model order, then the higher threshold. At 0.05 both models exceed the limit, with 237 and 231 reviews. The best eligible choices were these.

RuleThresholdTPFPFNReviewsCost
Constant 0.0870.1 or higher00450900
Calibrated logistic0.13510810143416
Calibrated forest0.129391668398

The forest’s cost is 2⋅39+20⋅16=3982 \cdot 39 + 20 \cdot 16 = 398, so it was selected, with precision 29/68≈0.42629/68 \approx 0.426 and recall 29/45≈0.64429/45 \approx 0.644. It finds fewer events than the logistic model but avoids enough unnecessary reviews to win under this cost table. A different table could reverse the order, and an 18-unit lead on one validation sample says little about forests and logistic models in general.

Frozen procedure

Definition A frozen procedure is the complete prediction and decision pipeline, fixed before the final evaluation and saved so that it can be applied unchanged. It includes the preprocessing values learned from the training data, the fitted model parameters, the calibration map, the selected model, the threshold with its rule for ties, the cost table and capacity rule, and the reporting conventions, such as the metrics, probability bins and interval methods. The final data are scored once with it, and only after it is saved. Any change made after seeing the final results, such as another threshold, another model or a recalibration, starts a new development cycle, and a claim about the changed procedure needs new data that played no part in it.

In plain words Freezing turns a collection of choices into one object that can be tested. The final evaluation tests that object, and a disappointing result is reported as it is rather than repaired on the same data.

Example The frozen pipeline fills missing indices, about 4 percent of the cells, with the training medians 0.521 and 0.516, applies the forest and its calibration map, and reviews a parcel when the calibrated probability is at least 0.1. Rerunning the whole workflow with only the random seed of the test data changed gives different test results but exactly the same saved procedure, a check that no test label reached an earlier step. The test sample has 96 events among 1200 parcels, a prevalence of 8 percent. The procedure finds 41 events, makes 110 unnecessary reviews, misses 55 events and correctly leaves 994 parcels alone, so the precision is 41/151≈27.241/151 \approx 27.2 percent and the recall is 41/96≈42.741/96 \approx 42.7 percent. Its cost is

2⋅110+20⋅55=13202 \cdot 110 + 20 \cdot 55 = 1320

units, or 1.1 per parcel, against 20⋅96=192020 \cdot 96 = 1920 for never reviewing. Its accuracy, 86.25 percent, is below the 92 percent of never reviewing, because accuracy counts both kinds of error equally while the cost table does not. Average precision fell from about 0.485 on validation to 0.259, ROC AUC from 0.851 to 0.762, and the Brier score rose from 0.0492 to 0.0670. The test data come from the same simulation, so no change of population is needed to explain the gap. Sampling variation and the optimism of picking the best validation result can produce it, one run cannot separate the two, and the report keeps the gap. A 95 percent Wilson interval for the recall runs from 33.3 to 52.7 percent, and resampling the 1200 test decisions 2000 times gives a percentile interval for the mean cost from 0.88 to 1.34 units per parcel. Both intervals assume independent parcels and treat the fitted procedure as fixed, so they leave out retraining, uncertain costs and future changes.

Subgroup audit

Definition A subgroup audit evaluates the frozen procedure separately within groups defined in advance, such as depots, regions or groups of people the decisions could affect differently. For each group it reports the number of cases, events and negatives, the prevalence, the recall with the number of events as its denominator, the false positive rate with the number of negatives, the precision, the share of cases reviewed and, where counts allow, calibration, each with its uncertainty. For a proportion p^=k/n\hat{p} = k/n from n≥1n \ge 1 independent cases and z≈1.96z \approx 1.96, the 95 percent Wilson interval is the center plus or minus the half-width, where

center=p^+z2/(2n)1+z2/n,half=z1+z2/n×p^(1−p^)n+z24n2.\begin{aligned} \text{center} &= \frac{\hat{p} + z^2/(2n)}{1 + z^2/n} , \\[0.4em] \text{half} &= \frac{z}{1 + z^2/n} \\[0.4em] &\quad \times \sqrt{\frac{\hat{p}(1 - \hat{p})}{n} + \frac{z^2}{4n^2}} . \end{aligned}

Leaving the group label out of the inputs does not make performance equal, since other inputs and the outcome can differ between groups. Overlapping intervals are not evidence that groups perform alike, and no single metric settles whether a procedure treats groups fairly.

In plain words An overall gain can hide a group that the procedure serves badly. Each group’s numbers need their own denominators, and a group with few events gives wide intervals, which calls for more evidence rather than a finding of equality.

Example Depot A has 943 test parcels with 81 events, and the procedure finds 34, a recall of about 42.0 percent with Wilson interval 31.8 to 52.8 percent. Depot B has 257 parcels with 15 events, and the procedure finds 7, a recall of about 46.7 percent with interval 24.8 to 69.9 percent. The false positive rates are 89/862≈10.389/862 \approx 10.3 percent and 21/242≈8.721/242 \approx 8.7 percent. The similar recalls do not show comparable performance, because the interval for B is more than twice as wide. The simulation gives depot B a higher event probability for the same indices, an effect the model cannot represent because depot is not one of its inputs. Yet B’s observed prevalence in this test sample is 5.8 percent against 8.6 percent for A, a reminder of how much counts in small groups vary. The final report states that the frozen procedure lowers the assumed cost relative to never reviewing on simulated data, that it misses more than half of the events, that its highest probability bin overpredicts and that the subgroup evidence is thin. Operational use would need representative data, verified label timing, agreed costs and capacity, and evidence that review actually helps.

Check yourself

  1. A fresh validation sample has 200 parcels with 12 events, and the costs are 2 for an unnecessary review and 20 for a missed event. A higher threshold gives 6 true positives and 9 false positives, and a lower threshold gives 9 true positives and 25 false positives. With capacity for 40 reviews, find the precision, recall, missed events and cost of each, and choose between them.
Show the answer

At the higher threshold there are 12−6=612 - 6 = 6 missed events and 6+9=156 + 9 = 15 reviews. The precision is 6/15=0.46/15 = 0.4 and the recall is 6/12=0.56/12 = 0.5. The cost is

2⋅9+20⋅6=138.2 \cdot 9 + 20 \cdot 6 = 138 .

At the lower threshold there are 12−9=312 - 9 = 3 missed events and 9+25=349 + 25 = 34 reviews. The precision is 9/34≈0.2659/34 \approx 0.265 and the recall is 9/12=0.759/12 = 0.75. The cost is

2⋅25+20⋅3=110.2 \cdot 25 + 20 \cdot 3 = 110 .

Both rules stay within 40 reviews, so the lower threshold wins the stated cost comparison despite its lower precision. Its accuracy, (9+163)/200=0.86(9 + 163)/200 = 0.86, is also below the (6+179)/200=0.925(6 + 179)/200 = 0.925 of the higher threshold, another case where accuracy and cost disagree.

  1. Keep the counts from question 1. First suppose an unnecessary review costs 10 instead of 2. Then go back to the original costs but allow only 30 reviews. Which threshold wins in each case, and what happens to the cost-based threshold in the first case?
Show the answer

With a false positive cost of 10, the higher threshold costs 10⋅9+20⋅6=21010 \cdot 9 + 20 \cdot 6 = 210 and the lower threshold costs 10⋅25+20⋅3=31010 \cdot 25 + 20 \cdot 3 = 310, so the higher threshold now wins. The cost-based threshold for calibrated probabilities rises from 2/22=1/112/22 = 1/11 to 10/30=1/310/30 = 1/3, since reviews are now relatively more expensive.

With the original costs and room for only 30 reviews, the lower threshold needs 34 reviews and is not allowed, even though its cost of 110 is smaller. The higher threshold, with 15 reviews, is chosen. Costs and capacity are separate constraints, and each has to be stated before the comparison rather than inferred from whichever metric makes a model look best.

  1. In another audit a small subgroup contains 2 events, and the procedure finds 1 of them. A colleague notes that the subgroup’s recall of 50 percent is close to the overall recall and calls the performance comparable. Compute the 95 percent Wilson interval for the subgroup’s recall and say what the evidence supports.
Show the answer

Here k=1k = 1, n=2n = 2 and p^=0.5\hat{p} = 0.5. With z≈1.96z \approx 1.96, z2≈3.841z^2 \approx 3.841 and z2/n≈1.921z^2/n \approx 1.921, the center is (0.5+0.960)/2.921(0.5 + 0.960)/2.921, which is exactly 0.5 because p^=0.5\hat{p} = 0.5. The square root term is

0.125+0.240≈0.604,\sqrt{0.125 + 0.240} \approx 0.604 ,

so the half-width is 1.96×0.604/2.921≈0.4051.96 \times 0.604 / 2.921 \approx 0.405, and the interval runs from about 0.095 to 0.905.

Recall values from under 10 percent to over 90 percent are all compatible with these data, so they cannot establish a recall near 50 percent, let alone comparable or acceptable performance. The report should give the counts, 1 of 2, with the interval and state that the evidence for this subgroup is insufficient, and more events from the subgroup would be needed before any conclusion.

How this lesson connects