Precision, Recall, and Classification Metrics, machine learning lesson

A classifier for delayed deliveries reports an accuracy of $90$ percent. Whether that is good depends on how common delays are, which errors the classifier makes and which threshold produced its decisions. If only $2$ deliveries in $10$ are delayed, a rule that never predicts a delay already reaches $80$ percent while catching nothing. This lesson defines the metrics that look at the positive class directly at one threshold, and the curves and summaries that compare score rankings across all thresholds.

Key terms

Accuracy

Definition For a binary decision rule evaluated on $n \ge 1$ examples, with counts TP, FP, TN and FN from its confusion matrix, the accuracy is

\text{accuracy} = \frac{\text{TP} + \text{TN}}{n},

the fraction of decisions that are correct. Let $\pi = n_1/n$ be the sample prevalence, the share of actual positives among the $n$ examples. When both classes are present, the accuracy equals $\pi$ times the sensitivity plus $1 - \pi$ times the specificity, so it is a weighted average of the two class rates with weights set by the class shares. A rule that always predicts the more common class has accuracy equal to the larger of $\pi$ and $1 - \pi$ .

In plain words Accuracy counts every correct decision equally, so the larger class dominates it. When positives are rare, a rule can reach high accuracy by ignoring them, and two rules of very different use can have the same accuracy. It is a fair first number only next to the prevalence and the accuracy of the majority rule.

Example In a constructed evaluation set of ten deliveries, deliveries $1$ and $5$ were delayed and the other eight were on time. Delayed is the positive class, so the sample prevalence is $2/10 = 0.2$ . The rule that always predicts on time is right for the eight on-time deliveries, an accuracy of $0.8$ , and it catches neither delay. Two models score the same deliveries, with a higher score meaning stronger evidence of delay. The scores are constructed rankings, not checked probabilities.

delivery	label	model A	model B
$1$	$1$	$0.95$	$0.86$
$2$	$0$	$0.85$	$0.96$
$3$	$0$	$0.75$	$0.66$
$4$	$0$	$0.65$	$0.56$
$5$	$1$	$0.55$	$0.76$
$6$	$0$	$0.45$	$0.46$
$7$	$0$	$0.35$	$0.36$
$8$	$0$	$0.25$	$0.26$
$9$	$0$	$0.15$	$0.16$
$10$	$0$	$0.05$	$0.06$

At the threshold $0.7$ , with a score equal to the threshold counted as positive, model A selects deliveries $1$ , $2$ and $3$ . That gives $1$ true positive, $2$ false positives, $1$ false negative and $6$ true negatives, so its accuracy is $7/10$ .

Precision and recall

Definition For a binary decision rule with counts TP, FP and FN,

\begin{aligned} \text{precision} &= \frac{\text{TP}}{\text{TP} + \text{FP}}, \\[0.4em] \text{recall} &= \frac{\text{TP}}{\text{TP} + \text{FN}}. \end{aligned}

Precision is defined only when the rule makes at least one positive decision, and recall only when there is at least one actual positive. Recall is the same quantity as sensitivity. When the evaluation examples are a random sample from the population of interest, precision estimates $P(Y = 1 \mid \hat{Y} = 1)$ , also called the positive predictive value, and recall estimates $P(\hat{Y} = 1 \mid Y = 1)$ . Suppose a rule has sensitivity $s$ and false positive rate $f$ in a population where the positive class has prevalence $\pi$ . Its precision there is

\frac{s\pi}{s\pi + f(1 - \pi)},

provided the denominator is positive. So precision changes with the prevalence even when the rule and its two rates stay the same.

In plain words Precision looks along the alerts and asks how many of them are real. Recall looks along the actual positives and asks how many were caught. The two have different denominators, and a rule can do well on one and badly on the other. A rule that raises no alerts has no precision at all, neither $0$ nor $1$ , and any number reported in that case is a software convention rather than an observed ratio. Because precision mixes the behavior of the rule with how common positives are, a precision measured where positives are common does not carry over to a setting where they are rare.

Example Model A at the threshold $0.7$ raises three alerts and one of them is a real delay, so its precision is $1/3$ . It catches one of the two delays, so its recall is $1/2$ . Model B at the same threshold selects deliveries $1$ , $2$ and $5$ , with precision $2/3$ and recall $1$ . For a rule with sensitivity $0.9$ and false positive rate $0.05$ , the precision is $0.45/0.475 \approx 0.947$ at prevalence $0.5$ but only $0.045/0.0925 \approx 0.486$ at prevalence $0.05$ .

F1 score

Definition The F1 score of a binary decision rule is the harmonic mean of its precision and recall,

\begin{aligned} F_1 &= \frac{2\,\text{prec} \cdot \text{rec}}{\text{prec} + \text{rec}} \\[0.4em] &= \frac{2\,\text{TP}}{2\,\text{TP} + \text{FP} + \text{FN}}. \end{aligned}

The two forms agree whenever $\text{TP} \ge 1$ . The count form is defined whenever $2\,\text{TP} + \text{FP} + \text{FN} \ge 1$ , and it gives $0$ when there are no true positives but some errors, even though the first form is then undefined. When there are no actual positives and no alerts, the F1 score is undefined. Like any harmonic mean of two positive numbers, it lies between them and is never larger than their ordinary average.

In plain words The F1 score is high only when precision and recall are both high, because the harmonic mean is pulled toward the smaller of the two. It ignores true negatives and weighs a false alarm and a miss equally. It is a convenient single number for comparing rules on the positive class, not a substitute for the costs of an application, and it is not automatically the right target whenever positives are rare.

Example Model A at the threshold $0.7$ has

\begin{aligned} F_1 &= \frac{2 \cdot \tfrac13 \cdot \tfrac12}{\tfrac13 + \tfrac12} \\[0.4em] &= \frac{2 \cdot 1}{2 + 2 + 1} = \frac{2}{5}. \end{aligned}

Model B at the same threshold has $F_1 = 4/(4 + 1 + 0) = 4/5$ . The always on time rule has no precision, recall $0$ and, by the count form, $F_1 = 0/(0 + 0 + 2) = 0$ .

ROC curve

Definition Let a score $s(\mathbf{x})$ be evaluated on examples with $n_1 \ge 1$ actual positives and $n_0 \ge 1$ actual negatives, and for each threshold $t$ let the rule decide positive when $s(\mathbf{x}) \ge t$ . The true positive rate and false positive rate at $t$ are

\begin{aligned} \text{TPR}(t) &= \text{TP}(t)/n_1, \\[0.4em] \text{FPR}(t) &= \text{FP}(t)/n_0. \end{aligned}

The empirical ROC curve, short for receiver operating characteristic, plots the true positive rate against the false positive rate. It joins with straight segments the point $(0, 0)$ , for a threshold above every score, and the points $\bigl(\text{FPR}(t), \text{TPR}(t)\bigr)$ for each distinct score $t$ taken in decreasing order. It ends at $(1, 1)$ . Each positive admitted moves the curve up by $1/n_1$ and each negative moves it right by $1/n_0$ . Examples with tied scores are admitted together, so a tie between classes appears as one diagonal segment.

In plain words Slide the threshold down from above the top score. Each time it passes a positive the curve steps up, and each time it passes a negative it steps right. A perfect ranking climbs straight to the top left corner $(0, 1)$ before moving right, and a score that gives every example the same value runs along the diagonal from $(0, 0)$ to $(1, 1)$ . The curve shows every threshold at once, so it describes the ranking. On its own it does not say which threshold to use, since that depends on costs and prevalence.

Example Ranked by model A, the labels read positive, then three negatives, then a positive, then five negatives. The curve rises to $(0, 1/2)$ , moves right in three steps of $1/8$ to $(3/8, 1/2)$ , rises to $(3/8, 1)$ and runs right to $(1, 1)$ . Model B ranks one negative first and then both positives, so its curve moves right to $(1/8, 0)$ , rises to $(1/8, 1)$ and runs right to $(1, 1)$ .

ROC AUC

Definition The ROC AUC is the area under the empirical ROC curve. Let $\mathcal{P}$ index the $n_1 \ge 1$ actual positives and $\mathcal{N}$ the $n_0 \ge 1$ actual negatives, with scores $s_i$ . The area equals

\text{AUC} = \frac{1}{n_1 n_0} \sum_{i \in \mathcal{P}} \sum_{j \in \mathcal{N}} h_{ij},

where

h_{ij} = \begin{cases} 1, & s_i > s_j, \\ 1/2, & s_i = s_j, \\ 0, & s_i < s_j. \end{cases}

So the AUC is the probability that the score ranks a randomly chosen positive above a randomly chosen negative, with ties counted as one half, when the two are drawn independently and uniformly from the actual positives and the actual negatives of the evaluation set. It is undefined when the evaluation set contains only one class. It equals $1$ when every positive scores above every negative and $1/2$ for a constant score, and a strictly increasing transformation of the scores leaves it unchanged.

In plain words The AUC compares the scores of one positive and one negative at a time and asks how often the positive comes out on top. It measures ranking only. An AUC of $0.875$ does not mean that $87.5$ percent of the decisions are correct at some threshold, and it says nothing about whether the scores are calibrated probabilities or about the costs of errors. Each axis of the ROC curve is a rate within one class, so when positives are rare the AUC can look high while the precision at every threshold is low.

Example The first positive of model A outranks all eight negatives and its second outranks five, so its AUC is $(8 + 5)/16 = 13/16 = 0.8125$ . Each positive of model B outranks seven negatives, which gives $14/16 = 0.875$ . The areas under the two curves agree, $\tfrac12 \cdot \tfrac38 + 1 \cdot \tfrac58 = \tfrac{13}{16}$ for model A and $1 \cdot \tfrac78 = 0.875$ for model B.

Precision-recall curve and average precision

Definition For the same threshold rules, the precision-recall curve plots precision against recall at each distinct score used as a threshold. Thresholds that select nothing have no precision, and the point $(0, 1)$ that plotting software often adds at the left end is a drawing convention. Let $t_1 > t_2 > \cdots > t_m$ be the distinct scores, let $R_k$ and $P_k$ be the recall and the precision of the rule with threshold $t_k$ , and set $R_0 = 0$ . With $n_1 \ge 1$ , the average precision is

\text{AP} = \sum_{k=1}^{m} (R_k - R_{k-1})\, P_k ,

the sum of each gain in recall times the precision at which it is reached. When no scores are tied, it is the mean, over the actual positives, of the precision of the rule whose threshold is that positive’s score. This sum is not the trapezoidal area under the precision-recall curve, which interpolates between points and can give a different value, so a report should state which summary it uses.

In plain words The precision-recall curve follows the alerts. Going down the ranking, it records what fraction of the cases selected so far are positive at each level of recall. Admitting a negative leaves the recall unchanged and never raises the precision, while admitting a positive raises the recall and never lowers the precision, so the curve has a sawtooth shape. Average precision rewards placing positives near the top. Its baseline is the prevalence rather than one half, since a constant score has average precision equal to the share of positives. ROC AUC and average precision answer different questions and can rank two models in opposite orders.

Example Model A reaches recall $1/2$ with its top case, at precision $1$ , and recall $1$ with its top five cases, at precision $2/5$ . Its average precision is $\tfrac12 \cdot 1 + \tfrac12 \cdot \tfrac25 = 0.7$ . Model B reaches recall $1/2$ at precision $1/2$ and recall $1$ at precision $2/3$ , so its average precision is $\tfrac14 + \tfrac13 = \tfrac{7}{12} \approx 0.583$ . Model B has the larger ROC AUC and model A the larger average precision. Model A puts a delay at the very top, while model B places both delays above more of the negatives. A constant score would give average precision $1/5$ here, the prevalence.

Check yourself

A rule is evaluated on $40$ cases, $6$ of which have the event. It raises $10$ alerts, $4$ of them for real events. Find the four counts, the accuracy, precision, recall, specificity and F1 score, and compare the accuracy with that of the rule that never raises an alert.

Show the answer

Of the $10$ alerts, $4$ are true positives and $6$ are false positives, so $\text{TP} = 4$ and $\text{FP} = 6$ . The other $6 - 4 = 2$ events are missed, so $\text{FN} = 2$ , and the remaining $34 - 6 = 28$ negatives are true negatives. The accuracy is $(4 + 28)/40 = 0.8$ , the precision is $4/10 = 0.4$ , the recall is $4/6 = 2/3$ , the specificity is $28/34 \approx 0.824$ and the F1 score is $8/(8 + 6 + 2) = 0.5$ .

The rule that never raises an alert is right on all $34$ negatives, an accuracy of $34/40 = 0.85$ , higher than the rule’s $0.8$ . It has recall $0$ and no precision. Accuracy prefers the useless rule here. A more useful report states the positive class, the prevalence $6/40 = 0.15$ and the threshold, chosen beforehand on validation data, and gives the four counts with precision and recall, together with the cost of alerts and misses when it is known.

Three positives have scores $0.9$ , $0.6$ and $0.4$ , and four negatives have scores $0.7$ , $0.4$ , $0.2$ and $0.1$ . Find the ROC AUC by comparing pairs, with ties counted as one half, and the average precision, with tied scores admitted together.

Show the answer

There are $3 \times 4 = 12$ pairs. The positive at $0.9$ outranks all four negatives. The positive at $0.6$ outranks three and loses to $0.7$ . The positive at $0.4$ loses to $0.7$ , ties with $0.4$ and outranks two, for $2.5$ . The AUC is

\frac{4 + 3 + 2.5}{12} = \frac{19}{24} \approx 0.792 .

For the average precision, the threshold $0.9$ gives recall $1/3$ at precision $1$ . The threshold $0.7$ admits a negative, so recall stays at $1/3$ . The threshold $0.6$ gives recall $2/3$ at precision $2/3$ . The threshold $0.4$ admits a positive and a negative together, giving recall $1$ at precision $3/5$ . So

\begin{aligned} \text{AP} &= \tfrac13 \bigl(1 + \tfrac23 + \tfrac35\bigr) \\[0.4em] &= \tfrac{34}{45} \approx 0.756 . \end{aligned}

Putting the tied positive ahead of the tied negative would give $10/12$ and $29/36$ , credit for an ordering the score does not make.

Model A’s scores are replaced by their squares after a software change. A colleague says that the ROC AUC and average precision are unchanged, so the rule that alerts when the score is at least $0.7$ behaves as before. Which part of the claim is right?

Show the answer

Squaring is strictly increasing on nonnegative numbers, so the ranking and its ties are unchanged, and the AUC stays $13/16$ and the average precision $0.7$ . The decisions at a fixed threshold do change. The three highest squared scores are $0.9025$ , $0.7225$ and $0.5625$ , so the rule at $0.7$ now alerts on deliveries $1$ and $2$ only. Its precision becomes $1/2$ instead of $1/3$ , with recall still $1/2$ . To reproduce the old decisions, the threshold must move to $0.7^2 = 0.49$ . Ranking quality, performance at a chosen threshold and calibration are separate properties, and each needs its own check.

Precision, Recall, and Classification Metrics

Key terms

Accuracy

Precision and recall

F1 score

ROC curve

ROC AUC

Precision-recall curve and average precision

Check yourself

How this lesson connects

Builds on

Used later in