Confusion Matrices and Decision Thresholds

Definition In binary classification each example has an actual label $y \in \{0, 1\}$, and the classifier makes a decision $\hat{y} \in \{0, 1\}$. The positive class is the class coded $1$, the one the task is set up to detect, and the positive decision $\hat{y} = 1$ is the action taken for it. Scores are oriented so that a larger score $s(\mathbf{x})$ means stronger evidence for the positive class. Which class is called positive is part of the problem statement, not a judgment that the outcome is good, and every count and rate on this page depends on that choice. Swapping the two classes exchanges true positives with true negatives and false positives with false negatives, and it exchanges sensitivity with specificity.

Definition Given a score $s(\mathbf{x})$ and a real number $t$, the threshold rule decides

The number $t$ is the decision threshold. On this page a score equal to the threshold counts as positive. That is a convention, and it should be stated, because software does not always use the same one. Changing $t$ changes the decisions without refitting the model that produced the scores. For fixed scores, lowering the threshold can only switch decisions from $0$ to $1$, so every example selected at one threshold is also selected at every lower threshold. A threshold above the largest score selects nothing, and a threshold at or below the smallest score selects everything.

Definition Let $y_1, \dots, y_n$ be the actual labels and $\hat{y}_1, \dots, \hat{y}_n$ the decisions for the same $n$ examples. Each example falls in exactly one of four cells. A true positive has $y_i = 1$ and $\hat{y}_i = 1$, a false positive has $y_i = 0$ and $\hat{y}_i = 1$, a true negative has $y_i = 0$ and $\hat{y}_i = 0$, and a false negative has $y_i = 1$ and $\hat{y}_i = 0$. The confusion matrix is the table of the four counts TP, FP, TN and FN. On this page its rows are the actual labels and its columns are the decisions, each in the order $0$ then $1$.

The four counts add to $n$. The row totals $n_0 = \text{TN} + \text{FP}$ and $n_1 = \text{FN} + \text{TP}$ are the numbers of actual negatives and actual positives. They are fixed by the labels, while the column totals count the decisions and change with the rule. Other texts and software may swap the rows and columns or list the positive class first, so read the axis labels before reading a cell.

Definition A false positive is an example with actual label $0$ that receives decision $1$, and a false negative is an example with actual label $1$ that receives decision $0$. These are the only two kinds of error a binary decision can make. For a threshold rule with fixed scores and two thresholds $t' < t$, every example selected at $t$ is also selected at $t'$, so

A change of threshold therefore never reduces both counts. Lowering it can only turn true negatives into false positives and false negatives into true positives, and raising it does the reverse.

Definition Let $n_1 = \text{TP} + \text{FN}$ be the number of actual positives and $n_0 = \text{TN} + \text{FP}$ the number of actual negatives. When $n_1 \ge 1$ and $n_0 \ge 1$,

Sensitivity is also called recall or the true positive rate. The false positive rate is $\text{FP}/(\text{TN} + \text{FP})$, which equals $1$ minus the specificity. When the evaluation examples are a random sample from the population of interest, the two rates estimate $P(\hat{Y} = 1 \mid Y = 1)$ and $P(\hat{Y} = 0 \mid Y = 0)$ for that rule. With no actual positives the sensitivity is undefined, and with no actual negatives the specificity is undefined. Report that absence rather than a value a program substitutes. For a threshold rule with fixed scores, lowering the threshold never decreases the sensitivity and never increases the specificity. Selecting every example gives sensitivity $1$ and specificity $0$, and selecting none gives sensitivity $0$ and specificity $1$.

Definition Let $C(a, y)$ be the cost of decision $a$ when the actual label is $y$, for $a, y \in \{0, 1\}$, and let $p = P(Y = 1 \mid \mathbf{x})$ be the conditional probability of the positive class for the case being decided. The expected cost of decision $a$ is

the probability-weighted average of its possible costs before the outcome is known. Choosing for each case the decision with the smaller expected cost minimizes the expected cost per case among all rules that use $\mathbf{x}$, provided $p$ is the true conditional probability, the costs are fixed and known, and each case is decided on its own with no limit on the number of positive decisions. When correct decisions cost nothing and the error costs $c_{\text{FP}} = C(1, 0)$ and $c_{\text{FN}} = C(0, 1)$ are positive, the rule decides $1$ exactly when

At $p = t^{*}$ both decisions have the same expected cost, and this page sends ties to the positive decision. More generally, if $\Delta_0 = C(1, 0) - C(0, 0)$ and $\Delta_1 = C(0, 1) - C(1, 1)$ are both positive, the threshold is $\Delta_0 / (\Delta_0 + \Delta_1)$. On labeled data with free correct decisions, the observed cost of a rule is $c_{\text{FP}}\,\text{FP} + c_{\text{FN}}\,\text{FN}$, and its average over a representative validation set estimates the expected cost per case of that rule.