Imbalanced Classes and Cost-Sensitive Learning

Definition The prevalence of the positive class in a population is $\pi = P(Y = 1)$, and in a data set with $n_1$ positives among $n$ examples the sample prevalence is $n_1/n$. With more than two classes, the prevalence of each class is defined the same way. A population or data set has class imbalance when the class prevalences differ substantially, for example when one class makes up a few percent of the cases. Prevalence belongs to a stated population, so the training data, the validation data and the cases met after deployment can each have a different prevalence.

Definition For binary classification with at least one actual positive and one actual negative, the balanced accuracy is

For $K$ classes, each with $n_k \ge 1$ actual examples, it is the unweighted mean of the per-class recalls $\text{TP}_k / n_k$, where $\text{TP}_k$ counts the examples of class $k$ that are predicted as class $k$. Ordinary accuracy is the same recalls weighted by the class shares $n_k / n$, so the two agree whenever the classes are equally frequent. A rule that always predicts one class has balanced accuracy $1/K$.

Definition Suppose a metric such as precision, recall or F1 is computed for each of $K$ classes by treating that class as positive and all the others as negative, which gives counts $\text{TP}_k$, $\text{FP}_k$ and $\text{FN}_k$. The macro average computes the metric for each class and takes the unweighted mean. The micro average adds the counts over the classes first and computes the metric once from the totals. For precision, with $\text{prec}_k = \text{TP}_k / (\text{TP}_k + \text{FP}_k)$,

In single-label classification, where each example has exactly one actual class and one predicted class, every error is a false positive for the predicted class and a false negative for the actual class. The false positives and the false negatives therefore both add up to the number of errors, and when every class is included, micro precision, micro recall and micro F1 all equal the accuracy. Macro recall equals the balanced accuracy. Macro F1 is the mean of the per-class F1 scores, which in general differs from the F1 score computed from macro precision and macro recall. A class with no predicted examples has undefined precision, and a macro average then needs a stated convention for it.

Definition Training with class weights gives every example of class $k$ a weight $w_k > 0$ in the training objective. For training examples $(\mathbf{x}_i, y_i)$ with $i = 1, \dots, n$, a model $f(\mathbf{x}; \boldsymbol{\theta})$ and the loss $L_i = L\bigl(y_i, f(\mathbf{x}_i; \boldsymbol{\theta})\bigr)$ on example $i$, the weighted empirical risk is

Dividing by the total weight does not change which $\boldsymbol{\theta}$ minimizes it, and equal weights give the ordinary empirical risk. A common choice is $w_k = n/(K n_k)$, which gives each class the same total weight. The weights change which model is fitted, and they also change what its scores mean. For binary labels and the log loss, the function that minimizes the weighted risk over the population has outputs $q(\mathbf{x})$ with

where $p(\mathbf{x}) = P(Y = 1 \mid \mathbf{x})$ lies strictly between $0$ and $1$. So a model that comes close to this minimizer with $w_1 > w_0$ overstates the probability of the positive class.

Definition Resampling changes which training rows a learner sees. Random oversampling repeats minority examples, random undersampling discards majority examples, and synthetic oversampling adds new minority examples. SMOTE, the synthetic minority oversampling technique, builds each synthetic example from a minority training example $\mathbf{x}_i$ and a neighbor $\mathbf{x}_j$ chosen at random among its $k$ nearest minority training examples, commonly with $k = 5$, as

with $\lambda$ drawn uniformly from $[0, 1]$, and gives it the minority label. The new point lies on the segment from $\mathbf{x}_i$ to $\mathbf{x}_j$. The method assumes numeric features for which distances and straight-line interpolation make sense, so features are usually scaled first, with the scaling fit on the training data. Like every resampling step, it is applied only to training data, inside each training fold of cross-validation, and never to validation or test data, which keeps it from causing data leakage.

Definition Cost-sensitive learning builds a classifier to minimize expected misclassification cost rather than the error rate. Suppose correct decisions cost nothing, a false positive costs $c_{\text{FP}} > 0$ and a false negative costs $c_{\text{FN}} > 0$. The costs can enter the training, by weighting examples with $w_1 / w_0 = c_{\text{FN}} / c_{\text{FP}}$ or by resampling in the same proportion. They can instead enter the decision, by applying a threshold to probability estimates $\hat{p}(\mathbf{x})$ and deciding positive when

This threshold rule minimizes the expected cost per case when $\hat{p}(\mathbf{x}) = P(Y = 1 \mid \mathbf{x})$, the costs are fixed and known, and each case is decided on its own with no limit on how many can be selected. If a model weighted with $w_1 / w_0 = c_{\text{FN}} / c_{\text{FP}}$ reaches the minimizer of the weighted log loss described under class weights, then thresholding its score at $1/2$ gives the same decisions as the cost-derived threshold applied to the true probabilities.