Unit 5, lesson 33

Probability Calibration

How reliability diagrams, the Brier score and log loss check predicted probabilities against event rates, how calibration differs from discrimination, and how Platt scaling and isotonic regression use a calibration set.

  • Video 33
  • Decisions, Metrics and Calibration

A maintenance team inspects machines in order of predicted failure risk and plans its spare parts from the predicted probabilities themselves. Two models can flag exactly the same machines and still disagree about how many failures to expect, so a good ranking alone does not tell the team which numbers to trust. Probability calibration asks whether, among the cases a model gives probability 0.80.8, the event happens about 8080 percent of the time. This lesson separates calibration from discrimination, scores probability forecasts with the Brier score and log loss, and fits a calibration map on data that the model never trained on.

Key terms

Probability calibration

Definition Let a classifier give each case with features x\mathbf{x} a predicted probability p^(x)\hat{p}(\mathbf{x}) in [0,1][0, 1] for the positive class Y=1Y = 1. The predictions are calibrated in a population of cases if, among the cases that receive the prediction qq, the proportion with Y=1Y = 1 equals qq,

P(Y=1∣p^(X)=q)=q,P\bigl(Y = 1 \mid \hat{p}(\mathbf{X}) = q\bigr) = q ,

for every value qq that the model produces. When the predictions vary continuously, the condition is stated as E[Y∣p^(X)]=p^(X)E[Y \mid \hat{p}(\mathbf{X})] = \hat{p}(\mathbf{X}) with probability 11. Calibration is a property of the predictions together with a population, so the same model can be calibrated in one population and miscalibrated in another with a different event rate. On a finite sample each observed rate only estimates the conditional probability, and an honest check uses labeled cases that played no part in fitting the model or its calibration.

In plain words A calibrated prediction of 0.30.3 means that about 3030 percent of the cases given 0.30.3 have the event. It does not say that any single case is partly positive, since each outcome is 00 or 11. A model is overconfident when its probabilities sit closer to 00 and 11 than the observed rates, and underconfident when they sit closer to the middle. Class imbalance on its own does not spoil calibration, and a model for a rare event can be well calibrated. Training methods that change the effective class proportions, such as resampling or class weights, tend to move the probabilities away from the event rate of the population where the model is used.

Example Eight constructed cases have outcomes 0,0,0,1,0,1,1,10, 0, 0, 1, 0, 1, 1, 1. Model A predicts 0.250.25 for the first four cases and 0.750.75 for the last four. Model B predicts 0.050.05 and 0.950.95 for the same two groups. At the threshold 0.50.5, with a prediction equal to 0.50.5 counted as positive, both models label six of the eight cases correctly, so both have accuracy 0.750.75. The observed event rates are 1/4=0.251/4 = 0.25 in the first group and 3/4=0.753/4 = 0.75 in the second. Model A matches both rates, while model B is overconfident in both groups. Four cases per group is a teaching example and far too few to judge a real model.

Discrimination

Definition Discrimination is the ability of a score to separate the classes by giving positive cases higher scores than negative cases. It depends only on the order of the scores. A common measure, defined when both classes occur, is the ROC AUC

AUC⁡=P(S+>S−)+12 P(S+=S−),\begin{aligned} \operatorname{AUC} &= P(S^{+} > S^{-}) \\[0.4em] &\quad + \tfrac{1}{2}\, P(S^{+} = S^{-}) , \end{aligned}

where S+S^{+} is the score of a randomly chosen positive case and S−S^{-} the score of an independently chosen negative case. On a data set these probabilities are proportions over all pairs of one positive and one negative case. Applying the same strictly increasing function to every score keeps the order, and therefore the AUC, while it generally changes calibration.

In plain words Discrimination asks which cases are riskier than which. Calibration asks whether the risk numbers themselves are right. A model that gives every case the overall event rate can be calibrated in a stable population and still has no discrimination, since all its scores tie and its AUC is 0.50.5. A model can also rank the cases in exactly the right order while overstating every probability.

Example Models A and B order the eight cases in the same way, so they have the same AUC. There are 4×4=164 \times 4 = 16 pairs of a positive and a negative case. Nine pairs have the positive case in the high group and the negative case in the low group, six pairs fall within one group and count half, and one pair has the negative case scored higher. Both models therefore have

AUC⁡=9+6⋅1216=0.75.\operatorname{AUC} = \frac{9 + 6 \cdot \tfrac{1}{2}}{16} = 0.75 .

Squaring the probabilities of model A gives 0.06250.0625 and 0.56250.5625. The order and the AUC stay the same, but the group event rates are still 0.250.25 and 0.750.75, so the squared predictions are no longer calibrated.

Reliability diagram

Definition A reliability diagram groups cases with similar predicted probabilities and plots, for each group, the average predicted probability on the horizontal axis against the observed fraction of positive cases on the vertical axis. The groups are usually bins of the predictions, either of equal width, such as [0,0.1)[0, 0.1), [0.1,0.2)[0.1, 0.2) and so on, or with equal numbers of cases. Perfect calibration on the plotted groups puts every point on the diagonal from (0,0)(0, 0) to (1,1)(1, 1). A point below the diagonal means the model overstated the event rate in that group, and a point above it means the model understated it. The number of cases in each group belongs with the plot, because a rate from a small group is noisy.

In plain words The diagram puts what the model claimed next to what happened. Points on the diagonal mean the stated probabilities agree with the observed frequencies. Points that lie above the diagonal on the left and below it on the right form a line flatter than the diagonal, the usual sign of overconfidence. The choice of bins can hide a deviation or create an apparent one, so the binning rule should be reported too.

Example Grouping the eight cases by their exact predictions gives model A the points (0.25,0.25)(0.25, 0.25) and (0.75,0.75)(0.75, 0.75), both on the diagonal. Model B gives (0.05,0.25)(0.05, 0.25), above the diagonal, and (0.95,0.75)(0.95, 0.75), below it. Each point rests on only four cases. Even for a correct forecast of 0.250.25, a group of four independent cases contains no event with probability 0.754≈0.320.75^4 \approx 0.32, so groups this small often land far from the diagonal by chance.

Brier score and log loss

Definition For nn cases with outcomes yiy_i in {0,1}\{0, 1\} and predicted probabilities pip_i of the outcome 11, the Brier score is

BS⁡=1n∑i=1n(pi−yi)2,\operatorname{BS} = \frac{1}{n} \sum_{i=1}^{n} (p_i - y_i)^2 ,

a number between 00 and 11. The log loss, also called the mean binary cross-entropy, uses natural logarithms,

LL⁡=−1n∑i=1n[yiln⁡pi+(1−yi)ln⁡(1−pi)],\begin{aligned} \operatorname{LL} &= -\frac{1}{n}\sum_{i=1}^{n} \bigl[ y_i \ln p_i \\[0.4em] &\quad + (1 - y_i)\ln(1 - p_i) \bigr] , \end{aligned}

so each case contributes minus the logarithm of the probability given to the outcome that occurred. A probability of 00 for an outcome that occurs makes the log loss infinite. Lower is better for both scores. Both are strictly proper, which means that for a case whose true event probability is π\pi, the expected score of a forecast qq is smallest exactly at q=πq = \pi. When the cases are grouped by their distinct forecast values fkf_k, with nkn_k cases and observed event rate oˉk\bar{o}_k in group kk and overall event rate yˉ\bar{y}, the Brier score splits exactly as

BS⁡=∑knkn(fk−oˉk)2−∑knkn(oˉk−yˉ)2+yˉ(1−yˉ).\begin{aligned} \operatorname{BS} &= \sum_k \tfrac{n_k}{n} (f_k - \bar{o}_k)^2 \\[0.4em] &\quad - \sum_k \tfrac{n_k}{n} (\bar{o}_k - \bar{y})^2 \\[0.4em] &\quad + \bar{y}(1 - \bar{y}) . \end{aligned}

The first sum, the reliability term, is zero exactly when every group is calibrated. The second sum, the resolution, grows when the groups have event rates far from the overall rate, and the last term depends only on the outcomes.

In plain words Both scores judge the whole probability forecast at once. They reward probabilities that agree with the observed rates, and they also reward probabilities that separate high-risk cases from low-risk ones. Log loss punishes a confident mistake far more harshly than the Brier score does, without limit as the probability of the observed outcome falls toward 00. Libraries often clip probabilities away from 00 and 11 to avoid infinite values, which is a numerical safeguard and not a sign that certainty is harmless. A lower score alone therefore does not show better calibration.

Example Model A has squared error 0.06250.0625 on each of its six correct cases and 0.56250.5625 on each of its two mistakes, so its Brier score is (6⋅0.0625+2⋅0.5625)/8=0.1875(6 \cdot 0.0625 + 2 \cdot 0.5625)/8 = 0.1875. Model B has 0.00250.0025 and 0.90250.9025, giving 0.22750.2275. The log losses are

LL⁡A=−68ln⁡0.75−28ln⁡0.25≈0.5623\begin{aligned} \operatorname{LL}_A &= -\tfrac{6}{8}\ln 0.75 \\[0.4em] &\quad - \tfrac{2}{8}\ln 0.25 \\[0.4em] &\approx 0.5623 \end{aligned}

and LL⁡B≈0.7874\operatorname{LL}_B \approx 0.7874. In the decomposition both models have resolution 0.06250.0625 and outcome term 0.250.25, while the reliability term is 00 for A and 0.040.04 for B. A third model C that predicts 0.50.5 for every case is calibrated on these eight cases, yet its Brier score 0.250.25 is worse than that of the miscalibrated model B, because C has no resolution. Its log loss ln⁡2≈0.6931\ln 2 \approx 0.6931 is better than that of B, so the two scores can even rank the same two models in opposite orders.

Platt scaling and isotonic regression

Definition Both methods learn a calibration map gg that sends a base model’s score ss to a probability g(s)g(s), using labeled pairs (si,yi)(s_i, y_i) from cases the base model was not trained on. The score can be a probability, a log-odds value or any other number that increases with risk. Platt scaling, in its basic form, uses

g(s)=11+e−(as+b)g(s) = \frac{1}{1 + e^{-(a s + b)}}

and chooses the two numbers aa and bb that minimize the log loss of the g(si)g(s_i) against the yiy_i, which is a logistic regression with the score as its only feature. Platt’s original version also replaces the targets 00 and 11 by values slightly inside the interval, and some libraries follow it. With a>0a > 0 the map is strictly increasing and keeps the order of the scores. Isotonic regression instead chooses gg among all nondecreasing functions to minimize

∑i(yi−g(si))2.\sum_{i} \bigl( y_i - g(s_i) \bigr)^2 .

Its fitted values at the calibration scores are unique. The pool adjacent violators algorithm finds them by sorting the distinct scores, starting from the observed event rate at each one, and repeatedly merging neighboring blocks whose rates decrease into one block with their pooled rate. Between and beyond the calibration scores an implementation needs a rule, for example linear interpolation inside the range and the end values outside it.

In plain words Platt scaling bends the scores along one S-shaped curve with a slope and a shift, so it needs little data but can only correct distortions of that shape. Isotonic regression only insists that a higher score never receives a lower probability, so it can follow any monotone distortion, at the price of noisier estimates and more overfitting on a small calibration set. Its steps can give several scores the same probability, and these new ties can change the AUC slightly. An increasing map never puts a lower-scored case above a higher-scored one, so calibration cannot fix a poor ranking.

Example A separate calibration set scored by model B has four cases at 0.050.05 with one event and four cases at 0.950.95 with three events. The observed rates 0.250.25 and 0.750.75 already increase, so isotonic regression returns them unchanged. Basic Platt scaling on the log-odds score z=ln⁡(pB/(1−pB))z = \ln\bigl(p_B/(1 - p_B)\bigr), which equals −ln⁡19-\ln 19 or ln⁡19\ln 19 here, fits b=0b = 0 and

a=ln⁡3ln⁡19≈0.3731,a = \frac{\ln 3}{\ln 19} \approx 0.3731 ,

which maps the two groups to exactly 0.250.25 and 0.750.75 as well, since two free numbers can match two increasing rates. A slope below 11 on the log-odds scale pulls overconfident probabilities toward 0.50.5. Applied to the eight evaluation cases, either map turns the predictions of model B into those of model A, so the calibrated model has Brier score 0.18750.1875 and log loss about 0.56230.5623. The data were constructed to make this agreement exact. On real data a calibration map is an estimate, and it can make predictions on new cases worse.

Calibration set

Definition A calibration set is a labeled set of cases used only to fit a calibration map, after the base model is trained and before the final evaluation. None of its cases may have been used to train the base model, and it is kept apart from the test set that measures the finished system. It should come from the population where the probabilities will be used, with that population’s class proportions, and with grouped or time-ordered data the split respects the groups or the time order. An alternative to a separate set is to calibrate on cross-validated predictions for the training data, where each training case is scored by a model fitted on folds that exclude it, so no case’s label trained the model that scored it. Choosing between calibration methods is also part of fitting and uses training or validation data, never the test set.

In plain words A model is usually more accurate and more confident on its own training cases than on new ones, because it has adapted to their labels. A calibrator fitted to those predictions learns to trust the model too much. Resampling and class weights belong to training only, the calibration and test sets keep the class mix of the deployed population, and a decision threshold is chosen afterward from the costs of errors without touching the test set.

Example A failure model is trained after the rare failures were oversampled until the two classes were equal in number. Suppose its probabilities are correct for that balanced mix, oversampling leaves the feature distribution within each class unchanged, and the failure rate in service is 0.020.02. A probability on the balanced scale then converts to the service scale by multiplying its odds by 0.02/0.980.02/0.98, so a prediction of 0.50.5 becomes 0.020.02 and a prediction of 0.90.9 becomes about 0.1550.155. A calibration set drawn from service data, with its natural failure rate of 22 percent, lets Platt scaling or isotonic regression learn such a correction instead of assuming it. Raising the decision threshold would change which machines are flagged and leave every one of these probabilities wrong.

Check yourself

  1. A model predicts probability 0.60.6 for each of ten cases in a separate evaluation group, and four of the ten have the event. Find the reliability point and the Brier score of the group. Does moving the decision threshold from 0.50.5 to 0.70.7 repair the calibration?
Show the answer

The reliability point is (0.6,0.4)(0.6, 0.4), below the diagonal, so the model overstated the event rate in this group. Each of the four events contributes (0.6−1)2=0.16(0.6 - 1)^2 = 0.16 and each of the six non-events contributes 0.62=0.360.6^2 = 0.36, so

BS⁡=4⋅0.16+6⋅0.3610=0.28.\begin{aligned} \operatorname{BS} &= \frac{4 \cdot 0.16 + 6 \cdot 0.36}{10} \\[0.4em] &= 0.28 . \end{aligned}

Moving the threshold to 0.70.7 changes all ten labels from positive to negative but leaves every probability at 0.60.6, so the reliability point and the Brier score do not change. A threshold acts on decisions, and calibration is a property of the probabilities. Ten cases are also weak evidence. If the true event probability were 0.60.6, four or fewer events among ten independent cases would occur with probability about 0.1660.166, so this group justifies a closer look rather than a verdict. Fitting a calibrator to these ten cases and then reporting them as fresh evidence would be leakage.

  1. A calibration set has five cases at each of the scores 0.10.1, 0.30.3, 0.50.5 and 0.70.7, with 11, 22, 11 and 44 events. Find the isotonic regression fit at the four scores. Why should these cases be kept out of the base model’s training data?
Show the answer

The observed rates are 0.20.2, 0.40.4, 0.20.2 and 0.80.8. The rate falls from 0.40.4 to 0.20.2 between the second and third scores, so those two blocks are pooled into one with rate (2+1)/10=0.3(2 + 1)/10 = 0.3. The values 0.20.2, 0.30.3, 0.30.3 and 0.80.8 are nondecreasing, so the fit is

g(0.1)=0.2,g(0.3)=g(0.5)=0.3,g(0.7)=0.8.\begin{gathered} g(0.1) = 0.2 , \\[0.4em] g(0.3) = g(0.5) = 0.3 , \\[0.4em] g(0.7) = 0.8 . \end{gathered}

Its sum of squared errors is 3.73.7, slightly above the 3.63.6 of the raw rates, which break the ordering. The scores 0.30.3 and 0.50.5 now share a probability. If these cases had trained the base model, its scores on them would reflect labels it had already fitted, typically looking more extreme and more accurate than on new cases. The calibrator would then learn a map that trusts the model too much and would be overconfident on new data.

  1. A team has records of 10,00010{,}000 machines, 22 percent of which failed. It oversamples the failures in the whole data set until half of the records are failures, splits the result at random into training, calibration and test sets, trains a classifier, fits isotonic regression on the calibration set, picks the threshold with the best F1 score on the test set, and reports the test Brier score. Find the problems and describe a sound order of steps.
Show the answer

Oversampling before splitting can put copies of the same failed machine into several sets, so the test set contains exact copies of training failures and the test results are optimistic. The calibration and test sets also have 5050 percent failures instead of the 22 percent seen in service, so the isotonic map is calibrated to the wrong population and its probabilities will be much too high in service, while the reported Brier score describes the balanced mix. Choosing the threshold on the test set uses the test labels for a decision, so the test F1 score is optimistic as well.

A sound order splits the original records first, by machine if a machine has several records and by time if the records are ordered in time. Oversampling, if it is used at all, is applied only to the training part, inside each training fold when cross-validation is used. The calibration and test sets keep the natural 22 percent failure rate. The calibrator and the threshold are fitted on the calibration or validation data, the threshold from the costs of the two kinds of error, and the test set is used once to report the Brier score or log loss with a reliability diagram, alongside measures of discrimination such as ROC AUC. The imbalance itself was never the problem. The leaks came from the order of the steps.

How this lesson connects