Distribution Shift and Robustness, machine learning lesson

A classifier errs on 14 percent of the cases it was developed on and on 26 percent of the cases it meets later, although the link between features and outcomes has not changed at all. The population changed, and an error rate always belongs to a population. This lesson defines the main kinds of distribution shift by what changes and what is assumed to stay fixed, shows when reweighting predicts the new error rate, and turns robustness to change into specific checks, boundaries and responses.

Key terms

Distribution shift

Definition Let $P_s$ , the source distribution, be the distribution of feature and label pairs $(\mathbf{X}, Y)$ from which the training and evaluation data were drawn, and let $P_t$ , the target distribution, describe the cases the model meets in use. A distribution shift is a difference $P_t \ne P_s$ . For a fixed model $f$ and a loss $L$ , the risk under a distribution $P$ is the expected loss for $(\mathbf{X}, Y) \sim P$ ,

R_P(f) = E_P\bigl[L(f(\mathbf{X}), Y)\bigr],

written $R_s(f)$ and $R_t(f)$ for the two populations. For the zero-one loss $L = \mathbf{1}\{f(\mathbf{X}) \ne Y\}$ it is the probability of an error. A test error computed on cases drawn from $P_s$ estimates $R_s(f)$ , and it says nothing exact about $R_t(f)$ unless an assumption links the two distributions. The joint distribution factors in two ways,

\begin{aligned} P(\mathbf{X}, Y) &= P(\mathbf{X})\,P(Y \mid \mathbf{X}) \\[0.4em] &= P(Y)\,P(\mathbf{X} \mid Y), \end{aligned}

and each named kind of shift states which factor changes and which factor is assumed to stay the same.

In plain words An error rate is an average over cases, so it is not a fixed property of a model. Changing which cases arrive changes the average even when the model and its behavior on each kind of case stay exactly the same. A statement about performance in use becomes checkable only when it says how the populations differ and what is assumed to carry over. A monitor that compares input histograms can raise an alarm for a change that does not affect the risk, and it can stay silent while the risk rises, so an alarm is a reason to investigate and a quiet monitor is not evidence of reliability.

Example A constructed inspection task has one feature, workload, which is low or high. In both populations a case needs inspection with probability 0.1 when workload is low and 0.7 when it is high. The fixed classifier predicts no inspection for low workload and inspection for high workload, so it errs with probability 0.1 on low-workload cases, the missed inspections, and 0.3 on high-workload cases, the unnecessary ones. The source population is 80 percent low workload and the target is 20 percent low workload, so

\begin{aligned} R_s &= 0.8 \cdot 0.1 + 0.2 \cdot 0.3 \\[0.4em] &= 0.14, \\[0.4em] R_t &= 0.2 \cdot 0.1 + 0.8 \cdot 0.3 \\[0.4em] &= 0.26 . \end{aligned}

The model and its error on each kind of case stayed the same. Only the mixture of cases changed.

Covariate shift

Definition Covariate shift is a distribution shift in which the distribution of the features changes while the conditional distribution of the label given the features stays the same,

\begin{gathered} P_t(\mathbf{X}) \ne P_s(\mathbf{X}), \\[0.4em] P_t(Y \mid \mathbf{X}) = P_s(Y \mid \mathbf{X}), \end{gathered}

the second equation holding at every feature value the target can produce. Write $p_s$ and $p_t$ for the probabilities or densities of the features, and $r(\mathbf{x}) = E[L(f(\mathbf{X}), Y) \mid \mathbf{X} = \mathbf{x}]$ for the conditional risk at $\mathbf{x}$ , which is the same in both populations under covariate shift. If also $p_s(\mathbf{x}) > 0$ wherever $p_t(\mathbf{x}) > 0$ , the overlap or support condition, then with the importance weight $w(\mathbf{x}) = p_t(\mathbf{x})/p_s(\mathbf{x})$ ,

R_t(f) = E_s\bigl[w(\mathbf{X})\,r(\mathbf{X})\bigr],

so the target risk equals a weighted average over the source population, and it can be estimated by weighting the losses of labeled source cases. Both conditions are needed. Without overlap the weight is undefined where the source has no cases, and no weighting can supply labels for them. In finite samples large weights make the estimate noisy, estimated weights can be unstable when there are many features, and clipping the weights lowers the variance but biases the estimate.

In plain words Under covariate shift the target contains the same kinds of cases as the source in different proportions, and each kind behaves as before. Reweighting the source cases to the target proportions then reproduces the target error rate. Covariate shift is an assumption, not an observation. A changed feature histogram shows that the features moved, but whether the label still depends on the features in the same way can only be checked with labels from the target.

Example The weights are $0.2/0.8 = 0.25$ for low workload and $0.8/0.2 = 4$ for high workload. Weighting the source errors gives $0.8 \cdot 0.25 \cdot 0.1 + 0.2 \cdot 4 \cdot 0.3 = 0.02 + 0.24 = 0.26$ , the target risk. The share of cases that need inspection moves as well, from $0.8 \cdot 0.1 + 0.2 \cdot 0.7 = 0.22$ to $0.2 \cdot 0.1 + 0.8 \cdot 0.7 = 0.58$ , although the probability of needing inspection at each workload level is fixed. In a source sample of 100 cases with 80 low and 20 high, the weights add to 100 and the 20 high-workload cases carry 80 of that total. A common way to summarize the cost of unequal weights is the effective sample size

\frac{(\sum_i w_i)^2}{\sum_i w_i^2} = \frac{100^2}{325} \approx 30.8 ,

so the weighted estimate behaves roughly like an average over 31 cases, not 100. If the source had no high-workload cases at all, the weight for high workload would be undefined.

Label shift and concept shift

Definition Label shift, also called prior probability shift, is a distribution shift in which the label distribution changes while the distribution of the features within each class stays the same,

\begin{gathered} P_t(Y) \ne P_s(Y), \\[0.4em] P_t(\mathbf{X} \mid Y) = P_s(\mathbf{X} \mid Y). \end{gathered}

For a binary label, Bayes’ rule then gives the target posterior from the source posterior. Let $O_s(\mathbf{x})$ and $O_t(\mathbf{x})$ be the odds of $Y = 1$ given $\mathbf{x}$ , and $\rho_s$ and $\rho_t$ the prior odds of $Y = 1$ , with both classes present in both populations. Then

O_t(\mathbf{x}) = O_s(\mathbf{x}) \cdot \frac{\rho_t}{\rho_s} .

Concept shift is a distribution shift in which the conditional distribution of the label given the features changes, $P_t(Y \mid \mathbf{X}) \ne P_s(Y \mid \mathbf{X})$ at feature values both populations produce. The feature distribution may change or stay the same. Terminology varies, and some sources use concept drift for any change over time. These named kinds are assumptions about what stays fixed, and a real shift can fit none of them exactly. A change in the base rate alone does not identify label shift, because covariate shift and concept shift can change the base rate too.

In plain words Under label shift each class still looks the same, but the classes arrive in different proportions, as when a condition becomes more common while its signs stay the same. Under concept shift the same features now point to different labels, as when the rule for what needs inspection changes. They call for different evidence. For a binary label under the label-shift assumption, the new class proportions can be estimated from unlabeled target inputs, as long as the classifier’s true and false positive rates differ, while concept shift generally needs new labels to be detected and corrected.

Example In a separate constructed example, the share of cases needing inspection rises from 0.2 in the source to 0.5 in the target, and in both populations high workload has probability 0.8 among cases that need inspection and 0.2 among those that do not. In the source, high workload has probability $0.2 \cdot 0.8 + 0.8 \cdot 0.2 = 0.32$ , and the probability of needing inspection given high workload is $0.16/0.32 = 0.5$ . In the target these become $0.5 \cdot 0.8 + 0.5 \cdot 0.2 = 0.5$ and $0.4/0.5 = 0.8$ . The prior odds rise from 0.25 to 1, a factor of 4, and the posterior odds given high workload rise from 1 to 4. The classifier that flags high workload has true positive rate 0.8 and false positive rate 0.2 in both populations, so its error rate is 0.2 in both. Its errors do not reveal the shift, yet a probability of 0.5 for high workload, calibrated in the source, is far too low in the target.

For concept shift, return to the source mixture of 80 percent low workload and let a case need inspection with probability 0.4 at either workload level. The classifier now errs with probability 0.4 on low-workload cases and 0.6 on high-workload cases, so its risk is $0.8 \cdot 0.4 + 0.2 \cdot 0.6 = 0.44$ . The feature distribution did not change, so a monitor that only watches the inputs sees nothing.

Stress test

Definition A stress test evaluates a fixed model under a stated family of changes to its data distribution or its inputs, and reports its performance across that family. A claim of robustness to a set of changes names the model, the performance criterion with its acceptable level, the population, and the family of changes, and it asserts that the criterion stays acceptable throughout that family. A test that perturbs inputs synthetically also needs valid labels for the perturbed cases, since a change that alters the true outcome cannot keep the old label.

In plain words A stress test replaces the vague question of whether a model is reliable with a specific one, how a stated measure behaves as a stated aspect of the data changes, and where it crosses the acceptable level. It is a transparent calculation or experiment, not a guarantee. The changes tested may not be the ones that occur, and conclusions drawn from estimated error rates carry the uncertainty of those estimates.

Example Keep the classifier and its errors of 0.1 and 0.3 on the two workload levels fixed, and let the share of high-workload cases be $h$ , with $0 \le h \le 1$ . Then

\begin{aligned} R(h) &= 0.1\,(1 - h) + 0.3\,h \\[0.4em] &= 0.1 + 0.2\,h , \end{aligned}

which is 0.1, 0.2 and 0.3 at $h = 0$ , $0.5$ and $1$ . If the intended use requires an error rate of at most 0.2, the model meets it exactly when $h \le 0.5$ , under the assumption that the errors on each workload level stay fixed. A short data-use note belongs with such a report. For this example it says that the data are constructed, contain no personal records and authorize no real inspection decision. For a real data set it would state the permitted purpose, the source permissions, the fields needed, who may access them and how long they are kept.

Out-of-distribution input and abstention

Definition An out-of-distribution input is an input outside the support of the distribution the model was developed on, or in a region that distribution represents too sparsely to support predictions there. Abstention is a designed option for a system to decline an automated prediction for an input and route it to a stated alternative process, such as human review. For a system that predicts on some inputs and abstains on others, the coverage $c$ is the probability of predicting and the selective risk $r$ is the expected loss among the predicted cases,

\begin{gathered} c = P(\text{predict}), \\[0.4em] r = E[L \mid \text{predict}], \end{gathered}

the latter defined when $c > 0$ . A detector flags only what its rule checks, so it can miss inputs that are unfamiliar in ways it does not measure, and a model can give a confident and wrong prediction on an unfamiliar input.

In plain words A model has evidence only about the kinds of input it was developed on. When an input falls outside them, the honest options are to flag it and handle it another way, not to force it into a familiar category. Abstaining trades coverage for a lower error on the cases the model keeps, so a report needs both numbers together with what happens to the abstained cases, whose handling has its own capacity, quality and delay.

Example The inspection pipeline receives the workload value extreme, a category that never occurred in development. Mapping it silently to low or high would invent a prediction rule without evidence, while a schema check that rejects unknown categories can route these cases to review. Abstention has its own cost. In the target population of the first example, a system that abstains on every high-workload case has coverage 0.2 and selective risk 0.1. It sends 80 percent of the cases to review, and its low error rate on the cases it keeps says nothing about the cases it passes on.

Feedback loop

Definition A feedback loop arises when a model’s decisions affect the data later used to train or evaluate it, for example by deciding which outcomes are observed or by changing the behavior of the people or processes that the model scores. The data collected afterward reflect the decision policy as well as the population. When a label is recorded only for cases that were acted on, the outcomes of the other cases are missing, not negative. Treating them as negatives understates the base rate whenever some of them were positive, and it can bias any model trained on the records.

In plain words A model that decides where to look also decides what gets seen. Retraining on what was seen can lock in its earlier choices, because outcomes in places it never looked stay unknown. Keeping a record of which cases were selected and why, and collecting some outcomes outside the model’s choices where that is possible and acceptable, keeps the next data set honest about what it shows.

Example Suppose that in the source population only the inspected cases, the high-workload ones, get an outcome recorded, and that every uninspected case is stored as not needing inspection. The true share needing inspection is $0.8 \cdot 0.1 + 0.2 \cdot 0.7 = 0.22$ , but the recorded share is $0.2 \cdot 0.7 = 0.14$ , and every low-workload case that needed inspection appears as a negative. A model retrained on these records would learn that low workload never needs inspection, which makes future inspections of low-workload cases even less likely and keeps their outcomes unrecorded.

Check yourself

A source population has 25 percent high-workload cases and a target population has 50 percent. A classifier errs with probability 0.2 on low-workload cases and 0.4 on high-workload cases in both populations. Find the source and target risks and the two importance weights, and check the weighted average. A colleague adds that an input monitor has stayed quiet since launch, so the model is safe for any future population. Is that right?

Show the answer

The risks are

\begin{aligned} R_s &= 0.75 \cdot 0.2 + 0.25 \cdot 0.4 \\[0.4em] &= 0.25, \\[0.4em] R_t &= 0.5 \cdot 0.2 + 0.5 \cdot 0.4 \\[0.4em] &= 0.3 . \end{aligned}

The weights are $0.5/0.75 = 2/3$ for low workload and $0.5/0.25 = 2$ for high workload. Both are defined because the source contains both workload levels. The weighted source average is $0.75 \cdot \tfrac23 \cdot 0.2 + 0.25 \cdot 2 \cdot 0.4 = 0.1 + 0.2 = 0.3$ , the target risk.

The colleague’s claim is unsupported. A quiet monitor says only that its particular test did not fire. It cannot show that the conditional error rates are stable, since a concept shift can leave the inputs unchanged, and no test of past inputs covers every future population. A defensible statement names the intended population, reports labeled evaluations under the shifts that are plausible there with their uncertainty, sets a boundary for unsupported inputs such as abstaining on unknown categories, and plans how delayed outcome labels will be collected and checked.

Assume the label-shift setting of the lesson, where high workload has probability 0.8 among cases that need inspection and 0.2 among those that do not, in every population. A new population shows 44 percent high-workload cases, and no labels are available yet. Estimate the share of cases needing inspection and the probability of needing inspection given high workload. When would this estimate be wrong?

Show the answer

With $\pi$ the share needing inspection, the share of high workload is

\begin{aligned} &0.8\,\pi + 0.2\,(1 - \pi) \\[0.4em] &\quad = 0.2 + 0.6\,\pi , \end{aligned}

and setting it to 0.44 gives $\pi = 0.24/0.6 = 0.4$ . Then the probability of needing inspection given high workload is $0.4 \cdot 0.8 / 0.44 = 8/11 \approx 0.727$ , and given low workload it is $0.4 \cdot 0.2 / 0.56 = 1/7 \approx 0.143$ .

The calculation relies on the label-shift assumption that each class keeps its distribution of workload. If the rule for what needs inspection changed, or if the workload mix changed while the inspection probability at each level stayed fixed, the same 44 percent would mean something else and the estimate would be wrong. It also needs the two class rates 0.8 and 0.2 to differ, since otherwise the workload share carries no information about $\pi$ , and in a finite sample the observed 44 percent and therefore $\pi$ carry sampling error.

A source population contains only low-workload cases, with error rate 0.1. The target population is 70 percent low workload and 30 percent high workload. Can importance weighting estimate the target risk? What can be said, and what would make the answer more precise?

Show the answer

No. The weight for high workload would be $0.3/0$ , which is undefined, because the overlap condition fails. The source contains no labeled high-workload cases, so no weighting of source cases can say how the classifier performs on them.

If the error on low-workload cases is still 0.1 in the target, the target risk is $0.7 \cdot 0.1 + 0.3\,e$ , where $e$ is the unknown error rate on high-workload cases. Since $0 \le e \le 1$ , the risk lies somewhere between 0.07 and 0.37, a range too wide to support most claims. To narrow it, collect labeled high-workload cases from the target, or justify an explicit assumption about $e$ . Until then, the system can abstain on high-workload cases and route them to review, which gives coverage 0.7 with the error rate 0.1 measured in the source on the kind of cases it keeps.

Distribution Shift and Robustness

Key terms

Distribution shift

Covariate shift

Label shift and concept shift

Stress test

Out-of-distribution input and abstention

Feedback loop

Check yourself

How this lesson connects

Builds on

Used later in