Learning Curves and Error Analysis, machine learning lesson

A first classifier misclassifies 20 percent of its validation cases, and the next month could go into collecting more data, engineering features, trying a more flexible model or cleaning labels. The validation score alone cannot say which of these will help. A learning curve changes only the amount of training data and watches how training and validation error respond, and error analysis looks at which cases are wrong and what they have in common. This lesson shows how to read both as evidence for the next experiment, and why the change they suggest still needs a protected evaluation.

Key terms

Learning curve

Definition Fix a learning procedure, meaning the model class, its hyperparameters or a stated rule for tuning them, and every preprocessing step, each fitted inside the procedure. Fix also a loss $L$ and an evaluation design. For each training set size $m$ in a list $m_1 < m_2 < \dots < m_r$ , draw a training subset $S_m$ of $m$ examples from the development training data, fit the whole procedure on $S_m$ alone to get $\hat{f}_m$ , and write $\ell_i = L\bigl(y_i, \hat{f}_m(\mathbf{x}_i)\bigr)$ for the loss on example $i$ . The training error and the validation error at size $m$ are

\begin{aligned} E_{\text{train}}(m) &= \frac{1}{m} \sum_{i \in S_m} \ell_i , \\[0.4em] E_{\text{val}}(m) &= \frac{1}{\lvert V \rvert} \sum_{j \in V} \ell_j , \end{aligned}

where the validation set $V$ is disjoint from every training subset and, for grouped data, shares no group with them. The learning curve plots both errors against $m$ at the measured sizes. The training error is the empirical risk of $\hat{f}_m$ on the data it was fitted to. The validation error estimates the population risk of $\hat{f}_m$ , its expected loss on new cases from the population that $V$ represents. Repeating the draws at each size and averaging estimates the expected risk of the procedure at that size, and the spread across repeats shows how far one draw can move a point. With the zero-one loss of a classifier, both errors are error rates.

In plain words A learning curve is one controlled experiment in which only the amount of training data changes. A flexible model can fit a small training set almost perfectly, so its training error often starts low and rises as $m$ grows, while its validation error often falls. A large gap between the two curves that narrows as $m$ grows is consistent with high variance, and more data of the same kind may keep helping. Two curves that stay close together at a level too high for the task are consistent with underfitting, where more data from the same source is unlikely to help much and richer features or a more flexible model are better candidates. These are patterns, not laws. Neither curve has to be monotone, noise in a small validation set can hide or fake a trend, and a curve that retunes hyperparameters at each size describes the whole tuning procedure rather than one fixed model. Points exist only at the measured sizes, and lines drawn between them are not measurements.

Example The counts below are constructed for illustration, not results of a fitted benchmark. Two binary classifiers, one flexible and one rigid, are trained on 20, 40 and 80 cases and scored each time on the same 20 validation cases. Each entry is the number of errors over the number of cases scored.

Training size	Flexible, training	Flexible, validation	Rigid, training	Rigid, validation
$20$	$1/20$	$8/20$	$5/20$	$6/20$
$40$	$4/40$	$5/20$	$10/40$	$6/20$
$80$	$12/80$	$4/20$	$20/80$	$6/20$

For the flexible classifier the training error rises from $0.05$ to $0.10$ to $0.15$ , the validation error falls from $0.40$ to $0.25$ to $0.20$ , and the gap shrinks from $0.35$ to $0.15$ to $0.05$ . That is consistent with a procedure that overfits small samples and gains from more data. It does not prove that variance was the only problem or that doubling the data again would bring a similar gain. The rigid classifier has training error $0.25$ and validation error $0.30$ at every size, a small constant gap at a high level, which is consistent with underfitting. Each validation case is worth $0.05$ here, so the flexible model’s last improvement, from $0.25$ to $0.20$ , is a single case. With 20 independent validation cases, an error rate near $0.2$ has a standard error of about $\sqrt{0.2 \cdot 0.8 / 20} \approx 0.09$ .

Training set size

Definition The training set size $m$ is the number of examples the procedure is fitted on. For a learning curve the sizes are chosen in advance, and the subsets are drawn by a stated rule that matches how the data arose. Independent examples can be drawn at random, or stratified by class. Grouped data, such as several rows per customer, are drawn by whole groups, and the number of groups is reported next to $m$ . For time-ordered data the training subsets come from earlier periods and the validation cases from later ones. Nested subsets $S_{m_1} \subseteq S_{m_2} \subseteq \cdots$ make each larger subset add data to the previous one instead of replacing it, and several independent draws at each size, with their spread shown, reveal how much one draw can move a point.

In plain words What a larger training set adds is independent information, and the row count can overstate it. Duplicated rows, extra measurements of the same customers or a longer stretch of one season raise $m$ faster than the information grows. Where the added data come from matters too. More data help only to the extent that they resemble the cases the model will meet, and if the larger subsets reach into a different period, or the validation cases come after a change in operations, an apparent effect of size may really be an effect of time.

Example A constructed training pool has 800 rows from 100 customers, 8 rows each, and the validation set holds other customers. Subsets of 200, 400 and 800 rows drawn by customer contain 25, 50 and 100 customers. A subset of 400 rows could instead take 4 rows from each of the 100 customers. Both subsets have $m = 400$ , but one contains twice as many independent customers as the other, so the two designs can produce different curves. A curve labeled only by rows hides which design was used.

Error analysis

Definition Error analysis is the systematic inspection of a fitted model’s mistakes on development data, such as a validation set or the out-of-fold predictions of cross-validation. The mistakes are the false positives and false negatives of a classifier, the large residuals of a regression model or probability predictions far from the observed outcome. The analyst groups them by the conditions under which they occur, such as a feature range, a data source, missing values or label quality, and states each recurring pattern as a hypothesis about a cause. Candidate causes include absent signal, a model too restrictive for the pattern, failed optimization, inconsistent labels and information that is not available at prediction time. Each hypothesis is tested by one controlled change, evaluated under the same split design as before.

In plain words A score compresses every case into one number, and error analysis opens it back up. Reading the actual rows often shows that the errors are not spread evenly. The labels stay as recorded while the model is studied. Rewriting an outcome because the model disagrees with it makes the evaluation agree with the model by construction, so a label correction needs evidence from outside the model about what actually happened. If labels in some region are unreliable, more data from the same labeling process may teach the same mistakes more confidently.

Example With 80 training cases the flexible classifier above makes 4 errors on the 20 validation cases. Splitting the validation cases by the range of their main feature puts 8 cases in a range where the measured signal is weak and 12 in a range where it is strong, and all 4 errors fall in the low-signal range. Three explanations fit this, and each points to a different next step. If the classes truly overlap there, part of the error is irreducible with these features. If a missing feature would separate them, the step is to find one that is legitimately available at prediction time. If the labels of those cases are noisy, the step is to audit the labeling with outside evidence.

Error slice

Definition An error slice is a subset $S \subseteq \{1, \dots, n\}$ of the evaluation cases, picked out by a stated rule on recorded attributes such as a feature range, a data source, a device version or the true class, together with the model’s error on it. For a classifier the slice error is

e_S = \frac{\text{errors in } S}{\lvert S \rvert},

defined when $\lvert S \rvert \ge 1$ . If slices $S_1, \dots, S_G$ partition the $n$ evaluation cases, with sizes $n_g$ and slice errors $e_g$ , the overall error is their size-weighted average

e = \frac{1}{n} \sum_{g=1}^{G} n_g e_g .

A rule built from the errors themselves, such as the cases the model got wrong, defines a slice whose error is known in advance and says nothing new.

In plain words A good overall error can hide a slice where the model fails, because a small slice carries little weight in the average. The plain average of the slice errors equals the overall error only when the slices have equal sizes. Slices found by searching through many rules need care. The worst of many observed slice errors tends to overstate that slice’s true error, since chance alone makes some small slices look bad, so a slice found this way is a lead for development, not a finding.

Example In the validation set above, the low-signal slice has error $4/8 = 0.5$ and the high-signal slice has error $0/12 = 0$ . The weighted average

\frac{8 \cdot 0.5 + 12 \cdot 0}{20} = 0.2

matches the overall error of $4/20$ . To see how searching misleads, suppose a model has true error $0.2$ on every case, the cases are independent, and an analyst inspects 10 disjoint slices of 8 cases each. One given slice shows at least 4 errors with probability about $0.056$ , but at least one of the 10 does with probability $1 - 0.9437^{10} \approx 0.44$ .

Subgroup evaluation

Definition Subgroup evaluation reports the chosen metric separately for each named group of the evaluation population, such as a region, a device version, a feature range or a population of users, together with the group’s case count $n_g$ and, for class-based metrics such as recall, the count of each class in the group. Every group is scored with the same frozen model and the same evaluation data as the overall metric. Groups specified before the results are seen can support claims on a protected test set. Groups discovered during error analysis are hypotheses, and a claim about them needs data that were not used to find them.

In plain words A rate without its denominator cannot be judged. One error in a group of two cases moves that group’s error rate by 50 percentage points, and with independent cases an error rate $e$ on $n_g$ cases has a standard error of about $\sqrt{e(1 - e)/n_g}$ . Before proposing a modeling fix for a weak group, compare the groups on label quality, missing values, which features are available at prediction time and how well each group is represented in the training data. A fix is chosen on development data, and whether it works for the group is confirmed on protected cases.

Example The low-signal group has error $4/8 = 0.5$ with a standard error of about $\sqrt{0.5 \cdot 0.5 / 8} \approx 0.18$ , twice the standard error of about $0.09$ for the overall error on 20 cases. A report lists the counts $4/8$ and $4/20$ , not only the rates 50 percent and 20 percent. Suppose the team adds a feature, legitimately available at prediction time, that it expects to separate the classes in the low-signal range, and the validation error of that slice falls. The slice was found and the feature chosen on the validation cases, so the claim that the change helps low-signal cases is checked on test cases that played no part in the analysis.

Check yourself

A classifier is trained on 50, 100 and 200 cases and scored each time on the same 50 validation cases. It makes 2, 6 and 16 training errors and 15, 11 and 9 validation errors. Compute the training error, the validation error and the gap at each size, and say what the pattern suggests.

Show the answer

The training errors are $2/50 = 0.04$ , $6/100 = 0.06$ and $16/200 = 0.08$ . The validation errors are $15/50 = 0.30$ , $11/50 = 0.22$ and $9/50 = 0.18$ . The gaps are $0.26$ , $0.16$ and $0.10$ .

Training error rises, validation error falls and the gap narrows without closing. That is consistent with a variance problem that shrinks as data grow, so more data of the same kind is a reasonable next experiment. It is not proof. Each validation case is worth $0.02$ , so the last improvement is two cases, and the curve does not promise that the trend continues past 200 cases. Repeated draws at each size or a larger validation set would make the reading firmer.

A validation set has two slices. Slice A has 30 cases with 3 errors, and slice B has 10 cases with 4 errors. A colleague reports the overall error as the plain average of the two slice errors. What is wrong with that, what is the correct overall error, and what is needed before claiming that the model is worse on slice B if B was the worst of 40 slicing rules the colleague tried?

Show the answer

The slice errors are $3/30 = 0.10$ and $4/10 = 0.40$ , and their plain average $0.25$ treats a slice of 10 cases as if it had as many cases as a slice of 30. The overall error is

\begin{aligned} \frac{30 \cdot 0.10 + 10 \cdot 0.40}{40} &= \frac{7}{40} \\[0.4em] &= 0.175 . \end{aligned}

Slice B is small, so one error moves its rate by $0.10$ , and it was chosen as the worst of 40 candidates, which favors slices that look bad by chance. Its rule should be written down and fixed, and the model’s error on that slice measured on protected cases that played no part in the search, reported with its count of errors and cases.

A model has training error $0.27$ and validation error $0.29$ when trained on 200 cases, and $0.26$ and $0.28$ when trained on 800 cases. A manual review finds that a feature needed to distinguish two common outcomes is missing at prediction time. What should be investigated before paying to collect four times as many records of the same kind?

Show the answer

The gap is $0.02$ at both sizes, and quadrupling the training data lowered the validation error by only $0.01$ . Both errors stay close together and barely move, which gives little evidence that more of the same data will remove the main limitation. The review points to missing information rather than too little data.

Investigate whether some other feature that is genuinely available at prediction time can separate the two outcomes, whether the labels for those outcomes are reliable, and whether the current model can express the relationship in the features it already has. The missing feature must not be added just because it appears later in the database, since using information that will not exist at prediction time is leakage. A more flexible model cannot recover information that is absent from its inputs. Each candidate change is tested under the same development design, and the selected improvement is confirmed on protected cases. The curve guides the next experiment but does not establish the cause of the errors.

Learning Curves and Error Analysis

Key terms

Learning curve

Training set size

Error analysis

Error slice

Subgroup evaluation

Check yourself

How this lesson connects

Builds on

Used later in