A small network sorts images of bars into vertical and horizontal and gets all 40 of its test images right. Whether that score means anything depends on choices made long before testing, such as which images counted as independent, which edits were allowed to keep their labels, and what the network was compared with. This lesson follows one small image project through those choices and shows exactly how much its final score supports. The images are generated by a program for teaching, so every pixel and every label can be checked.
Key terms
Image classification
Definition Image classification assigns each image one label from a fixed set of classes. The input is an array of pixel intensities, of shape for a grayscale image or for a color image, and the classifier usually returns a probability for each class. With it returns one number, the modeled probability for the image array , and a declared threshold turns that number into a predicted class. A claim about the classifier’s performance refers to a population of images, fixed by the label definition and by how the images arise, including which subjects, cameras, lighting and preprocessing are involved. It is estimated on new images from that population and says nothing direct about images outside it. A classifier uses whatever pattern in the pixel values predicts the training labels, whether that is the intended shape or an accidental cue such as brightness, background or camera.
In plain words The model sees a grid of numbers, not objects. A high test score shows that something in those numbers predicts the label on images like the test images. Which something, and how far it carries beyond those images, are separate questions.
Example In a constructed task each image is a grid of dimensionless intensities, with label for a vertical bar and for a horizontal one. A generator draws a subject, meaning an orientation, a bar position in one of the four inner rows or columns, a background level between 0 and 0.15 and a bar that adds an amplitude between 0.6 and 1 to the background. It then makes two views of the subject by adding independent Gaussian noise with standard deviation 0.04 to every pixel. Both orientations light six pixels, so the average intensity of an image has the same distribution for both classes and carries no information about orientation. A small sample can still show an accidental difference. In the 60 training images the vertical bars average about 0.231 and the horizontal bars about 0.199. The first candidate, a brightness baseline, uses exactly that average,
and has two parameters. The second, a small CNN, applies four filters with biases , stride 1 and no padding, so each image gives four response maps. After ReLU each map is averaged over its 16 positions, and one linear unit with weights and bias turns the four averages into a logit,
where is plus the sum of the nine products of filter with the window of image whose top left pixel sits in row and column . The shapes run from to , then to and to , and the parameter count is . Each model predicts a vertical bar when its logit is at least 0, which is the same as a probability of at least 0.5. The third candidate, a frozen-feature model, is described in the last card.
Subject-level split
Definition A subject is the source behind one or more images, such as a person, a physical object, a patient or a generated base pattern. A subject-level split assigns each subject, together with all of its images, to exactly one role, training, validation or test, before any image is augmented or used for fitting. Repeated photos, near-duplicates and augmented copies of a subject always share its role. It is a group split with the subject as the group, and it is the right split for a model that will be used on new subjects. The subject key must identify a subject uniquely across the whole study, and images whose subject was never recorded, such as one photo saved twice under two names, have to be matched and kept together before the split. Images of one subject are correlated, so the number of independent units in a test set is its number of subjects, and uncertainty is computed by subject, for example by resampling subjects together with all their images.
In plain words If the model will meet new people, objects or patterns, every image of a given one belongs on one side of the line. Otherwise the test lets the model recognize a familiar subject and credits that as recognizing the class.
Example In the bar project, separate random seeds generate 30 training, 16 validation and 20 test subjects with two views each, giving 60, 32 and 40 images. Each collection numbers its subjects from 0, so a subject’s key is its collection and its number together, and a key made of the number alone would merge unrelated subjects. After model selection the chosen CNN classifies all 40 test images correctly. The evidence consists of 20 independent subjects, not 40 independent images. The test subjects are independent draws from the generator and the model was fixed before testing. If a new subject had probability of at least one misclassified view, all 20 test subjects would be fully correct with probability , which is below 0.05 when
So 0.139 is an exact one-sided 95 percent upper confidence bound for . A randomly chosen view can be misclassified only if its subject has a misclassified view, so the error rate per image is at most and the bound covers it too. Treating the 40 images as independent trials would give the narrower bound , which the shared position, brightness and background of the two views of a subject do not justify. A bootstrap over subjects would put every resample at 100 percent accuracy and show no spread at all, a limitation of that method when there are no errors, not evidence of certainty.
Data augmentation
Definition Data augmentation adds training examples made by applying transformations from a declared family to training inputs, each paired with a label given by a declared rule, usually the original label. The new examples can be stored once as fixed copies or drawn afresh at random each time an example is used. Augmentation is applied after the subject-level split and to training data only, and every copy keeps the role of its subject. Copies are further views of known subjects, so they do not raise the number of independent units, and their value depends on whether the transformations reflect changes that the label should ignore. Validation and test images keep the generating procedure of the population being evaluated. Test-time augmentation, which averages the predictions for several altered versions of each test image, is part of the inference procedure, so it is chosen on validation data before the final evaluation. Adding it after seeing test errors makes the test part of model selection. The family of transformations, their probabilities and their order belong in the reproducible record of the project.
In plain words Augmentation shows the model the same subjects in more ways, so it can learn which changes should not matter. It encodes a belief about which changes leave the label alone, and it adds no new subjects.
Example The 60 training images of the bar project receive horizontally mirrored copies.
X_aug = np.concatenate((X_train, X_train[:, :, ::-1]))
y_aug = np.concatenate((y_train, y_train))A mirrored vertical bar in the second column moves to the fifth column and stays vertical, and a mirrored horizontal bar keeps its row. Every copy keeps its label and still looks like an image the generator could produce. The training set grows to 120 images, still from 30 subjects, while the 32 validation and 40 test images are left exactly as generated. All three candidates train on the same 120 images.
Label-preserving augmentation
Definition Let be the set of images the task covers and the correct label of an image in under the task’s label definition. A transformation is label-preserving on if, for every image in , the image is again in and . Whether a transformation preserves labels depends on the label definition and on the images, not on its name, and calling an operation augmentation does not make it safe. A transformation that changes the label in a known way, such as one that swaps two classes, can be used only with the label changed to match, and only when the result is a valid image of its new class. The same requirement applies to the views used in consistency regularization and contrastive learning.
In plain words Before keeping a copy with its old label, ask whether someone who knows the task would still give it that label. If the honest answer is that it depends on the image, the transformation is not safe as a blanket rule.
Example For the bar task a horizontal or a vertical mirror preserves every label. A quarter-turn rotation does not, since it turns a vertical bar into a horizontal one and the reverse. A crop that removes most of the bar can leave no clear orientation, and a dimming that lowers the bar to the background level erases the contrast that defines it. A deliberately broken check shows what a label-changing transformation does to an evaluation. Rotate each of the 40 test images by a quarter turn and keep the old labels. The selected CNN then disagrees with all 40, with 20 false positives and 20 false negatives, and its mean cross-entropy against the old labels rises from about 0.0048 to about 5.97. Against the correct labels its decisions agree with all 40. The accuracy of 0 out of 40 measured broken labels, not a loss of recognition.
Frozen-feature model
Definition A frozen-feature model combines a feature extractor , whose parameters were learned beforehand on a source task, with a new prediction head trained on the target task. During target training stays fixed and only the head’s parameters are updated. For a binary target with head weights and bias the modeled probability is
It differs from training from scratch, where all parameters start at random values and are fitted on target data, and from fine-tuning, where starts from the source values and is updated as well. The source data and the computation spent on them are extra resources and are reported with any comparison. A fair comparison gives every candidate the same target training data, the same validation data and selection rule, and the same test set, used once after the selection. The source data must not contain test subjects or near-copies of them, or the test score leaks.
In plain words Reuse a feature extractor trained elsewhere and fit only a small final layer for the new task. That needs few target labels and little computation, but the model can only use the features the source task produced, and whether they suit the new task is settled on validation data, not assumed.
Example In the bar project the extractor is the CNN’s convolution layer, trained on a separate source collection of 80 training and 20 validation subjects with the same orientation label. Its 36 filter weights and 4 filter biases are frozen, and a new head with 5 parameters is trained on the 120 augmented target images. Every candidate is trained by full-batch gradient descent with step size 0.5 for 600 epochs and keeps the parameters from the epoch with the lowest validation cross-entropy, which turned out to be the last epoch for all three. The validation set has 32 images from 16 subjects, and cross-entropy is measured in nats per image.
| Candidate | Parameters fitted on target data | Validation cross-entropy | Correct of 32 |
|---|---|---|---|
| Brightness baseline | 2 | 0.6840 | 21 |
| CNN from scratch | 45 | 0.005905 | 32 |
| Frozen-feature CNN | 5 | 0.007607 | 32 |
The rule declared in advance selects the lowest validation cross-entropy, which belongs to the CNN trained from scratch, although both networks classify every validation image correctly. The baseline’s 0.6840 is only a little below , the cross-entropy of always predicting 0.5. The selected CNN then classifies all 40 test images correctly, with mean cross-entropy about 0.0048. That the frozen-feature model came second here does not show that transfer is worse in general, and a win would not have shown that it is better. Both results describe this generator and these settings.
Check yourself
- An already selected classifier is evaluated on 20 independent cases and gets 8 true positives, 2 false negatives, 1 false positive and 9 true negatives. Compute its recall, precision and accuracy.
Show the answer
There are actual positives, so the recall is . The classifier predicts positive times, so the precision is . Accuracy counts every correct decision, which gives .
- A project has 12 subjects photographed 3 times each, and the 36 images are split at random into 24 training and 12 test images. What is the probability that a given subject has photos on both sides? How many subjects are expected to be split this way, and how likely is it that no subject is split?
Show the answer
A subject stays whole when all three of its photos land in training or all three land in test. These events have the probabilities
so a given subject is split with probability . By linearity of expectation about of the 12 subjects are split. No subject is split only when the 12 test images are the photos of exactly 4 whole subjects, which has probability
A random image split almost surely puts the same subjects on both sides, so its test score mixes class recognition with recognition of familiar subjects. Splitting the 12 subjects instead, for example 8 for training and 4 for test, keeps them apart by construction.
- A project classifies symbols as upright or upside down, where upside down means rotated by a half-turn. It has 12 subjects photographed three times each. The author makes a half-turned copy of every photo with its original label, splits all 72 images at random, and reports near-perfect accuracy. Name the two main problems and describe a corrected design.
Show the answer
First, a half-turn changes the label in this task. It turns an upright symbol into an upside-down one and the reverse, so each copy of a symbol that changes under a half-turn carries the wrong label. A copy can be used only with its label switched, and only for symbols that look different after a half-turn. A symbol such as S, N, Z, H, O or X looks the same both ways in common typefaces, so it has no orientation label and does not belong in this task at all.
Second, the random image split puts photos and copies of the same subject on both sides, so the test rewards recognizing familiar subjects. With wrong labels in training and shared subjects across the split, the near-perfect score cannot be read as evidence that the model judges orientation.
A corrected design assigns the 12 subjects to roles before anything else, for example 8 for training, 2 for validation and 2 for test, or uses cross-validation grouped by subject for development, and keeps all three photos of a subject together. Augmentation then runs on training images only, after the split, with changes that keep orientation, such as small shifts, mild changes of brightness or contrast and rotations by a few degrees, or half-turned copies with switched labels for symbols that have an orientation. Validation and test photos stay unaltered. With so few subjects any test estimate is imprecise, so the report gives its uncertainty by subject and describes the subjects, the image sources, the allowed transformations, a simple baseline, the selection procedure and the errors that remain.