Semi-Supervised and Self-Supervised Learning

Definition Semi-supervised learning fits a model for a supervised task from a labeled set and an unlabeled set of inputs from the same input space,

where the labels of the inputs $\mathbf{u}_j$ are not available to the learner and usually $m$ is much larger than $n$. Unlabeled inputs describe only the distribution of the inputs, so they can help predict labels only through an assumption that links that distribution to the labels. A common one is the cluster assumption, that inputs in one dense region tend to share a label, which suggests that a decision boundary should run through regions where inputs are rare. A semi-supervised procedure also declares its data access. An inductive procedure produces a rule for any future input, and it is judged on labeled inputs that took no part in fitting in any form, with or without their labels. A transductive procedure only has to label the particular inputs in $D_U$, which may be exactly the inputs that need predictions, and it is judged on a random sample of those inputs whose labels are obtained for the audit and kept out of training. If the unlabeled inputs come from a different population than the cases the model will face, or if their clusters follow something other than the label, adding them can make the model worse than a model fitted on $D_L$ alone.

Definition Pseudo-labeling, a form of self-training, is a semi-supervised procedure for classification. For a binary task with a confidence threshold $c$ in $(0.5, 1)$ it runs as follows.

1. Fit a classifier on the labeled set $D_L$.
2. For each unlabeled input $\mathbf{u}_j$, compute the predicted probability $p_j$ of class 1 and the pseudo-label $\tilde{y}_j$, which is 1 when $p_j \ge 0.5$ and 0 otherwise.
3. Accept the pseudo-label only when the prediction is confident, that is, when $p_j \ge c$ or $p_j \le 1 - c$.
4. Refit on the labeled set together with the accepted pairs $(\mathbf{u}_j, \tilde{y}_j)$, and if desired repeat from step 2 with the new model.

With $K$ classes the pseudo-label is the class with the largest predicted probability, and it is accepted when that probability is at least $c$. A pseudo-label is a model output, not an annotation. The threshold selects by the model’s confidence, and confidence is not correctness, since a poorly calibrated model, or one applied to inputs unlike its training data, can be confidently wrong. An accepted mistake becomes a training target, which tends to shift the boundary so that inputs like the mislabeled one get the same wrong class, and so to produce more mistaken pseudo-labels in later rounds, a failure known as confirmation bias. Calibration does not remove the errors either. If the probabilities are calibrated, so that among cases given probability $p$ a fraction $p$ is positive, the expected number of wrong pseudo-labels is the sum of $\min(p_j, 1 - p_j)$ over the accepted cases, which is positive unless every accepted probability is exactly 0 or 1. The threshold and the number of rounds are tuned on validation data with genuine labels, the final evaluation uses genuinely labeled cases kept outside the whole selection and refitting procedure, and a report counts pseudo-labels separately from supplied labels.

Definition Let $\mathcal{T}$ be a family of random transformations chosen so that each one preserves the correct label for the task, and let $\mathbf{p}_{\boldsymbol{\theta}}(\mathbf{x})$ be the model’s vector of predicted class probabilities. Consistency regularization adds to the supervised loss a penalty for disagreement between the predictions on two views $T_1\mathbf{u}$ and $T_2\mathbf{u}$ of the same input, with $T_1$ and $T_2$ drawn independently from $\mathcal{T}$ each time the input is used,

where $\ell$ is a supervised loss such as cross-entropy, $\lambda \ge 0$ sets the strength of the penalty, and

compares the two probability vectors with a discrepancy $d$, such as the squared distance. With $\lambda = 0$ the objective is ordinary supervised training, and the penalty is often applied to labeled inputs as well. A transformation is label-preserving for a task when every input the task covers keeps its correct label after the transformation, which depends on the task and not on the transformation alone. Penalizing disagreement under a transformation that changes the label teaches the model a wrong invariance. A model that outputs the same probability vector for every input has zero penalty, so consistency cannot replace the supervised term or another signal that forces the model to tell inputs apart.

Definition Self-supervised learning trains a model on a pretext task whose targets are constructed from unlabeled inputs by a declared procedure, without using labels of the task the model is finally meant for. Typical pretext tasks hide part of an input and predict it, such as a masked patch of an image or a masked word, predict the next token of a sequence, or identify which of several candidates is another view of the same example. The objective is as explicit as a supervised loss. Only the source of the targets differs, and those targets reflect design choices, such as what is hidden and which views count as the same, rather than categories revealed by the data. The usual aim is a representation, an encoder $f_{\boldsymbol{\theta}}$ that maps inputs to vectors reused by a downstream supervised task. The encoder is then either frozen, with a small prediction head trained on the labeled training split, or fine-tuned under a declared validation procedure. A low pretext loss does not show that the representation helps downstream, because a shortcut, such as matching image borders or acquisition artifacts, can solve the pretext task without capturing what the downstream task needs. Pretraining choices, such as the pretext task, the augmentations and the training length, are tuned on validation data, and the final test set is used once, after all of them are fixed. If the pretraining pool contains the test inputs without labels, the report says so. That access suits a transductive question about those inputs, but for a claim about unseen future inputs, test inputs or near-copies of them in the pretraining pool make the score less informative.

Definition Contrastive learning is a self-supervised method that trains an encoder by comparing views. A view is an input after a random transformation from a family chosen to keep what downstream tasks need. The encoder maps each view to an embedding, often normalized to unit length, and the similarity of two embeddings is their dot product, which equals the cosine similarity when both have unit length. For an anchor view, the positive pair is the anchor with another view of the same original example, and the negative pairs are the anchor with views of other examples, often the rest of the batch. Let $\mathcal{C}$ be the set of candidates, the positive and the negatives but never the anchor itself, let $s_j$ be the anchor’s similarity to candidate $j$ and $s_{+}$ its similarity to the positive, and let $\tau > 0$ be a temperature. The contrastive loss of the anchor, often called InfoNCE, is

and the batch loss averages it over anchors. It is the cross-entropy of a softmax over the candidates whose correct class is the positive, so its targets come from sample identity and augmentation, not from class labels. A smaller temperature sharpens the softmax, so differences in similarity count for more. Two different examples of the same class still form a negative pair, a false negative that a setup based only on sample identity cannot avoid. Representation collapse is the failure in which distinct inputs receive identical or nearly identical embeddings, so the embedding keeps little or no information about the input. In a partial form the embeddings vary along only a few directions. If all embeddings are equal, every similarity is the same, each candidate gets probability $1/\lvert \mathcal{C} \rvert$, and the loss is $\log \lvert \mathcal{C} \rvert$ at every temperature. Negative pairs make that solution costly. Methods without negative pairs rely on other mechanisms, such as asymmetric networks, stopped gradients or constraints on the statistics of the embeddings.

Definition Active learning spends a labeling budget on examples chosen by a rule. Starting from a small labeled set and a pool of unlabeled inputs, it repeats three steps until the budget is used. It fits a model on the current labeled set, applies an acquisition rule to choose one or more pool inputs, and has a person supply their true labels, which join the labeled set. Uncertainty sampling, the simplest acquisition rule for a binary classifier with predicted probabilities $p_j$, requests

with a tie-breaking rule declared in advance. Other rules favor diverse inputs or the expected improvement of the model. The requested labels are genuine annotations, unlike pseudo-labels. Uncertainty sampling relies on the model’s own probabilities, so it gives the lowest priority to inputs on which the current model is confidently wrong. The labeled set it produces is chosen by the rule rather than drawn at random, so it cannot serve as an evaluation set for claims about the intended population, which needs a separately drawn representative sample. Its class proportions and mix of inputs can differ from those of the population, which can distort calibration and any population estimate computed from it, so a report states how the examples were selected.