Embeddings and Transfer Learning, machine learning lesson

A new prediction task has only 24 labeled examples, while a network trained on hundreds of examples of a related task already exists. Training a new network from scratch on 24 examples ignores what the other network learned, and copying it unchanged ignores that the tasks differ. This lesson reads the inner layers of a network as learned feature vectors, called embeddings, and compares the ways of reusing them for a new task. A small constructed experiment shows what such a comparison can support and what it cannot.

Key terms

Embedding

Definition An embedding is a map $f$ that sends each object of some input set to a vector $\mathbf{z} = f(\mathbf{x})$ in $\mathbb{R}^d$ , and the vector of one particular object is also called its embedding. The activations of an inner layer of a trained network form an embedding of the network’s inputs. For a categorical input with $K$ possible values, an embedding layer is a learned $K \times d$ table whose row $k$ is the vector of value $k$ , which is the same as multiplying a one-hot vector by the table. Two embeddings $\mathbf{u}$ and $\mathbf{v}$ are often compared by

\begin{gathered} \lVert \mathbf{u} - \mathbf{v} \rVert_2 \quad \text{or} \\[0.4em] \cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u}^{\top}\mathbf{v}}{\lVert \mathbf{u} \rVert_2\, \lVert \mathbf{v} \rVert_2} . \end{gathered}

Cosine similarity is defined only when both vectors are nonzero. It lies in $[-1, 1]$ , it does not change when either vector is multiplied by a positive number, and it changes sign when one of them is multiplied by a negative number. For vectors of length 1 the two measures rank pairs the same way, a smaller distance going with a larger cosine, since then $\lVert \mathbf{u} - \mathbf{v} \rVert_2^2 = 2 - 2\cos(\mathbf{u}, \mathbf{v})$ .

In plain words An embedding gives each object coordinates in a learned space, and those coordinates usually have no human names. Two objects are close in that space when the training objective and data made them close. Closeness is useful for that purpose, but it is not a universal measure of meaning, and an embedding coordinate or a nearest neighbor is not a causal explanation of any outcome.

Example Take the constructed embeddings $\mathbf{u} = (2, 1)$ and $\mathbf{v} = (1, 2)$ . Their difference is $(1, -1)$ , so their Euclidean distance is $\sqrt{2} \approx 1.414$ . Their dot product is 4 and both lengths are $\sqrt{5}$ , so

\cos(\mathbf{u}, \mathbf{v}) = \frac{4}{\sqrt{5}\,\sqrt{5}} = 0.8 .

Replacing $\mathbf{u}$ by $10\mathbf{u} = (20, 10)$ raises the distance to $\sqrt{425} \approx 20.62$ and leaves the cosine at 0.8. Neither measure is right in general. Cosine ignores length, which loses information when the length carries useful signal and helps when the length mostly reflects an unwanted scale. The choice is settled by performance on the actual retrieval or prediction task, and a zero vector needs an explicit rule because its cosine is undefined.

Representation learning

Definition Representation learning means learning the feature map from data instead of designing the features by hand. In a model $\hat{y} = g_{\boldsymbol{\phi}}(f_{\boldsymbol{\theta}}(\mathbf{x}))$ , the parameters $\boldsymbol{\theta}$ of the representation $\mathbf{z} = f_{\boldsymbol{\theta}}(\mathbf{x})$ and the parameters $\boldsymbol{\phi}$ of the predictor are fitted together by minimizing one training objective. What the representation keeps is decided by that objective, the training data and the architecture. Some objectives need no labels, such as reconstructing the input, which Autoencoders and Nonlinear Representations takes up.

In plain words The hidden layers of a trained network are features the network built for itself because they lowered its training loss. They keep what that objective needed and may drop everything else, so a representation learned for one task makes no promise about another task.

Example In a constructed source task, each input $\mathbf{x} = (x_1, x_2)$ has coordinates drawn independently and uniformly from $[-1, 1]$ , and the label is 1 when $x_1 x_2 \ge 0$ . A network with one hidden layer of ten tanh units computes the embedding $\mathbf{z} = \tanh(W\mathbf{x} + \mathbf{b})$ , with a $10 \times 2$ weight matrix and tanh applied to each entry, followed by a logistic output. It was trained on 400 source examples, and its checkpoint was chosen by the lowest loss on 100 separate source validation examples, all of which it then classified correctly. Nothing told the hidden units about the product $x_1 x_2$ . Each input now has a ten-dimensional embedding shaped only by the source labels.

Pretrained model and transfer learning

Definition A pretrained model is a model whose parameters were fitted on a source task, with its own data and objective, and which is then reused. It comes with an input contract, the preprocessing it expects, such as scaling, channel order and image resolution, and it should come with a record of its source data, its version and the terms under which it may be used. Transfer learning uses a model or representation learned on a source task to help learn a target task, usually one with few labels. The source and target can differ in their input distributions, their labels or both. Transfer helps only when the source representation keeps the information the target needs, and it can hurt when the tasks or the input distributions differ too much.

In plain words Instead of starting from random weights, start from weights that already solved something related. Whether the source is related closely enough is an empirical question, answered with target data that played no part in fitting. Public availability of a model does not establish that a given use is permitted or that its source data suit the task, and pretraining data that overlap the target evaluation examples can make scores look better than they are.

Example The target task uses the same kind of inputs, but its label is 1 only when $x_1 x_2 \ge 0.1$ . Separate random draws give 24 target training examples, 80 validation examples and 80 test examples, with 7, 31 and 29 positives. Two procedures got the same budget of 500 epochs of mini-batch gradient descent with batch size 12 and learning rate 0.1, and each kept the epoch with the lowest validation loss. The first trains a logistic head on the frozen source embeddings, and the second trains a network of the same shape from scratch on the 24 examples.

Procedure	Validation loss	Test loss	Test correct
Frozen embedding and logistic head	0.1254	0.1592	75 of 80
Network trained from scratch	0.2988	0.3178	68 of 80

Losses are mean cross-entropy in nats, and predicting every test case negative would get 51 of 80 right. The transfer procedure also had the 500 source examples, so this is not a comparison with equal data. It is also one run of one constructed task. Both procedures kept their final epoch, so a longer budget might change the numbers, and nothing here shows that transfer wins in general.

Frozen feature extractor

Definition A frozen feature extractor is the part $f_{\boldsymbol{\theta}}$ of a pretrained model whose parameters stay at their pretrained values $\hat{\boldsymbol{\theta}}$ while a model for the target task is trained. With $\mathbf{z}_i = f_{\hat{\boldsymbol{\theta}}}(\mathbf{x}_i)$ , training solves

\min_{\boldsymbol{\phi}} \; \frac{1}{n} \sum_{i=1}^{n} L\bigl(y_i,\, g_{\boldsymbol{\phi}}(\mathbf{z}_i)\bigr)

over the head parameters only. Because the embeddings do not change, they can be computed once and cached, together with the exact preprocessing and model version that produced them. Freezing the parameters is separate from the mode a network runs in. A batch normalization layer left in training mode still updates its running statistics, and dropout still drops units, so a frozen extractor is normally run in evaluation mode. With a linear head and the logistic loss, the training problem is convex in $\boldsymbol{\phi}$ .

In plain words Freezing turns the pretrained network into a fixed function that converts each input into a feature vector, and a small model is then fitted to those vectors. Few parameters adapt to the scarce target labels, so there is less room to overfit and every comparison is cheap. The price is that the head can use only what the extractor kept.

Example In the experiment above, the extractor $\mathbf{z} = \tanh(W\mathbf{x} + \mathbf{b})$ has $20 + 10 = 30$ parameters, all held fixed. The logistic head outputs $\sigma(\mathbf{a}^{\top}\mathbf{z} + c)$ as its estimate of $P(Y = 1 \mid \mathbf{x})$ , where $\sigma$ is the logistic sigmoid. It has 11 trainable parameters, and fitting it is a logistic regression on the 24 embedded training examples, here run by gradient descent with the checkpoint rule above. The network trained from scratch has the same shape but trains all 41 parameters from random starting values.

Prediction head

Definition The prediction head $g_{\boldsymbol{\phi}}$ is the part of a model that maps an embedding $\mathbf{z}$ to the output the target task needs, such as one logit for a binary label, one logit per class, or a real number for regression. It is usually a linear layer or a small network trained on the target labels. A head reads only the embedding. If two inputs have the same embedding, $f(\mathbf{x}) = f(\mathbf{x}')$ , then every deterministic head gives them the same output, $g(f(\mathbf{x})) = g(f(\mathbf{x}'))$ . So when their target labels differ, at least one of the two is misclassified, whatever the size of the head.

In plain words A more flexible head can draw more complicated boundaries in the embedding space, but it cannot pull apart points that the extractor has already merged. Information lost by the extractor is lost for every head built on it.

Example An extractor that computes $\lvert x \rvert$ of a one-dimensional input sends both $-2$ and $2$ to the embedding 2. If the target label is 1 for positive inputs and 0 for negative inputs, every head gives the two inputs the same prediction, so one of them is wrong. Fine-tuning could only help if the extractor is a trainable model whose input still contains the sign and whose parameters are allowed to change.

Fine-tuning

Definition Fine-tuning starts from the pretrained parameters $\hat{\boldsymbol{\theta}}$ and continues to train a chosen subset of them, possibly all, together with the head, on the target training data. The remaining parameters stay fixed. A common procedure first trains the head with the extractor frozen and then unfreezes the later layers with a smaller learning rate. Which layers to unfreeze, the learning rates and the number of epochs are chosen on target validation data, and the test data are used once at the end. Earlier layers often hold more general patterns and later layers more task-specific ones, but that is a tendency to check, not a guarantee for every architecture and data set.

In plain words Fine-tuning lets the borrowed features move toward the new task. It has more freedom than a frozen extractor, so with few labels it can overfit or overwrite features that were useful, and it needs evidence that the extra freedom helps. Neither choice wins in general.

Example Fine-tuning here would train the 30 extractor parameters together with the 11 head parameters, 41 in all, the same number as the network trained from scratch, but starting from the source weights and the fitted head instead of random values. Whether that improves on the frozen head would be decided by comparing validation losses, with the frozen result of 0.1254 as the baseline to beat, before the test examples are touched.

Check yourself

Find the Euclidean distance and the cosine similarity of the embeddings $(1, 0)$ and $(1, 1)$ . What happens to each measure when the second vector is replaced by $(3, 3)$ , and what is the cosine similarity of $(1, 0)$ with the zero vector?

Show the answer

The difference is $(0, -1)$ , so the distance is 1. The dot product is 1 and the lengths are 1 and $\sqrt{2}$ , so the cosine similarity is $1/\sqrt{2} \approx 0.7071$ .

For $(3, 3)$ the difference is $(-2, -3)$ and the distance becomes $\sqrt{13} \approx 3.606$ . The cosine similarity stays $3/(1 \cdot 3\sqrt{2}) = 1/\sqrt{2}$ , because multiplying a vector by a positive number does not change its direction.

With the zero vector the denominator is 0, so the cosine similarity is undefined. An implementation has to state a rule for that case instead of letting it produce an arbitrary number.

A frozen extractor maps the inputs $-3$ and $3$ to the same embedding 9. The target label is 1 for positive inputs and 0 for negative inputs. A colleague proposes a deeper head with more units. Can any deterministic head classify both inputs correctly, and what should be tried instead?

Show the answer

No. Both inputs reach the head as the same number 9, so any deterministic head returns the same prediction for both, and one of the two labels is wrong. A deeper head adds flexibility in the embedding space but cannot recover the sign that the extractor discarded.

Better options keep the sign available. One is an extractor whose embeddings differ for the two inputs, which can be checked by searching the training data for pairs with equal embeddings and different labels. Another is to give the head the raw input alongside the embedding. A third is fine-tuning, which can help only if the extractor’s own input contains the sign and its parameters may change. Each option is compared on validation data.

A team has 300 labeled target images and a pretrained image model. They split the images into 200 for training and 100 for testing, try a frozen extractor with a new head and three fine-tuning settings, pick the setting with the best test accuracy, 91 percent, and report 91 percent as the accuracy to expect. Name the problems and describe a better protocol.

Show the answer

The test set chose among four settings, so 91 percent is the best of four noisy scores on the same 100 images and is biased upward. Selection needs its own data, either a validation split taken from the 200 training images or cross-validation within them, and the test images should be used once, for the setting already chosen.

Several other checks matter. The images must be prepared the way the pretrained model expects, and a frozen extractor should run in evaluation mode. Images of the same subject or from the same source should not be split across training and test. The test images should not appear in the model’s pretraining data, its license and documentation should permit this use, and the frozen extractor with a simple head, together with a trivial baseline such as the majority class, should be reported for comparison.

Even then the result estimates accuracy on images like these 300. It says nothing certain about conditions that the target data do not contain.

Embeddings and Transfer Learning

Key terms

Embedding

Representation learning

Pretrained model and transfer learning

Frozen feature extractor

Prediction head

Fine-tuning

Check yourself

How this lesson connects

Builds on

Used later in