Autoencoders and Nonlinear Representations, machine learning lesson

Five constructed points lie exactly on a parabola in the plane, so one number, the position along the curve, determines each of them. Yet the best one-dimensional summary that PCA can give, a straight line, passes through none of the five points. This lesson builds an encoder that compresses each point to one number and a decoder that rebuilds the point from it, measures the rebuilding error on points that were not used for fitting, and compares a curved decoder with PCA. It also shows why a small reconstruction error is not, by itself, evidence of a useful representation.

Key terms

Autoencoder, encoder and decoder

Definition An autoencoder consists of an encoder $f_{\boldsymbol{\theta}}$ , which maps an input $\mathbf{x}$ in $\mathbb{R}^d$ to a code $\mathbf{z}$ in $\mathbb{R}^m$ , and a decoder $g_{\boldsymbol{\phi}}$ , which maps the code back to a reconstruction $\hat{\mathbf{x}} = g_{\boldsymbol{\phi}}(f_{\boldsymbol{\theta}}(\mathbf{x}))$ in $\mathbb{R}^d$ . Both are trained together to make the reconstruction close to the input itself, by minimizing the average reconstruction loss over the training inputs,

\min_{\boldsymbol{\theta},\, \boldsymbol{\phi}} \; \frac{1}{n} \sum_{i=1}^{n} \ell\bigl(\mathbf{x}_i,\, \hat{\mathbf{x}}_i\bigr) ,

usually by gradient descent with backpropagation. No labels are needed, because each input is its own target. Any preprocessing, such as centering or scaling, is fitted on the training inputs, the architecture and settings are chosen by reconstruction error on validation inputs, and test inputs are used once at the end.

In plain words The encoder writes a short summary of each input and the decoder tries to rebuild the input from that summary alone. The model learns what to put into the summary only through the score of the rebuild.

Example The constructed training points are $\mathbf{x} = (t, t^2)$ for $t = -1, -0.5, 0, 0.5, 1$ , with dimensionless coordinates. A deliberately tiny autoencoder has the encoder $z = s x_1$ , which reads only the first coordinate, and the decoder $\hat{\mathbf{x}} = (cz,\ az^2 + b)$ , with trainable parameters $s$ , $c$ , $a$ and $b$ . Squaring is its nonlinearity, so this model has a strong built-in preference for parabolas and is not a general-purpose autoencoder. With errors $e_1 = cz - x_1$ and $e_2 = az^2 + b - x_2$ , written $e_{1i}$ and $e_{2i}$ for point $i$ , the training loss $L$ is the mean over the five points of $\tfrac12(e_1^2 + e_2^2)$ . The parameter $s$ affects both outputs, so its gradient adds two paths,

\begin{aligned} \frac{\partial L}{\partial s} &= \frac{1}{5} \sum_{i=1}^{5} \bigl(e_{1i}\, c\, t_i \\[0.4em] &\qquad + 2\, e_{2i}\, a z_i t_i\bigr) . \end{aligned}

Starting from $s = 0.8$ , $c = 1$ , $a = 0.7$ and $b = 0.1$ , 4,000 gradient descent steps with step size 0.05 end at $s \approx 1.0612$ , $c \approx 0.9424$ , $a \approx 0.8880$ and $b$ within $10^{-14}$ of 0. Then $cs \approx 1$ and $as^2 \approx 1$ , so the decoder returns $(t, t^2)$ up to rounding error.

Bottleneck and latent representation

Definition The latent representation, or code, of an input is the encoder’s output $\mathbf{z} = f_{\boldsymbol{\theta}}(\mathbf{x})$ . A bottleneck is a code layer of smaller dimension than the input, $m < d$ , and an autoencoder with one is called undercomplete. When $m \ge d$ and the layers are flexible enough, copying is available. For example, the linear encoder that appends $m - d$ zeros to $\mathbf{x}$ and the linear decoder that drops them reproduce every input exactly. The reconstruction objective then gives no reason to prefer any other code over plain copying, so such an overcomplete autoencoder relies on another constraint, such as corrupted inputs, a sparsity penalty on the code or a weight penalty, to discourage the identity. Codes are also not unique. If $h$ is any invertible map of $\mathbb{R}^m$ to itself, the encoder $h \circ f$ and the decoder $g \circ h^{-1}$ give exactly the same reconstructions as $f$ and $g$ .

In plain words The bottleneck is the narrow point that forces a summary, and the code is the summary. A bottleneck limits how much passes through, but the reconstruction objective decides what passes. Because many different codes rebuild the inputs equally well, a single latent coordinate has no fixed meaning merely because the reconstructions are good.

Example The curved autoencoder above has $d = 2$ and $m = 1$ . In the run described, the point with $t = 0.75$ gets the code $z = s \cdot 0.75 \approx 0.7959$ . Doubling $s$ , halving $c$ and dividing $a$ by 4 leaves $cs$ and $as^2$ unchanged, so every reconstruction stays the same, while the code of the same point becomes about $1.5918$ . Neither code is more correct than the other.

Reconstruction loss

Definition A reconstruction loss measures how far $\hat{\mathbf{x}}$ is from $\mathbf{x}$ . For real-valued inputs the usual choice is squared error, reduced in one of several ways that differ by constant factors,

\begin{gathered} \text{SSE}_i = \lVert \mathbf{x}_i - \hat{\mathbf{x}}_i \rVert_2^2, \\[0.4em] \text{MSE} = \frac{1}{nd} \sum_{i=1}^{n} \text{SSE}_i , \end{gathered}

the sum of squared errors of one example and the mean squared error per coordinate, and half of either is also common because it simplifies derivatives. A constant factor leaves the minimizer unchanged but scales the gradients and the reported number, so the reduction has to be stated. Error that the code cannot predict sets a floor. Under squared loss, the best decoder output for a part of the input given the code is its conditional mean, and the expected conditional variance remains as error that no decoder can remove. Training loss describes fit to the training inputs, and only held-out inputs show how well new inputs are rebuilt.

In plain words Reconstruction loss scores how well the rebuild matches the original, in squared units of the input. Zero is not a reachable target when the inputs carry noise that the code does not keep, and a low score on training inputs says nothing by itself about new inputs.

Example An input $(2, 3)$ rebuilt as $(1.5, 2.5)$ has errors $-0.5$ and $-0.5$ , so its SSE is $0.5$ and its MSE per coordinate is $0.25$ . Half the SSE is also $0.25$ , because $d = 2$ , which is why the training loss of the curved autoencoder equals its MSE per coordinate. If the second coordinate of the parabola data were $t^2$ plus independent noise with mean 0 and variance $\sigma^2$ , a code that keeps only $t$ could at best rebuild $t^2$ , and the expected MSE per coordinate on new points could not fall below $\sigma^2/2$ .

Linear autoencoders and PCA

Definition Let the training inputs $\mathbf{x}_1, \dots, \mathbf{x}_n$ in $\mathbb{R}^d$ be centered at their mean, with covariance $S = \frac{1}{n}\sum_i \mathbf{x}_i \mathbf{x}_i^{\top}$ and eigenvalues $\lambda_1 \ge \dots \ge \lambda_d \ge 0$ . A linear autoencoder with code size $m < d$ has encoder $\mathbf{z} = E\mathbf{x}$ and decoder $\hat{\mathbf{x}} = D\mathbf{z}$ , with $E$ of size $m \times d$ and $D$ of size $d \times m$ , and is trained on

\frac{1}{n} \sum_{i=1}^{n} \lVert \mathbf{x}_i - DE\mathbf{x}_i \rVert_2^2 .

The smallest possible value is $\lambda_{m+1} + \dots + \lambda_d$ . If $\lambda_m > \lambda_{m+1}$ , every pair $(E, D)$ that attains it gives each training input its PCA reconstruction with $m$ components. If in addition $S$ is invertible, then $DE = U_m U_m^{\top}$ , where the columns of $U_m$ are orthonormal eigenvectors for the $m$ largest eigenvalues, and the code is $\mathbf{z} = B\,U_m^{\top}\mathbf{x}$ for some invertible $m \times m$ matrix $B$ . With bias terms and uncentered inputs, the same holds for the inputs minus their training mean, and the code may also be shifted by a constant vector. The theorem describes the minimum, and a training run has to reach it.

In plain words A linear autoencoder can do no better than PCA, and at its best it finds the same subspace, the one PCA with SVD and reconstruction projects onto. Its code is an invertible linear mixture of the principal component scores, so its coordinates need not equal those scores, need not be uncorrelated and need not be ordered by variance. When $\lambda_m = \lambda_{m+1}$ , even the subspace is not determined by the data.

Example For the five parabola points the mean is $(0, \tfrac12)$ , and with denominator $n = 5$ the covariance is

S = \begin{bmatrix} 1/2 & 0 \\ 0 & 7/40 \end{bmatrix} ,

since the symmetric values of $t$ make the cross term $\tfrac15 \sum t_i (t_i^2 - \tfrac12)$ equal to 0. The larger eigenvalue belongs to the horizontal direction, so the one-component PCA reconstruction of $(t, t^2)$ is $(t, \tfrac12)$ , and the smallest average SSE is $\lambda_2 = 7/40 = 0.175$ , or $0.0875$ per coordinate. Here $S$ is invertible and $\lambda_1 > \lambda_2$ , so a linear autoencoder with $m = 1$ that reaches this minimum on the centered points has a code equal to a nonzero multiple of $t$ , and after the mean is added back it rebuilds every point as $(t, \tfrac12)$ .

Nonlinear dimensionality reduction

Definition Dimensionality reduction maps each input $\mathbf{x}$ in $\mathbb{R}^d$ to a representation $\mathbf{z}$ in $\mathbb{R}^m$ with $m < d$ that keeps the structure needed for a stated purpose. It is linear when the encoder and decoder are linear, as in PCA, and then the reconstructions lie in an affine subspace of dimension at most $m$ . It is nonlinear when the encoder or the decoder is nonlinear, so that the reconstructions can lie on a curved surface of dimension $m$ , such as a curve when $m = 1$ . An autoencoder with nonlinear layers and a bottleneck is one such method.

In plain words When data lie near a curve or a curved surface, a flat subspace of the same dimension cannot follow it, so PCA may need more components to reach the same error. A nonlinear decoder can bend. Whether that helps depends on the data, and the comparison should keep the same inputs, the same error definition and the same roles for training and held-out data while only the model family changes.

Example Evaluate both one-number summaries on the held-out values $t = -0.75, -0.25, 0.25, 0.75$ , none of which was used for fitting. PCA, fitted on the five training points, rebuilds $(t, t^2)$ as $(t, \tfrac12)$ . Its first coordinates are exact, and its second-coordinate errors are $-0.0625$ at $t = \pm 0.75$ and $0.4375$ at $t = \pm 0.25$ , so the eight squared errors sum to $0.390625$ and

\begin{aligned} \text{MSE}_{\text{PCA}} &= \frac{0.390625}{8} = \frac{25}{512} \\[0.4em] &\approx 0.0488 . \end{aligned}

The trained curved autoencoder rebuilds the same points with an MSE of about $2 \times 10^{-30}$ , which is rounding error. That is possible only because the data come from a noise-free curve lying exactly inside the chosen model family. It is not evidence that autoencoders rebuild real data perfectly.

Denoising autoencoder

Definition A denoising autoencoder is trained on corrupted inputs $\tilde{\mathbf{x}}$ , drawn from a chosen corruption distribution $q(\tilde{\mathbf{x}} \mid \mathbf{x})$ such as added Gaussian noise or randomly zeroed entries, with the clean input as the target,

\begin{gathered} \min_{\boldsymbol{\theta},\, \boldsymbol{\phi}} \; E\bigl[\lVert \mathbf{x} - \hat{\mathbf{x}} \rVert_2^2\bigr], \\[0.4em] \hat{\mathbf{x}} = g_{\boldsymbol{\phi}}\bigl(f_{\boldsymbol{\theta}}(\tilde{\mathbf{x}})\bigr), \end{gathered}

where in practice the expectation is replaced by an average over training inputs and sampled corruptions. When clean inputs have finite second moments, the best reconstruction function of the corrupted input over all functions is the conditional mean $E[\mathbf{x} \mid \tilde{\mathbf{x}}]$ . Because the network input and the target now differ, copying is no longer optimal for corruptions such as added independent noise with positive variance, even with an overcomplete code.

In plain words Damage the input on purpose and ask for the undamaged version. To undo the damage the model has to learn what typical clean inputs look like. The corruption process is a design choice, and the model learns to undo that kind of damage, not every kind.

Example Take the clean point $(0.5, 0.25)$ on the parabola and a corruption that changes only its second coordinate to $0.4$ , a squared error of $0.15^2 = 0.0225$ in that coordinate. The curved autoencoder trained above, given $(0.5, 0.4)$ , returns $(0.5, 0.25)$ up to rounding, because its encoder reads only the first coordinate. That cleanup comes from an architecture chosen by hand, not from denoising training. If the corruption moves the first coordinate to $0.6$ instead, the same model returns $(0.6, 0.36)$ , which misses the clean point by $0.1$ and $0.11$ , an MSE per coordinate of $0.01105$ .

Check yourself

An input $(4, 0, 2, 1)$ is rebuilt as $(3, 1, 2, 1)$ . Find the sum of squared errors, the mean squared error per coordinate and half the sum of squared errors. Why must a report say which of these it calls the loss?

Show the answer

The errors are $-1$ , $1$ , $0$ and $0$ , so the sum of squared errors is 2. Dividing by the four coordinates gives an MSE per coordinate of 0.5, and half the sum is 1.

All three are common, and they differ by constant factors that depend on the convention and on the input dimension. The same model would be reported as 2, 0.5 or 1, so a number without its reduction cannot be compared with another model’s number.

An autoencoder for two-dimensional inputs uses a three-dimensional code and reaches a training reconstruction error of exactly 0. A colleague concludes that the code is an excellent representation. What is wrong with that conclusion, and what evidence would help?

Show the answer

A code of dimension 3 for inputs of dimension 2 allows plain copying. The linear encoder $\mathbf{z} = (x_1, x_2, 0)$ and the decoder that drops the last coordinate rebuild every input exactly, training and new inputs alike, while the code is just the input again. Zero error is therefore possible without learning anything about the data.

Better evidence starts with reconstruction error on held-out inputs for a model that cannot simply copy, for example one with a bottleneck, corrupted inputs or a penalty on the code, compared with a baseline such as PCA with the same code size. If the code is meant for a downstream task, freeze it, train a head on training data, choose settings on validation data, and compare with a head trained on the raw inputs. Low reconstruction error, even on held-out data, does not show that the code serves that task.

Four centered training points are $(1, 0)$ , $(-1, 0)$ , $(0, 1)$ and $(0, -1)$ . A linear autoencoder with a one-dimensional code is trained on them, and a classmate reports that it found the principal direction $(1, 1)/\sqrt{2}$ . Compute the covariance and the average SSE for that direction and for $(1, 0)$ , and explain what the result means.

Show the answer

The covariance with denominator 4 is

S = \frac14 \begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix} = \begin{bmatrix} 1/2 & 0 \\ 0 & 1/2 \end{bmatrix} ,

so both eigenvalues equal $\tfrac12$ . Projecting onto the unit vector $\mathbf{u} = (1, 1)/\sqrt{2}$ sends $(1, 0)$ to $(\tfrac12, \tfrac12)$ , an SSE of $\tfrac14 + \tfrac14 = \tfrac12$ , and by symmetry each of the four points has SSE $\tfrac12$ , so the average is $\tfrac12$ . Projecting onto $(1, 0)$ gives SSE 0 for the first two points and 1 for the other two, again an average of $\tfrac12$ .

For any unit vector $\mathbf{u}$ the average SSE is $\operatorname{tr}(S) - \mathbf{u}^{\top}S\mathbf{u} = 1 - \tfrac12 = \tfrac12$ , the minimum $\lambda_2$ of the theorem. Since $\lambda_1 = \lambda_2$ , the condition $\lambda_m > \lambda_{m+1}$ fails, every direction is equally good, and no direction is the principal one. The direction the network ended with reflects its initialization and training, not a property of the data.

Autoencoders and Nonlinear Representations

Key terms

Autoencoder, encoder and decoder

Bottleneck and latent representation

Reconstruction loss

Linear autoencoders and PCA

Nonlinear dimensionality reduction

Denoising autoencoder

Check yourself

How this lesson connects

Builds on

Used later in