Covariance, Eigenvectors, and PCA

Definition Let $X$ be an $n \times p$ data matrix whose rows $\mathbf{x}_1^{\top}, \dots, \mathbf{x}_n^{\top}$ are observations and whose columns are features. The mean vector is $\bar{\mathbf{x}} = \frac{1}{n} \sum_{i=1}^{n} \mathbf{x}_i$, and centering replaces each row $\mathbf{x}_i$ by $\mathbf{x}_i - \bar{\mathbf{x}}$. The centered matrix $X_c$ has column sums 0. For every vector $\mathbf{v}$ in $\mathbb{R}^p$,

so squared projections of uncentered rows mix the spread of the data with the position of the mean. In a predictive pipeline $\bar{\mathbf{x}}$ is computed on the training rows only, and the same $\bar{\mathbf{x}}$ is subtracted from validation, test and new rows. Centering is distinct from standardizing, which also divides each centered feature by its standard deviation, and a PCA should state which of the two was done.

Definition For $n \ge 2$ rows, the sample covariance matrix is the $p \times p$ matrix

Its diagonal entry $C_{jj}$ is the sample variance of feature $j$, and its entry $C_{jk}$ off the diagonal is the sample covariance of features $j$ and $k$, the sum of the products $(x_{ij} - \bar{x}_j)(x_{ik} - \bar{x}_k)$ over the rows divided by $n - 1$. The matrix is symmetric, and it is positive semidefinite, because for every $\mathbf{v}$

Dividing by $n$ instead of $n - 1$ multiplies every entry by $(n - 1)/n$, which rescales all eigenvalues by that factor and leaves every eigenvector unchanged. Each entry carries the product of the units of its two features. When the rows are independent draws from one distribution with finite variances, $C$ is an unbiased estimate of that distribution’s covariance matrix $\Sigma$. The matrix $C$ is computed from the sample, while $\Sigma$ belongs to the population.

Definition Let $A$ be a $p \times p$ matrix. A vector $\mathbf{v}$ is an eigenvector of $A$ with eigenvalue $\lambda$ when

The eigenvalues are the roots of $\det(A - \lambda I) = 0$. Every nonzero multiple of an eigenvector is again an eigenvector with the same eigenvalue, so eigenvectors are usually scaled to length 1, which still leaves a choice of sign. For a real symmetric matrix the spectral theorem gives real eigenvalues $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_p$ and an orthonormal basis $\mathbf{e}_1, \dots, \mathbf{e}_p$ of eigenvectors, so that

For a covariance matrix every eigenvalue is at least 0, since $\lambda = \mathbf{e}^{\top} C\, \mathbf{e} \ge 0$ for a unit eigenvector $\mathbf{e}$. When an eigenvalue of a symmetric matrix is repeated $r$ times, its eigenvectors together with $\mathbf{0}$ form a subspace of dimension $r$, its eigenspace, and every orthonormal basis of that subspace serves equally well.

Definition Let $\mathbf{v}$ be a unit vector, $\lVert \mathbf{v} \rVert_2 = 1$. The projection score of row $i$ along $\mathbf{v}$ is $z_i = (\mathbf{x}_i - \bar{\mathbf{x}})^{\top}\mathbf{v}$, the signed coordinate of the centered row along the line through the origin in direction $\mathbf{v}$. The scores have mean 0, and their sample variance, the projected variance along $\mathbf{v}$, is

The unit length matters. Replacing $\mathbf{v}$ by $c\mathbf{v}$ multiplies every score by $c$ and the variance by $c^2$, so without the constraint the variance would have no maximum. Writing $\mathbf{v} = a_1 \mathbf{e}_1 + \dots + a_p \mathbf{e}_p$ in an orthonormal eigenvector basis of $C$, with $a_1^2 + \dots + a_p^2 = 1$, gives

a weighted average of the eigenvalues with the weights $a_j^2$.

Definition Let $C$ be a sample covariance matrix with eigenvalues $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_p \ge 0$. A first principal direction is a unit vector $\mathbf{v}_1$ that maximizes $\mathbf{v}^{\top} C\, \mathbf{v}$ over all unit vectors $\mathbf{v}$. Since the projected variance is a weighted average of eigenvalues,

the maximum equals $\lambda_1$, and it is attained exactly at the unit vectors of the eigenspace of $\lambda_1$. The Lagrangian $\mathbf{v}^{\top} C\, \mathbf{v} - \lambda(\mathbf{v}^{\top}\mathbf{v} - 1)$ leads to the same answer. Its gradient with respect to $\mathbf{v}$ is $2C\mathbf{v} - 2\lambda\mathbf{v}$ for symmetric $C$, which vanishes exactly when $C\mathbf{v} = \lambda\mathbf{v}$, so every constrained stationary point is a unit eigenvector, with projected variance equal to its eigenvalue. For $k = 2, \dots, p$, a $k$th principal direction maximizes $\mathbf{v}^{\top} C\, \mathbf{v}$ over unit vectors orthogonal to $\mathbf{v}_1, \dots, \mathbf{v}_{k-1}$. It can be taken as a unit eigenvector for $\lambda_k$, and the maximum is $\lambda_k$. The principal directions are mutually orthogonal. Each is fixed only up to sign, and when an eigenvalue is repeated only the subspace spanned by its directions is fixed.

Definition Principal component analysis (PCA) takes an $n \times p$ training matrix with $n \ge 2$ and a number of components $q$ with $1 \le q \le p$, and computes every statistic from the training rows only.

1. Compute the training mean $\bar{\mathbf{x}}$ and center every row with it.
2. If the features are to be standardized, divide each centered feature by its training standard deviation, which must be positive.
3. Form $C$ from the prepared training rows and find orthonormal eigenvectors $\mathbf{v}_1, \dots, \mathbf{v}_q$ for its $q$ largest eigenvalues.
4. Give any row $\mathbf{x}$, from training or new, the scores $z_j = \tilde{\mathbf{x}}^{\top}\mathbf{v}_j$ for $j = 1, \dots, q$, where $\tilde{\mathbf{x}}$ is $\mathbf{x}$ prepared by steps 1 and 2 with the training statistics.

The scores of the training rows have sample variances $\lambda_1, \dots, \lambda_q$ and are uncorrelated, since $\mathbf{v}_j^{\top} C\, \mathbf{v}_k = \lambda_k \mathbf{v}_j^{\top}\mathbf{v}_k = 0$ for $j \ne k$. For PCA on centered features without standardization, the reconstruction of $\mathbf{x}$ from its $q$ scores is the point $\hat{\mathbf{x}} = \bar{\mathbf{x}} + z_1 \mathbf{v}_1 + \dots + z_q \mathbf{v}_q$. PCA uses no target values, so it is an unsupervised method. Its directions change with the units of the features unless the features are standardized, each direction is fixed only up to sign, and when $\lambda_q = \lambda_{q+1}$ the choice of the $q$th direction is not unique.