Principal Component Analysis with Centering and Variance

Definition A data matrix is a real $N \times p$ matrix $X$ whose $N$ rows are observations and whose $p$ columns are features, the quantities recorded for each observation. Row $i$ is $\mathbf{x}_i^{\top}$ with $\mathbf{x}_i$ in $\mathbb{R}^p$. The mean of feature $j$ is the average $\bar{x}_j$ of column $j$, and the feature means form the mean vector

where every entry of $\mathbf{1}$ in $\mathbb{R}^N$ equals $1$. The centered data matrix $X_c$ has rows $(\mathbf{x}_i - \bar{\mathbf{x}})^{\top}$, that is

Every column of $X_c$ sums to zero, which says $X_c^{\top}\mathbf{1} = \mathbf{0}$. Adding the same vector $\mathbf{a}$ to every observation changes the mean to $\bar{\mathbf{x}} + \mathbf{a}$ and leaves $X_c$ unchanged. Centering subtracts the means and nothing else. It does not change units or divide by any measure of spread.

Definition Let $N > 1$ and let $X_c$ be the centered $N \times p$ data matrix. The sample covariance matrix is the $p \times p$ matrix

Multiplying $X_c^{\top}$ by $X_c$ column by row shows that $(N - 1)C$ is the sum of the outer products $(\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^{\top}$ of the centered observations. The diagonal entry $c_{jj}$ of $C$ is the sample variance of feature $j$, the sum of the squared centered values in column $j$ divided by $N - 1$. For $j \ne k$, the entry $c_{jk}$ is the sample covariance of features $j$ and $k$, the sum of the products of their centered values divided by $N - 1$. The matrix $C$ is symmetric and positive semidefinite, since $\mathbf{w}^{\top}C\mathbf{w} = \lVert X_c\mathbf{w} \rVert^2/(N - 1) \ge 0$ for every $\mathbf{w}$. Its trace, the sum of the feature variances, is the total variance. Dividing by $N$ instead of $N - 1$ gives mean squared deviations, which equal the sample variances times $(N - 1)/N$, and this page uses $N - 1$ throughout. A covariance of zero says only that the products sum to zero. It does not mean that the features are statistically independent.

Definition Let $\mathbf{w}$ be a unit vector in $\mathbb{R}^p$. The score of observation $i$ along $\mathbf{w}$ is $t_i = \mathbf{w} \cdot (\mathbf{x}_i - \bar{\mathbf{x}})$, and the scores form the vector $\mathbf{t} = X_c\mathbf{w}$ in $\mathbb{R}^N$. They have mean zero, because $\mathbf{1}^{\top}X_c\mathbf{w} = 0$, and their sample variance, the projected variance along $\mathbf{w}$, is

The point $\bar{\mathbf{x}} + t_i\mathbf{w}$ is the orthogonal projection of $\mathbf{x}_i$ onto the line through $\bar{\mathbf{x}}$ with direction $\mathbf{w}$. The condition $\lVert \mathbf{w} \rVert = 1$ matters, because replacing $\mathbf{w}$ by $c\mathbf{w}$ multiplies the quadratic form by $c^2$ without choosing a better direction.

Definition Let $C$ be the sample covariance matrix. By the spectral theorem it has real eigenvalues and an orthonormal basis of eigenvectors $\mathbf{w}_1, \dots, \mathbf{w}_p$ with $C\mathbf{w}_j = \lambda_j\mathbf{w}_j$. The eigenvalues are nonnegative because $C$ is positive semidefinite, and they are listed here in decreasing order $\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_p \ge 0$. Writing a unit vector as $\mathbf{w} = \sum_j \alpha_j\mathbf{w}_j$ with $\sum_j \alpha_j^2 = 1$ gives

with equality at $\mathbf{w} = \mathbf{w}_1$. So the first principal direction $\mathbf{w}_1$ has the largest projected variance, namely $\lambda_1$. In the same way, $\mathbf{w}_k$ has the largest projected variance among unit vectors perpendicular to $\mathbf{w}_1, \dots, \mathbf{w}_{k-1}$, and that variance is $\lambda_k$. The $k$th principal component is the new variable measured along $\mathbf{w}_k$, with scores $X_c\mathbf{w}_k$ and sample variance $\lambda_k$. Many texts also call the direction $\mathbf{w}_k$ itself a principal component. A direction can be replaced by its negative, which flips the sign of every score and changes nothing else, and when an eigenvalue is repeated any orthonormal basis of its eigenspace may be used. In terms of a full SVD $X_c = U\Sigma V^{\top}$, the right singular vectors $\mathbf{v}_1, \dots, \mathbf{v}_p$ serve as principal directions, and the eigenvalues of $C$ are the numbers $\sigma_j^2/(N - 1)$ for $j = 1, \dots, \min(N, p)$, followed by $p - N$ zeros when $p > N$. For $k \le \min(N, p)$, the scores of the first $k$ components are $X_cV_k = U_k\Sigma_k$, not $U_k$ alone.

Definition For an integer $k$ with $1 \le k \le p$, keep the first $k$ principal directions as the columns of the $p \times k$ matrix $W_k = [\mathbf{w}_1 \ \cdots \ \mathbf{w}_k]$, which has orthonormal columns. The reconstruction of observation $i$ from its $k$ scores is

the orthogonal projection of $\mathbf{x}_i$ onto the affine subspace $\bar{\mathbf{x}} + \operatorname{span}\{\mathbf{w}_1, \dots, \mathbf{w}_k\}$. The retained variance is $\lambda_1 + \cdots + \lambda_k$, the total variance is $\operatorname{tr} C = \lambda_1 + \cdots + \lambda_p$, and when the total is positive the retained fraction is their quotient. The reconstruction error is the total squared distance

Each residual is perpendicular to the kept directions, so the total squared length $\lVert X_c \rVert_F^2 = (N - 1)\operatorname{tr} C$ splits into the retained part $(N - 1)(\lambda_1 + \cdots + \lambda_k)$ plus the reconstruction error. The matrix $X_cW_kW_k^{\top}$ is a truncated SVD of $X_c$ of rank at most $k$, so by the Eckart-Young theorem no affine subspace of dimension $k$ through $\bar{\mathbf{x}}$ gives a smaller reconstruction error. One that misses $\bar{\mathbf{x}}$ does no better, because sliding it parallel to itself until it contains $\bar{\mathbf{x}}$ never increases the total squared distance. Maximizing the retained variance and minimizing the reconstruction error select the same directions.

Definition Principal directions depend on two choices made before any eigenvalue is computed. The first is centering. Without it, the matrix $\tfrac1N X^{\top}X$ of second moments about the origin takes the place of $C$, and

so a mean far from the origin adds a large rank-one term in the direction of $\bar{\mathbf{x}}$, which can dominate the leading eigenvector. The second choice is the units. Measuring feature $j$ in different units multiplies its centered column by a positive factor $d_j$. With $D = \operatorname{diag}(d_1, \dots, d_p)$, the centered data become $X_cD$ and the covariance matrix becomes $DCD$, which in general has different eigenvectors from $C$. Scaling every feature by the same factor $d$ multiplies $C$ by $d^2$ and leaves the directions alone. Standardizing, which divides each centered feature by its sample standard deviation so that every variance becomes $1$, is one particular choice of $D$ and requires every feature variance to be positive. It answers a different question about the data, and it is not an automatic correction.