Factor models and covariance estimation

Definition Let $R \in \mathbb{R}^d$ be one-period asset returns with finite second moments, mean $\mu$ and covariance $\Sigma$, under one fixed measure such as the physical measure $\mathbb{P}$. A linear factor model with $k$ factors writes

where $f \in \mathbb{R}^k$ is a centered factor vector with covariance $F$, $\varepsilon \in \mathbb{R}^d$ is a centered residual vector with covariance $D$, and $\operatorname{Cov}(f, \varepsilon) = 0$, the $k \times d$ zero matrix. The $d \times k$ matrix $B$ holds the factor loadings, and $B_{ij}$ is the sensitivity of asset $i$ to factor $j$. Expanding the covariance of the sum, the two cross terms vanish by assumption, so

If $F$ is invertible, then $\operatorname{Cov}(R, f) = BF$, so $B = \operatorname{Cov}(R, f) F^{-1}$ is the matrix of population regression coefficients of the returns on the factors, and with one factor the loading of asset $i$ is $\beta_i = \operatorname{Cov}(R_i, f)/\operatorname{Var}(f)$. Conversely, defining $B$ by this formula makes $\operatorname{Cov}(f, \varepsilon) = 0$ automatically. Loadings and factors are identified only together. For any invertible $k \times k$ matrix $H$, the loadings $BH^{-1}$ and the factors $Hf$, with covariance $HFH^{\top}$, give the same returns and the same $\Sigma$, so a normalization, such as $F = I$ or a factor that is a traded return with its own scale, has to be fixed before a loading has a size. For a portfolio $w$ the factor exposure is $B^{\top} w$, and

Definition In a factor model the residual covariance is $D = \operatorname{Cov}(\varepsilon)$, the part of $\Sigma$ that the factors do not explain. It is a covariance matrix, hence symmetric and positive semidefinite, so $\Sigma = BFB^{\top} + D$ is positive semidefinite, and it is positive definite whenever $D$ is. A strict factor model assumes that $D$ is diagonal, which means the residuals of different assets are uncorrelated. That is an additional assumption. Regression on the factors makes each residual uncorrelated with the factors, but it does not make the residuals uncorrelated with each other, and a common factor missing from $f$ appears as off-diagonal entries of $D$. A proposed nondiagonal $D$ must itself be positive semidefinite, which for two assets means nonnegative diagonal entries and $D_{12}^2 \le D_{11} D_{22}$.

Definition Given observations $r_1, \dots, r_n \in \mathbb{R}^d$ of the return vector, with $n \ge 2$ and sample mean $\bar{r} = \frac1n \sum_k r_k$, the sample covariance matrix is

It is an estimate computed from data and a different object from the population covariance $\Sigma$. For independent, identically distributed observations with finite second moments, $E[\hat{\Sigma}] = \Sigma$. It is positive semidefinite, because for every $w \in \mathbb{R}^d$ the projected deviations $y_k = w^{\top} (r_k - \bar{r})$ give

The centered vectors $r_k - \bar{r}$ sum to zero, so they span a subspace of dimension at most $n - 1$, and $\operatorname{rank} \hat{\Sigma} \le n - 1$. When $d \ge n$ the matrix is singular, and any nonzero $w$ orthogonal to every centered observation is a position with zero sample variance. The bound is only an upper bound, and data that vary in fewer directions give a smaller rank. Even when $\hat{\Sigma}$ is invertible, its extreme eigenvalues are biased outward. The largest eigenvalue of a symmetric matrix is a convex function of the matrix and the smallest is concave, so Jensen’s inequality and $E[\hat{\Sigma}] = \Sigma$ give

Definition Let $\hat{\Sigma}$ be a sample covariance and $T$ a symmetric positive semidefinite target, for example the diagonal of $\hat{\Sigma}$, a multiple of the identity or a factor-model covariance. The shrinkage estimator with intensity $\alpha \in [0, 1]$ is

Its quadratic form $w^{\top} \hat{\Sigma}_{\alpha} w = (1 - \alpha) w^{\top} \hat{\Sigma} w + \alpha\, w^{\top} T w$ is a convex combination of two nonnegative numbers, so $\hat{\Sigma}_{\alpha}$ is positive semidefinite. If $T$ is positive definite and $\alpha > 0$, then $w^{\top} \hat{\Sigma}_{\alpha} w \ge \alpha\, w^{\top} T w > 0$ for every $w \ne 0$, so $\hat{\Sigma}_{\alpha}$ is positive definite and invertible however singular $\hat{\Sigma}$ is, with smallest eigenvalue at least $\alpha \lambda_{\min}(T)$. With the diagonal target the variances stay as they are and every covariance is multiplied by $1 - \alpha$. Shrinkage trades estimation variance for bias. For a target $T$ that does not depend on the data and an unbiased $\hat{\Sigma}$ with finite expected squared error, the cross term has mean zero and

where $\lVert \cdot \rVert_F$ is the Frobenius norm. Its derivative at $\alpha = 0$ is $-2E \lVert \hat{\Sigma} - \Sigma \rVert_F^2$, so a small positive intensity lowers the expected error whenever $\hat{\Sigma}$ has any. The best intensity depends on the unknown $\Sigma$. Estimating it from the data, for example by the Ledoit-Wolf method with its own target and assumptions, is a statistical result that is only referred to here, and an intensity supplied for an illustration is not an optimal estimate.

Definition For a real symmetric $d \times d$ matrix $\Sigma$ and a vector $v \ne 0$, the Rayleigh quotient is

which does not change when $v$ is multiplied by a nonzero number. By the spectral theorem, $\Sigma$ has an orthonormal basis of eigenvectors $e_1, \dots, e_d$ with eigenvalues $\lambda_1 \ge \dots \ge \lambda_d$. Writing $v = \sum_j c_j e_j$ gives

a weighted average of the eigenvalues with weights $c_j^2 / \sum_i c_i^2$. Hence $\lambda_d \le Q_{\Sigma}(v) \le \lambda_1$, with the maximum attained at $v = e_1$ and the minimum at $v = e_d$. In particular the maximum of $v^{\top} \Sigma v$ subject to $v^{\top} v = 1$ is $\lambda_{\max}(\Sigma) = \lambda_1$. The Lagrange condition of that problem, $\Sigma v = \ell v$, holds at every unit eigenvector, so it only lists the candidates, and the weighted average shows which candidate is the maximum. Restricting $v$ to be orthogonal to $e_1, \dots, e_{j-1}$ removes their coefficients, and the same argument gives the maximum $\lambda_j$, attained at $e_j$.

Definition Let $X = R - \mu$ be centered returns whose covariance $\Sigma$ has orthonormal eigenvectors $e_1, \dots, e_d$ and eigenvalues $\lambda_1 \ge \dots \ge \lambda_d \ge 0$. The $j$th principal component is the scalar $e_j^{\top} X$, with loading vector $e_j$. By the Rayleigh quotient card, $e_1$ maximizes $\operatorname{Var}(v^{\top} X) = v^{\top} \Sigma v$ over unit vectors, and each $e_j$ maximizes it over unit vectors orthogonal to $e_1, \dots, e_{j-1}$. The components are uncorrelated, since $\operatorname{Cov}(e_i^{\top} X, e_j^{\top} X) = e_i^{\top} \Sigma e_j$ is $\lambda_j$ for $i = j$ and $0$ otherwise, and their variances add up to the total variance, $\sum_j \lambda_j = \operatorname{tr} \Sigma = \sum_i \operatorname{Var}(R_i)$. Keeping the first $k$ components splits

which is a factor model with uncorrelated factors $e_j^{\top} X$ of variances $\lambda_j$, residuals uncorrelated with them, and a residual covariance equal to the second sum, which is not diagonal in general. When $\operatorname{tr} \Sigma > 0$, the explained-variance ratio of the first $k$ components is $(\lambda_1 + \dots + \lambda_k)/\operatorname{tr} \Sigma$. Stated without proof, the first sum is a best approximation of $\Sigma$ by a matrix of rank at most $k$ in the Frobenius norm, which is the Eckart-Young theorem. Applied to $\hat{\Sigma}$, the same construction gives sample principal components. The sign of each eigenvector is arbitrary, and when eigenvalues are equal or close the directions are not determined or are unstable under small changes in the data. Principal components of the correlation matrix solve a different problem from those of the covariance matrix, because standardizing each return to unit variance changes the quadratic form.