Unit 7 of 10

Singular Value Decomposition

  • 7 lessons
  • Videos 31 to 37

This unit proves the singular value decomposition for every rectangular matrix and builds the pseudoinverse, best low-rank approximation and the polar decomposition on it.

The proof takes singular vectors from the spectral theorem and keeps track of full, thin and compact forms and of which factors are unique. The pseudoinverse gives the minimum-norm least-squares solution and four orthogonal projectors. The Eckart-Young theorem shows that truncating the decomposition gives a best approximation of at most a given rank in both the spectral and Frobenius norms, principal component analysis of centered data follows from it, and positive square roots lead to the polar decomposition.

Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.

The Decomposition and the Pseudoinverse

  1. The Singular Value Decomposition

    Why every real or complex matrix has a singular value decomposition, built from the Gram operator, with singular values, left and right singular vectors, thin and compact forms, and the choices that remain.

    Video 31

  2. The Pseudoinverse and Four Projectors

    How the pseudoinverse built from the SVD gives every least-squares solution and the minimum-norm one, and how four orthogonal projectors split the input and output spaces of a matrix.

    Video 32

  3. The Moore-Penrose Conditions

    How four matrix equations, the Moore-Penrose conditions, characterize the pseudoinverse, why Hermitian idempotents make it unique, and how it compares with weaker generalized inverses of singular matrices.

    Video 33

Optimal Approximation and Principal Components

  1. The Eckart-Young Theorem

    The spectral and Frobenius norms as error measures, the truncated SVD, the Eckart-Young theorem on best approximation of rank at most k, and when that best approximation is unique.

    Video 34

  2. Principal Component Analysis

    Centering a data matrix, the sample covariance matrix, principal directions from the SVD, why maximum projected variance and minimum reconstruction error agree, and how centering and units change the answer.

    Video 35

Square Roots and Polar Factors

  1. Positive Square Roots of Matrices

    The unique positive semidefinite square root of a Hermitian positive semidefinite matrix, its construction from the spectral theorem, other square roots of a matrix, and how it differs from a Cholesky factor.

    Video 36

  2. Polar Decomposition

    The polar decomposition of a real or complex matrix into a partial isometry times its absolute value, when that factor is an isometry or unitary, unitary extensions, and the polar form with the positive factor on the left.

    Video 37