Unit 7 of 11

Least Squares

  • 5 lessons
  • Videos 33 to 37

When a system has no exact solution, least squares finds the inputs whose output comes closest. This unit derives it from projection and computes fits with normal equations and QR.

Fitting a line to three points that do not lie on one line is the opening example. The vertical residuals in its plot become one residual vector, which at the best fit is perpendicular to the column space in observation space. The unit builds design matrices, solves the normal equations, repeats the fit with QR, and separates the fitted values, which are always unique, from the coefficients, which may not be. Conditioning then measures how much a small change in the data can move a solution.

Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.

Least-Squares Fitting

  1. Least Squares and Orthogonal Projection

    What a least-squares solution is, how vertical residuals form one residual vector in observation space whose squared length is the sum of squared errors, and how projection gives the normal equations and a unique fitted vector.

    Video 33

  2. Least-Squares Models and Normal Equations

    How to build a design matrix for a model linear in its coefficients, solve the normal equations, compute fitted values and residuals, check orthogonality, and recognize nonunique coefficients.

    Video 34

  3. Least Squares with QR

    How a reduced QR factorization of a design matrix with independent columns reduces least squares to a triangular equation, what the orthonormal coordinates of the data are, and how the squared error splits.

    Video 35

Conditioning and Sensitivity

  1. Conditioning and Sensitivity

    How far a small change in the data can move the solution of a linear system, measured with relative error, the induced matrix norm and the condition number, and how conditioning differs from stability.

    Video 36

Projection and Least Squares Workshop

  1. Projection and Least Squares Workshop

    Mixed practice in choosing between an exact solve, a projection and a least-squares fit, with projection onto a plane, a line fit, proportional design columns, a sensitive measurement and residual checks.

    Video 37