Unit 4 of 15

Regression, Loss, and Regularization

  • 8 lessons
  • Videos 19 to 26

Regression predicts a number. This unit chooses a loss, fits linear models by least squares and gradient methods, adds features and penalties, and compares models with regression metrics.

A loss measures one error, empirical risk averages it over training data and population risk concerns new observations. Squared loss targets a conditional mean and absolute loss a conditional median. Least squares fits linear regression, and its fitted values are a projection, unique even when the coefficients are not. Stochastic gradient descent uses mini-batches. Polynomial features, interactions and ridge, lasso and elastic net penalties follow, with penalty strength chosen on validation data. Regression metrics and baselines, robust regression and quantile prediction end the unit.

Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.

Fitting Linear Regression Models

  1. Loss Functions and Empirical Risk

    How a loss function prices one prediction error, how squared and absolute error differ, and how empirical risk, population risk, the training objective and an evaluation metric fit together.

    Video 19

  2. Simple Linear Regression

    How simple linear regression fits a line to one feature by least squares, how the slope and intercept are derived and read, and what fitted values, residuals and extrapolation mean.

    Video 20

  3. Multiple Linear Regression and Least Squares

    How multiple linear regression fits several features by least squares, why the normal equations make the fitted values a projection, when full column rank makes the coefficients unique, and how QR solves the fit stably.

    Video 21

  4. Stochastic Gradient Descent and Mini-Batches

    How stochastic and mini-batch gradients estimate the full training gradient, and how epochs, learning-rate schedules, step sizes and feature scaling decide whether gradient descent converges.

    Video 22

Feature Engineering and Regularization

  1. Polynomial Features and Interactions

    How feature engineering adds polynomial features, indicator variables, interaction terms and spline basis functions to a linear model, why the model stays linear in parameters, and how to build them without leakage.

    Video 23

  2. Ridge, Lasso, and Elastic Net

    How ridge, lasso and elastic-net penalties regularize linear regression, how the regularization strength shrinks coefficients or sets them to zero, and how a regularization path and validation data choose it.

    Video 24

Regression Metrics and Loss Choice

  1. Regression Metrics and Baseline Comparisons

    How mean absolute error, mean squared error, root mean squared error, R-squared and mean absolute percentage error score held-out predictions, and why mean and median baselines are fitted on training data.

    Video 25

  2. Robust Regression and Quantile Prediction

    How the loss decides what a regression predicts, how robust regression with Huber loss limits the pull of large residuals, and how pinball loss, quantile regression and asymmetric costs target a conditional quantile.

    Video 26