Unit 6 of 15

Generalization and Reliable Model Selection

  • 9 lessons
  • Videos 36 to 44

A useful model predicts well on cases it has not seen. This unit estimates that performance honestly, keeps tuning apart from the final evaluation and measures the uncertainty in the results.

Expected squared error splits into squared bias, variance and irreducible noise. Cross-validation refits every learned preprocessing step inside each fold, and nested evaluation keeps a hyperparameter search from inflating the reported score. Learning curves and error analysis suggest what to change next. The bootstrap measures the uncertainty of a metric or a paired comparison, split conformal prediction gives intervals with marginal coverage under exchangeability, and a finite-class bound shows what a guarantee about risk requires. Feature selection and a complete workflow end the unit.

Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.

Bias, Variance and Cross-Validation

  1. Bias, Variance, and Irreducible Error

    How expected squared error at a fixed input splits into squared bias, prediction variance and irreducible error, what training randomness the average covers, and why the bias-variance tradeoff is a tendency.

    Video 36

  2. Cross-Validation and Preprocessing Pipelines

    How k-fold cross-validation scores a whole training procedure, how stratified, grouped and temporal folds match the data, and how a preprocessing pipeline refits every learned step in each fold to prevent fold leakage.

    Video 37

Hyperparameter Search and Diagnostics

  1. Hyperparameter Search and Nested Evaluation

    How grid search and random search tune hyperparameters within a stated search budget, why the best validation score is optimistic, and how nested cross-validation with inner and outer folds evaluates the whole search.

    Video 38

  2. Learning Curves and Error Analysis

    How a learning curve tracks training and validation error as the training set size grows, and how error analysis, error slices and subgroup evaluation turn mistakes into hypotheses to test on protected data.

    Video 39

Uncertainty and Learning Guarantees

  1. Bootstrap Uncertainty and Model Comparisons

    How bootstrap resampling of held-out cases measures metric uncertainty, how to read a bootstrap distribution and a percentile interval, and why a paired comparison resamples the independent sampling unit.

    Video 40

  2. Prediction Intervals and Split Conformal Prediction

    How a prediction interval differs from a confidence interval, what aleatoric and epistemic uncertainty mean, and how split conformal prediction turns calibration conformity scores into marginal coverage under exchangeability.

    Video 41

  3. Generalization, Model Capacity, and Learning Guarantees

    How Hoeffding's inequality and the union bound give a generalization bound for a finite hypothesis class with bounded losses, what capacity measures in it, and why inductive bias and the no free lunch theorem matter.

    Video 42

Feature Selection and a Complete Workflow

  1. Feature Selection and Collinearity

    How collinearity and feature redundancy make linear coefficients unstable or nonunique, how to read a coefficient as a conditional statement, and why feature selection and post-selection inference must account for the search.

    Video 43

  2. A Complete Supervised Learning Workflow

    How a supervised learning workflow runs from a data contract through development data, a baseline comparison, tuning and refitting after selection to one final evaluation, and what reproducibility requires.

    Video 44