Unit 7 of 15

Margins, Trees, and Ensembles

  • 8 lessons
  • Videos 45 to 52

Separating hyperplanes, decision trees and ensembles of models are three classic families of learning methods. This unit builds each one and compares them under the same evaluation plan.

The perceptron corrects a linear boundary after each mistake and is guaranteed to converge for separable classes. Support vector machines maximize the margin and allow violations through the hinge loss, and kernels give nonlinear boundaries through feature-space inner products. Classification trees split by Gini impurity or entropy, and regression trees predict leaf means and are pruned on validation data. Bagging and random forests average trees grown on bootstrap samples, stagewise additive trees fit negative gradients, AdaBoost reweights mistakes and stacking uses out-of-fold predictions.

Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.

Margins and Kernels

  1. Perceptrons and Separating Hyperplanes

    How a linear classifier splits feature space with a hyperplane, how the perceptron update rule corrects one mistake at a time, and when linear separability guarantees that the perceptron stops.

    Video 45

  2. Support Vector Machines and Soft Margins

    How functional and geometric margins differ, how a support vector machine maximizes the margin, how hinge loss and the penalty C define a soft margin, and which training points are support vectors.

    Video 46

  3. Kernel Methods

    How a feature map makes a curved boundary linear, how kernel functions and Gram matrices supply feature-space inner products, which similarities are valid kernels, and how polynomial, RBF and kernel SVM models work.

    Video 47

Decision Trees

  1. Classification Trees

    How a classification tree splits feature space with yes or no questions, how Gini impurity, entropy, information gain and weighted child impurity rank splits, and what a leaf probability estimates.

    Video 48

  2. Regression Trees and Pruning

    How a regression tree predicts leaf means, how a squared-error split is chosen, how tree depth and minimum leaf size limit flexibility, and how cost-complexity pruning and validation choose the size of a tree.

    Video 49

Ensemble Methods

  1. Bagging and Random Forests

    How bagging averages trees fitted to bootstrap samples, how random forests add random feature subsets, why correlation between trees limits ensemble variance, and when out-of-bag error is honest evidence.

    Video 50

  2. Boosting and Gradient-Boosted Trees

    How boosting methods build an additive model stage by stage, why gradient boosting fits weak learners to negative gradients with residual fitting as the squared-error case, and how shrinkage, stage count and AdaBoost work.

    Video 51

  3. Stacking and Ensemble Comparison

    How to compare models on one evaluation plan, when averaging models helps, how ensemble selection works, and how stacking fits a meta-model on out-of-fold predictions without leaking training targets.

    Video 52