Unit 9 of 15

Neural Network Foundations

  • 5 lessons
  • Videos 62 to 66

A neural network composes linear maps and nonlinear activations. This unit computes a forward pass, derives the gradients by backpropagation and trains and evaluates a small network.

Hidden layers with an activation such as ReLU give nonlinear decision regions. The universal approximation theorem says that wide enough networks can approximate any continuous function on a closed bounded region, which says nothing about training or generalization. Backpropagation applies the chain rule through a computational graph, and automatic differentiation does it in code. Training uses mini-batches, initialization and optimizers such as Adam, and weight decay, dropout and early stopping are compared on validation data. A complete example compares a network with a simpler baseline.

Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.

Forward and Backward Computation

  1. Neural Networks and Forward Propagation

    How a feed-forward neural network computes its output by forward propagation through hidden layers with ReLU activations, how tensor shapes and the output layer fit the task, and what universal approximation promises.

    Video 62

  2. Backpropagation and Automatic Differentiation

    How a computational graph and the multivariable chain rule give every parameter derivative of a network, how backpropagation and reverse-mode automatic differentiation compute them, and how a gradient check tests the result.

    Video 63

Training and Evaluating Networks

  1. Training Neural Networks

    How a training loop with mini-batches fits a neural network, why weight initialization and vanishing or exploding gradients matter, and how momentum, the Adam optimizer and batch normalization work in training and evaluation.

    Video 64

  2. Neural Regularization and Early Stopping

    How weight decay, decoupled weight decay and dropout regularize a neural network, why validation runs in evaluation mode, and how early stopping selects a checkpoint by validation loss.

    Video 65

  3. A Complete Neural Classification Example

    How a small neural network classifier is trained and selected against a logistic regression baseline, with its architecture, data roles, decision threshold and random seeds recorded.

    Video 66