Neural Networks and Forward Propagation

Definition A feed-forward neural network maps an input $\mathbf{x}$ in $\mathbb{R}^p$ to an output through a fixed sequence of layers. Set $\mathbf{h}^{(0)} = \mathbf{x}$ and $d_0 = p$. For $l = 1, \dots, L - 1$, layer $l$ computes

where the weight matrix $W^{(l)}$ is $d_l \times d_{l-1}$, the bias $\mathbf{b}^{(l)}$ lies in $\mathbb{R}^{d_l}$, and the activation function $\phi$ acts on each entry of the preactivation $\mathbf{z}^{(l)}$. The output layer $L$ forms $\mathbf{z}^{(L)} = W^{(L)}\mathbf{h}^{(L-1)} + \mathbf{b}^{(L)}$ and applies an output function chosen for the task. Layers $1$ to $L - 1$ are the hidden layers, each entry of $\mathbf{h}^{(l)}$ is a unit, and $d_l$ is the width of layer $l$. Every unit receives every value of the previous layer, so these layers are called fully connected. The parameters $\boldsymbol{\theta}$ are all the weights and biases, so the architecture defines a family of functions $f_{\boldsymbol{\theta}}$, and training picks $\boldsymbol{\theta}$ by minimizing a loss on training data. A hidden layer is hidden because the data contain no target values for its units. The loss is computed from the output alone.

Definition An activation function is a fixed function $\phi$ from $\mathbb{R}$ to $\mathbb{R}$ that a layer applies to each entry of its preactivation vector. The rectified linear unit, ReLU, is

which equals $z$ for $z \ge 0$ and $0$ for $z < 0$. It is continuous and piecewise linear but not linear. Its derivative is $1$ for $z > 0$ and $0$ for $z < 0$, and at $z = 0$ it has no derivative, so software uses a convention there, usually $0$. Other common activations are the logistic sigmoid $\sigma(z) = 1/(1 + e^{-z})$ and $\tanh(z)$. Hidden layers add flexibility only through a nonlinear activation. With the identity in place of $\phi$, two layers compute

which is again an affine function of $\mathbf{x}$, and by induction any number of such layers computes a single affine map.

Definition Forward propagation, also called the forward pass, evaluates a network on a given input with fixed parameters. It computes each layer’s preactivation and activation in dependency order, from the input through the hidden layers to the output, using the layer equations above. It changes no parameters. Training code keeps the intermediate values $\mathbf{z}^{(l)}$ and $\mathbf{h}^{(l)}$, because the backward pass that computes gradients reuses them.

Definition The output layer maps the last hidden vector to the quantity the task needs. Its output function, the format of the labels and the training loss have to describe the same task.

• A real-valued target uses an identity output, the prediction $\hat{y} = z^{(L)}$, with a regression loss such as squared error.
• A binary label in $\{0, 1\}$ uses one score $z$, the modeled probability $P(Y = 1 \mid \mathbf{x}) = \sigma(z)$ and binary cross-entropy. The score is then called a logit.
• A task with $K$ mutually exclusive classes uses $K$ scores, the softmax of the score vector as the class probabilities and multiclass cross-entropy, with one class label per example.
• A task where several of $K$ labels can hold at once uses $K$ sigmoids, one binary cross-entropy for each label, and probabilities that need not add to $1$.

A number between $0$ and $1$ produced by a sigmoid or softmax is the model’s probability. Whether such numbers match observed frequencies on new data is calibration, a separate property that must be checked on held-out data.

Definition In machine learning code a tensor is a multidimensional array, and its shape lists the length of each axis. For one input written as a column, $\mathbf{x}$ is $p \times 1$, a first layer of width $d_1$ has $W^{(1)}$ of shape $d_1 \times p$, the vectors $\mathbf{z}^{(1)}$ and $\mathbf{h}^{(1)}$ are $d_1 \times 1$, and a scalar output with weight vector $\mathbf{v}$ of shape $d_1 \times 1$ gives $\mathbf{v}^{\top}\mathbf{h}^{(1)}$ of shape $1 \times 1$. For a batch of $n$ examples stored as the rows of an $n \times p$ matrix $X$, the same layer is

an $n \times d_1$ array whose row $i$ is the transposed hidden vector of example $i$. Here $\mathbf{1}$ is the $n \times 1$ vector of ones, so $\mathbf{1}\mathbf{b}^{(1)\top}$ copies the bias into every row, and libraries make this copy implicitly by broadcasting. Each layer’s input axis must have the length of the previous layer’s output axis, and which axis holds examples, features or classes is part of the specification.

Definition Let $\phi$ be a continuous function from $\mathbb{R}$ to $\mathbb{R}$ that is not a polynomial, such as ReLU, the sigmoid or $\tanh$. Let $K \subseteq \mathbb{R}^p$ be compact, that is, closed and bounded, and let $f$ be a continuous real-valued function on $K$. Then for every $\varepsilon > 0$ there are a width $N$, weight vectors $\mathbf{w}_j$ in $\mathbb{R}^p$ and numbers $b_j$, $a_j$ and $c$ such that the one-hidden-layer network

with the preactivations $u_j = \mathbf{w}_j^{\top}\mathbf{x} + b_j$, satisfies

If $\phi$ is a polynomial of degree $d$, every such $g$ is a polynomial of degree at most $d$, and such polynomials cannot approximate every continuous function on the cube $[0, 1]^p$, so the condition on $\phi$ cannot be dropped. A function with several outputs is approximated one output at a time. The theorem only asserts existence. It gives no practical bound on $N$ in general, no method that finds the parameters, no number of training examples, and no guarantee about accuracy on new data or outside $K$. It also does not say that a network of some fixed width represents every function exactly.