A Complete Neural Classification Example

Example The same-sign rule says $y = 1$ exactly when $x_1 x_2 \ge 0$. Four ReLU hidden units represent it exactly. With $\operatorname{ReLU}(u) = \max(u, 0)$, take the logit

Since $\operatorname{ReLU}(u) + \operatorname{ReLU}(-u) = \lvert u \rvert$, the logit equals $\lvert x_1 + x_2 \rvert - \lvert x_1 - x_2 \rvert$. It is nonnegative exactly when $(x_1 + x_2)^2 \ge (x_1 - x_2)^2$, that is, when $4x_1 x_2 \ge 0$. At the point $(0.5, 0.2)$ the logit is $0.7 - 0.3 = 0.4$, a positive prediction, and at $(0.5, -0.2)$ it is $0.3 - 0.7 = -0.4$. The tanh networks trained in this lesson have to find a representation from data, and the ones found only approximate the rule.

Example With $p = 2$ the count is $4h + 1$, so the widths $4$ and $12$ used below give $17$ and $49$ parameters, against $3$ for logistic regression on the same two features. For a batch of $16$ examples and width $12$, the arrays $X$, $W$, $H$ and $\mathbf{z}$ have shapes $16 \times 2$, $2 \times 12$, $16 \times 12$ and $16$. In NumPy the forward pass, the loss and the gradients by backpropagation take a few lines. The loss uses the form $\ln(1 + e^{z}) - yz$ through np.logaddexp, which never takes the logarithm of a probability that has rounded to $0$ or $1$.

import numpy as np

def forward(X, W, b, v, c):
    H = np.tanh(X @ W + b)                  # hidden array, shape (m, h)
    return H, H @ v + c                     # logits, shape (m,)

def mean_bce(y, z):
    return np.mean(np.logaddexp(0.0, z) - y * z)

def gradients(X, y, W, b, v, c):
    H, z = forward(X, W, b, v, c)
    prob = np.exp(-np.logaddexp(0.0, -z))   # sigmoid(z), computed stably
    dz = (prob - y) / len(y)                # derivative of the mean loss in z
    dA = np.outer(dz, v) * (1.0 - H**2)     # back through tanh
    return X.T @ dA, dA.sum(axis=0), H.T @ dz, dz.sum()

Example On the constructed data of this lesson, the baseline’s best validation checkpoint comes at epoch $2$, with validation loss $0.6912$, barely below the value $\ln 2 \approx 0.6931$ of a constant score $0.5$. All its logits are negative, so it predicts the negative class for every training, validation and test case. Since $50$ of the $120$ training cases are positive, it makes the same decisions as the majority-class rule. On the $80$ test cases it is right for the $46$ negatives, an accuracy of $0.575$, its recall is $0$, and its precision is undefined because it makes no positive predictions. A designed feature would change the picture. With the inputs $x_1$, $x_2$ and $x_1 x_2$, the logit $w_3 x_1 x_2$ with any $w_3 > 0$ is nonnegative exactly when $x_1 x_2 \ge 0$, so a logistic regression with $4$ parameters can represent the rule exactly. In real data such a feature is rarely known in advance, and trying it adds a new candidate that has to go through the same validation step.

Example The three sets are drawn with separate fixed seeds from the uniform distribution on $[-1, 1]^2$, with $120$ training, $60$ validation and $80$ test cases and the label $1$ exactly when $x_1 x_2 \ge 0$. Both features already share one dimensionless range, so no scaling is fitted. Had scaling been needed, its means and standard deviations would come from the training set and be applied unchanged to the other two. Every candidate is trained on mean cross-entropy with learning rate $0.1$, shuffled mini-batches of at most $16$ cases and a budget of $500$ epochs, and keeps its own best validation checkpoint. The record kept with the code lists the seeds, the data rule, the absence of fitted preprocessing, the candidates, the selection metric and the threshold.

The width-12 network has the lowest validation loss and is selected. The two networks tie on accuracy, and the loss separates them because it also measures how confident each correct and incorrect score is. With the parameters frozen and the threshold kept at $0.5$, the test set is opened once. The selected network has $\text{TP} = 31$, $\text{FP} = 0$, $\text{FN} = 3$ and $\text{TN} = 46$, so its accuracy is $77/80 = 0.9625$, its recall is $31/34 \approx 0.912$, its precision is $31/31 = 1$ and its test cross-entropy is about $0.1116$. Having no false positives among $46$ negatives does not show that the false positive rate is zero. If the test negatives are independent draws, an exact one-sided $95$ percent upper confidence bound for that rate is $1 - 0.05^{1/46} \approx 0.063$. The three missed positives lie at about $(0.096, 0.710)$, $(0.650, 0.024)$ and $(0.154, 0.935)$, each within $0.16$ of an axis, with scores near $0.42$, and all seven validation errors lie within $0.08$ of an axis. The errors sit close to the true boundary, where a slightly misplaced learned boundary produces mistakes. Inspecting them explains the result without changing the model. The claim also depends on the labeling rule. If the same test points were labeled by the reversed rule, positive for opposite signs, the frozen network’s accuracy would fall to $3/80 = 0.0375$.

Example The training, validation and test data of this lesson come from the seeds $661$, $662$ and $663$, and seed $7$ sets the initial weights and the mini-batch order. The network weights start as normal draws with mean $0$ and standard deviation $1/\sqrt{\text{fan-in}}$, which is $1/\sqrt{2}$ for $W$ and $1/\sqrt{h}$ for $\mathbf{v}$, the biases start at $0$, and the baseline starts from all zeros. In an additional check on the same data, the training seeds $1$ to $10$, which include seed $7$, give best validation losses from $0.2226$ to $0.3682$ for the width-4 network, with mean about $0.2595$, and from $0.2157$ to $0.2501$ for the width-12 network, with mean about $0.2360$. With seed $1$ the width-4 network scores $0.2226$ and the width-12 network $0.2501$, the reverse of the seed-7 ranking in the table above. The average over the ten seeds favors width $12$ on this construction, but the ranges overlap, so a choice between the widths should rest on several seeds and not on one.