Derivatives, Partial Derivatives, and the Chain Rule, machine learning lesson

A one-feature model predicts 7 for a training example whose target is 8. If the weight is raised slightly, does the squared error go up or down, and how fast? Would raising the intercept instead have the same effect? Derivatives answer such questions for small changes. This lesson defines the derivative and the partial derivative, collects partial derivatives into the gradient, uses the chain rule to differentiate a squared prediction error through every intermediate step, and checks the result with a directional derivative and a finite difference.

Key terms

Derivative and differentiability

Definition Let $f$ be a real function defined on an open interval containing $t$ . The derivative of $f$ at $t$ is

f'(t) = \lim_{h \to 0} \frac{f(t + h) - f(t)}{h}

when this limit exists as a finite number, with $h$ approaching $0$ through nonzero values from both sides. In that case $f$ is called differentiable at $t$ . Equivalently, there is a number $f'(t)$ with

\begin{aligned} f(t + h) &= f(t) + f'(t)\,h \\[0.4em] &\quad + e(h), \end{aligned}

where $e(h)/h \to 0$ as $h \to 0$ . So near $t$ the function is approximated by a line with slope $f'(t)$ . The derivative has the unit of $f$ divided by the unit of $t$ . If the difference quotient has different one-sided limits at $t$ , then $f$ is not differentiable there, and a method that still needs a slope at $t$ must state a convention. For a convex function such as $\lvert r \rvert$ at $r = 0$ , a standard choice is a subgradient, any number $g$ with $\lvert s \rvert \ge g\,s$ for every real $s$ , which means any $g$ in $[-1, 1]$ .

In plain words The difference quotient is the average rate of change over a step of size $h$ , and the derivative is the value it settles on as the step shrinks. Being differentiable means that close enough to $t$ the graph looks like a straight line, so a small step $h$ changes the output by about $f'(t)\,h$ . It says nothing about large steps.

Example For $f(t) = t^2$ and any $h \ne 0$ ,

\begin{aligned} \frac{(t + h)^2 - t^2}{h} &= \frac{2th + h^2}{h} \\[0.4em] &= 2t + h , \end{aligned}

which tends to $2t$ as $h \to 0$ . So $f'(3) = 6$ , and the limit argument is a proof for every $t$ , not a pattern read from a few numbers. The absolute value $\lvert r \rvert$ has the difference quotient $\lvert h \rvert / h$ at $r = 0$ , which equals $-1$ for $h < 0$ and $1$ for $h > 0$ . The one-sided limits disagree, so $\lvert r \rvert$ has no derivative at $0$ . Software often uses the subgradient $0$ there, which is a convention and not the value of a derivative.

Partial derivative

Definition Let $f(w, b)$ be defined near the point $(w_0, b_0)$ . The partial derivative of $f$ with respect to $w$ at that point is the ordinary derivative at $w_0$ of the one-variable function $g(w) = f(w, b_0)$ , in which $b$ is held at $b_0$ ,

\frac{\partial f}{\partial w}(w_0, b_0) = g'(w_0) ,

so the steps $h$ in the limit defining $g'(w_0)$ change $w$ alone. The partial derivative with respect to $b$ holds $w$ fixed in the same way. With more parameters, one coordinate moves and all the others stay fixed. In a training loss the data are held fixed as well.

In plain words Slice the loss surface along one coordinate axis and take the slope of the slice. Each partial derivative answers what happens when only one parameter moves while everything else stays put.

Example Fix one training example with feature $x = 3$ and target $y = 8$ . The model predicts $\hat{y} = wx + b$ , the residual on this page is prediction minus target, $r = \hat{y} - y$ , and the loss is $L(w, b) = \tfrac12 r^2$ . The factor $\tfrac12$ only simplifies the derivatives. At $w = 2$ and $b = 1$ the prediction is 7, the residual is $-1$ and the loss is $0.5$ . Holding $b = 1$ gives the slice $L(w, 1) = \tfrac12 (3w - 7)^2$ , whose derivative $3(3w - 7)$ equals $-3$ at $w = 2$ . Holding $w = 2$ gives the slice $L(2, b) = \tfrac12 (b - 2)^2$ , whose derivative $b - 2$ equals $-1$ at $b = 1$ . So at this point $\partial L / \partial w = -3$ and $\partial L / \partial b = -1$ .

Gradient

Definition For a real function $f$ of a parameter vector $\boldsymbol{\theta} = (\theta_1, \dots, \theta_k)^{\top}$ , the gradient $\nabla f(\boldsymbol{\theta})$ is the vector of partial derivatives $\partial f / \partial \theta_j$ in the same order as the parameters. On this page it is a column with the same shape as $\boldsymbol{\theta}$ , so for $\boldsymbol{\theta} = (w, b)^{\top}$

\nabla L = \begin{bmatrix} \partial L / \partial w \\ \partial L / \partial b \end{bmatrix} .

The function $f$ is differentiable at $\boldsymbol{\theta}$ if its partial derivatives exist there and

\begin{aligned} & f(\boldsymbol{\theta} + \boldsymbol{\Delta}) - f(\boldsymbol{\theta}) \\[0.4em] &\quad = \nabla f(\boldsymbol{\theta})^{\top}\boldsymbol{\Delta} + e(\boldsymbol{\Delta}) \end{aligned}

with an error term satisfying $e(\boldsymbol{\Delta}) / \lVert \boldsymbol{\Delta} \rVert_2 \to 0$ as $\boldsymbol{\Delta} \to \mathbf{0}$ . Having all partial derivatives is not enough. The function equal to $wb / (w^2 + b^2)$ away from the origin, with $f(0, 0) = 0$ , has both partial derivatives equal to $0$ at the origin, yet it equals $\tfrac12$ at every point $(t, t)$ with $t \ne 0$ , so it is not even continuous there. If the partial derivatives exist and are continuous on an open set containing $\boldsymbol{\theta}$ , then $f$ is differentiable at $\boldsymbol{\theta}$ . Polynomials, such as the loss on this page, are differentiable everywhere.

In plain words The gradient packs the one-at-a-time slopes into one vector. For a differentiable function, a small step that moves several parameters at once changes the value by about the dot product of the gradient with the step. The approximation is local, and curvature makes it less accurate for larger steps.

Example At $w = 2$ and $b = 1$ the gradient is $\nabla L = (-3, -1)^{\top}$ . For the step $\boldsymbol{\Delta} = (0.01, 0.01)^{\top}$ it predicts a change of $-3(0.01) - 1(0.01) = -0.04$ . The actual new prediction is $2.01 \cdot 3 + 1.01 = 7.04$ , the residual is $-0.96$ and the loss is $0.4608$ , a change of $-0.0392$ from $0.5$ . The gap of $0.0008$ is the curvature term $\tfrac12 (0.04)^2$ , which shrinks like the square of the step.

Chain rule

Definition If $g$ is differentiable at $t$ and $f$ is differentiable at $g(t)$ , then the composition $f \circ g$ is differentiable at $t$ and

(f \circ g)'(t) = f'(g(t))\, g'(t) .

For several variables, suppose intermediate quantities $u_1, \dots, u_m$ are differentiable functions of $\boldsymbol{\theta}$ and $L$ is a differentiable function of $u_1, \dots, u_m$ . Then

\frac{\partial L}{\partial \theta_j} = \sum_{k=1}^{m} \frac{\partial L}{\partial u_k}\, \frac{\partial u_k}{\partial \theta_j} .

A computational graph draws each quantity as a node and each direct dependency as an edge labeled with its local derivative. The derivative of the output with respect to a parameter is the sum, over all paths from the parameter to the output, of the product of the local derivatives along each path.

In plain words Rates multiply along a chain of dependencies, and contributions from separate routes add. Dropping one factor, such as the derivative of the prediction with respect to the weight, gives a wrong gradient even when every other step is right.

Example The graph for one example multiplies $w$ by $x$ , adds $b$ to give $\hat{y}$ , subtracts $y$ to give $r$ , and squares and halves $r$ to give $L$ . The local derivatives are $\partial L / \partial r = r$ , $\partial r / \partial \hat{y} = 1$ , $\partial \hat{y} / \partial w = x$ and $\partial \hat{y} / \partial b = 1$ , so

\begin{aligned} \frac{\partial L}{\partial w} &= \frac{\partial L}{\partial r}\,\frac{\partial r}{\partial \hat{y}}\,\frac{\partial \hat{y}}{\partial w} = r x, \\[0.4em] \frac{\partial L}{\partial b} &= \frac{\partial L}{\partial r}\,\frac{\partial r}{\partial \hat{y}}\,\frac{\partial \hat{y}}{\partial b} = r . \end{aligned}

With $x = 3$ and $r = -1$ this gives $(-3, -1)^{\top}$ , the gradient found from the slices. Expanding the loss as $\tfrac12 (3w + b - 8)^2$ gives the same result, with the inner factor 3 for $w$ and 1 for $b$ . Forgetting the inner factor would make the two derivatives equal, which is wrong. With two training examples and total loss $\tfrac12 (r_1^2 + r_2^2)$ , the weight reaches the loss through both predictions, and the two paths add to $\partial L / \partial w = r_1 x_1 + r_2 x_2$ .

Directional derivative

Definition The directional derivative of $f$ at $\boldsymbol{\theta}$ along a vector $\mathbf{v}$ is the rate of change of $f$ along the line $\boldsymbol{\theta} + t\mathbf{v}$ at $t = 0$ ,

D_{\mathbf{v}} f(\boldsymbol{\theta}) = \lim_{t \to 0} \frac{\varphi(t) - \varphi(0)}{t},

where $\varphi(t) = f(\boldsymbol{\theta} + t\mathbf{v})$ . If $f$ is differentiable at $\boldsymbol{\theta}$ , the limit exists for every $\mathbf{v}$ and

D_{\mathbf{v}} f(\boldsymbol{\theta}) = \nabla f(\boldsymbol{\theta})^{\top}\mathbf{v} .

For a unit vector, with $\lVert \mathbf{v} \rVert_2 = 1$ , this is the rate of change per unit of Euclidean distance in parameter space. For other vectors it is the rate per unit of $t$ . The partial derivative with respect to $\theta_j$ is the directional derivative along the coordinate vector $\mathbf{e}_j$ . When $\nabla f(\boldsymbol{\theta}) \ne \mathbf{0}$ , the Cauchy-Schwarz inequality shows that among unit vectors the rate is largest for $\mathbf{v} = \nabla f / \lVert \nabla f \rVert_2$ and smallest, equal to $-\lVert \nabla f \rVert_2$ , for the opposite vector. These steepest directions depend on the chosen coordinates and their scales.

In plain words Pick a straight path through parameter space and ask how fast the function changes as you set off along it. A step that moves several parameters at once mixes their partial derivatives, so it measures something different from any single one of them.

Example At $w = 2$ and $b = 1$ , with gradient $(-3, -1)^{\top}$ , the unit vector $\mathbf{v} = (3/5, 4/5)^{\top}$ gives

\begin{aligned} D_{\mathbf{v}} L &= -3 \cdot \tfrac35 - 1 \cdot \tfrac45 \\[0.4em] &= -\tfrac{13}{5} = -2.6 . \end{aligned}

The direction $(1, 1)^{\top}$ , which raises both parameters together, has rate $-4$ per unit of $t$ , or $-4/\sqrt{2} \approx -2.83$ per unit distance. Neither number equals the partial derivative $-3$ , which is the rate along $(1, 0)^{\top}$ . No unit direction decreases the loss faster than at the rate $-\sqrt{10} \approx -3.16$ , reached along $(3, 1)^{\top}/\sqrt{10}$ , the direction of $-\nabla L$ .

Finite-difference approximation

Definition For a step $h > 0$ , the forward difference and the central difference of $f$ at $t$ are

\begin{gathered} \frac{f(t + h) - f(t)}{h}, \\[0.4em] \frac{f(t + h) - f(t - h)}{2h} . \end{gathered}

For a partial derivative, only the chosen coordinate is stepped and the others are held fixed. If $f$ has a continuous second derivative on $[t, t + h]$ , the forward difference differs from $f'(t)$ by $\tfrac12 f''(\xi)\,h$ for some $\xi$ in that interval. If $f$ has a continuous third derivative on $[t - h, t + h]$ , the central difference differs from $f'(t)$ by $\tfrac16 f'''(\xi)\,h^2$ for some $\xi$ in that interval. In floating-point arithmetic a very small $h$ makes the two function values nearly equal, and their difference loses significant digits to rounding. Agreement between a finite difference and a formula is evidence that the formula or code is right at the tested points. It is not a proof, and it cannot show that a function is differentiable.

In plain words Nudge one parameter, measure how much the loss moves and divide by the nudge. It needs no derivative formula, which makes it a good independent test of a hand-derived or coded gradient, but it costs extra loss evaluations for every parameter. Too large a step picks up curvature and too small a step loses digits to rounding, so try a few step sizes and compare relative as well as absolute differences.

Example Hold $b = 1$ and step $w$ around $2$ with $h = 0.0001$ . The loss values are $L(2.0001, 1) = 0.499700045$ and $L(1.9999, 1) = 0.500300045$ , so the central difference is $-0.0006 / 0.0002 = -3$ , exact up to rounding because the loss is quadratic in $w$ . The forward difference is $(0.499700045 - 0.5) / 0.0001 = -2.99955$ . Its error $0.00045$ is exactly $\tfrac12 \cdot 9 \cdot 0.0001$ , since the second derivative in $w$ is $x^2 = 9$ . With $h = 10^{-12}$ rounding dominates instead, and in double-precision arithmetic both differences are off by about $0.0003$ .

Check yourself

Take a new example with $x = 2$ and $y = 5$ , the same loss $L = \tfrac12 r^2$ with $r = \hat{y} - y$ , and the parameters $w = 1$ and $b = 1$ . Find the prediction, the residual, the loss and the gradient, naming each chain-rule factor. Then check $\partial L / \partial w$ with a forward difference with step $h = 0.001$ , holding $b$ fixed.

Show the answer

The prediction is $\hat{y} = 1 \cdot 2 + 1 = 3$ , the residual is $r = 3 - 5 = -2$ and the loss is $\tfrac12 \cdot 4 = 2$ . The factors are $\partial L / \partial r = r = -2$ , $\partial r / \partial \hat{y} = 1$ , $\partial \hat{y} / \partial w = x = 2$ and $\partial \hat{y} / \partial b = 1$ . So $\partial L / \partial w = (-2)(1)(2) = -4$ and $\partial L / \partial b = (-2)(1)(1) = -2$ , and the gradient in the order $(w, b)$ is $(-4, -2)^{\top}$ .

At $w = 1.001$ the loss is $\tfrac12 (-1.998)^2 = 1.996002$ , so the forward difference is $(1.996002 - 2) / 0.001 = -3.998$ . It differs from $-4$ by $0.002$ , which is the curvature term $\tfrac12 x^2 h$ , and a central difference with the same step gives $-4$ up to rounding. The agreement supports the derivation at this point but does not prove it.

For the same example, find the rate of change of the loss along the path that raises $w$ by $t$ and lowers $b$ by $t$ . Is it the partial derivative with respect to $w$ ?

Show the answer

The direction is $\mathbf{v} = (1, -1)^{\top}$ , and the directional derivative is $\nabla L^{\top}\mathbf{v} = -4 \cdot 1 + (-2)(-1) = -2$ per unit of $t$ . A direct calculation agrees. Along the path the prediction is $2(1 + t) + (1 - t) = 3 + t$ , so the residual is $t - 2$ , the loss is $\tfrac12 (t - 2)^2$ and its derivative at $t = 0$ is $-2$ .

It is not the partial derivative with respect to $w$ , which is $-4$ and moves only $w$ . On this path the drop in $b$ cancels part of the effect of raising $w$ , so the prediction rises by $t$ instead of $2t$ . Because $\mathbf{v}$ has length $\sqrt{2}$ , the rate per unit distance is $-2/\sqrt{2} = -\sqrt{2} \approx -1.41$ .

Someone checks the derivative of $f(r) = \lvert r \rvert$ at $r = 0$ with a central difference and gets $0$ for every step $h$ . They conclude that $f'(0) = 0$ . What is wrong?

Show the answer

The central difference is $(\lvert h \rvert - \lvert -h \rvert) / (2h) = 0$ for every $h > 0$ , but the derivative does not exist at $0$ . The forward difference $\lvert h \rvert / h = 1$ and the backward difference $(0 - \lvert -h \rvert) / h = -1$ show the two one-sided limits, and they disagree. The central difference is the average of these two, so it returns a tidy number at a corner where no derivative exists.

A finite difference cannot establish differentiability. If an algorithm needs a value at the corner, it has to state a convention, such as a subgradient in $[-1, 1]$ . The value $0$ is one valid subgradient, not the derivative.

Derivatives, Partial Derivatives, and the Chain Rule

Key terms

Derivative and differentiability

Partial derivative

Gradient

Chain rule

Directional derivative

Finite-difference approximation

Check yourself

How this lesson connects

Builds on

Used later in