Inner Products and Cauchy-Schwarz, linear algebra I lesson

The vectors $(1, 1)$ and $(1, -1)$ have dot product $0$ , so in the usual geometry they are perpendicular. Now measure differently and let the first coordinate count four times as much as the second in every squared length. The arrows have not moved, yet under the new rule they are no longer perpendicular. This lesson pins down which rules for measuring are allowed, called inner products, and proves from those rules alone the Cauchy-Schwarz inequality and the triangle inequality. As before, a tuple such as $(1, 1)$ stands for a column vector.

Key terms

Inner product

Definition Let $V$ be a real vector space. A real inner product on $V$ assigns to each pair of vectors $\mathbf{u}, \mathbf{v}$ in $V$ a real number $\langle \mathbf{u}, \mathbf{v} \rangle$ so that the following inner product axioms hold for all $\mathbf{u}, \mathbf{v}, \mathbf{w}$ in $V$ and every real $c$ .

\begin{aligned} &\langle \mathbf{u}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{u} \rangle \\[0.4em] &\langle \mathbf{u} + \mathbf{w}, \mathbf{v} \rangle = \langle \mathbf{u}, \mathbf{v} \rangle + \langle \mathbf{w}, \mathbf{v} \rangle \\[0.4em] &\langle c\mathbf{u}, \mathbf{v} \rangle = c\,\langle \mathbf{u}, \mathbf{v} \rangle \\[0.4em] &\langle \mathbf{v}, \mathbf{v} \rangle > 0 \text{ whenever } \mathbf{v} \ne \mathbf{0} \end{aligned}

The first axiom is symmetry, the next two are linearity in the first argument, and the last is positive definiteness. Symmetry carries linearity over to the second argument, so the inner product is linear in each argument while the other is held fixed. Linearity gives $\langle \mathbf{0}, \mathbf{v} \rangle = 0$ , so $\langle \mathbf{v}, \mathbf{v} \rangle \ge 0$ for every $\mathbf{v}$ , with equality only for $\mathbf{v} = \mathbf{0}$ . A real vector space together with a chosen inner product is a real inner product space.

In plain words An inner product is a rule for multiplying two vectors to get a number, and it is what lets a vector space speak about length, distance and right angles. Different rules on the same space give different geometries, so a claim such as “these vectors are orthogonal” means something only once the inner product is named.

Example The dot product $\langle \mathbf{x}, \mathbf{y} \rangle = \mathbf{x}^{\top}\mathbf{y}$ is an inner product on $\mathbb{R}^n$ , since $\mathbf{x}^{\top}\mathbf{x}$ is a sum of squares that is positive unless every entry is $0$ . On $P_2$ , the polynomials of degree at most $2$ , compare coefficients in the basis $(1, t, t^2)$ . For $p = a_0 + a_1 t + a_2 t^2$ and $q = b_0 + b_1 t + b_2 t^2$ set

\langle p, q \rangle = a_0 b_0 + a_1 b_1 + a_2 b_2.

This is an inner product, because coefficients are unique and $\langle p, p \rangle = a_0^2 + a_1^2 + a_2^2$ is positive for every nonzero $p$ . For $p = 1 + 2t - t^2$ and $q = 3 + t + t^2$ it gives $3 + 2 - 1 = 4$ . Under this rule the polynomials $1 + t$ and $1 - t$ are orthogonal, since $1 \cdot 1 + 1 \cdot (-1) = 0$ .

Weighted inner product

Definition Let $w_1, \dots, w_n$ be positive real numbers and $G = \operatorname{diag}(w_1, \dots, w_n)$ . The weighted inner product on $\mathbb{R}^n$ with these weights is

\begin{aligned} &\langle \mathbf{x}, \mathbf{y} \rangle_G = \mathbf{x}^{\top} G \mathbf{y} \\[0.4em] &\quad = w_1 x_1 y_1 + \cdots + w_n x_n y_n. \end{aligned}

It is symmetric and linear in each argument, and $\langle \mathbf{x}, \mathbf{x} \rangle_G = w_1 x_1^2 + \cdots + w_n x_n^2 > 0$ for $\mathbf{x} \ne \mathbf{0}$ , so it satisfies the axioms. With every weight equal to $1$ it is the dot product. The weights must all be positive. If some $w_k \le 0$ , the formula is still symmetric and linear, but the nonzero vector $\mathbf{e}_k$ has $\langle \mathbf{e}_k, \mathbf{e}_k \rangle_G = w_k \le 0$ , so positive definiteness fails.

In plain words Each coordinate direction gets its own rate for turning coordinates into squared length. The vectors stay where they are and only the way of measuring them changes, so lengths, distances and angles can all change.

Example Take $G = \operatorname{diag}(4, 1)$ , $\mathbf{u} = (1, 1)$ and $\mathbf{v} = (1, -1)$ . Then

\begin{gathered} \langle \mathbf{u}, \mathbf{v} \rangle_G \\[0.4em] = 4 \cdot 1 \cdot 1 + 1 \cdot 1 \cdot (-1) = 3, \end{gathered}

while the dot product of the same vectors is $0$ . They are orthogonal for the dot product but not for this inner product. Doubling every first coordinate turns the weighted rule into the ordinary one, since $\langle \mathbf{x}, \mathbf{y} \rangle_G = (D\mathbf{x}) \cdot (D\mathbf{y})$ with $D = \operatorname{diag}(2, 1)$ , and indeed $(2, 1) \cdot (2, -1) = 3$ . The weights $-1$ and $1$ are not allowed, because $\mathbf{e}_1$ would pair with itself to give $-1$ . The weights $1$ and $0$ are not allowed either, because the nonzero vector $\mathbf{e}_2$ would pair with itself to give $0$ .

Norm from an inner product

Definition In a real inner product space, the norm of a vector $\mathbf{v}$ is

\lVert \mathbf{v} \rVert = \sqrt{\langle \mathbf{v}, \mathbf{v} \rangle},

which is defined because $\langle \mathbf{v}, \mathbf{v} \rangle \ge 0$ . It satisfies $\lVert \mathbf{v} \rVert \ge 0$ , with equality only for $\mathbf{v} = \mathbf{0}$ , and $\lVert c\mathbf{v} \rVert = \lvert c \rvert \, \lVert \mathbf{v} \rVert$ for every real $c$ . For a weighted inner product we write $\lVert \mathbf{v} \rVert_G$ . The same inner product defines the distance $\lVert \mathbf{u} - \mathbf{v} \rVert$ , orthogonality $\langle \mathbf{u}, \mathbf{v} \rangle = 0$ and, for nonzero $\mathbf{u}$ and $\mathbf{v}$ , the angle $\theta$ in $[0, \pi]$ with

\cos \theta = \frac{\langle \mathbf{u}, \mathbf{v} \rangle}{\lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert}.

The Cauchy-Schwarz inequality below guarantees that this quotient lies in $[-1, 1]$ . Symmetry and linearity give, for every real $t$ ,

\begin{aligned} \lVert \mathbf{u} + t\mathbf{v} \rVert^2 &= \lVert \mathbf{u} \rVert^2 + 2t\langle \mathbf{u}, \mathbf{v} \rangle \\[0.4em] &\quad + t^2 \lVert \mathbf{v} \rVert^2. \end{aligned}

In plain words Every inner product comes with its own ruler. Length is the square root of a vector’s inner product with itself, and distance and angle are built from the same rule, so changing the inner product can change all three.

Example With $G = \operatorname{diag}(4, 1)$ the standard vectors have norms $\lVert \mathbf{e}_1 \rVert_G = 2$ and $\lVert \mathbf{e}_2 \rVert_G = 1$ . The unit vectors for this norm are the vectors $(x_1, x_2)$ with $4x_1^2 + x_2^2 = 1$ , which lie on an ellipse and include $(\frac{1}{2}, 0)$ . For $\mathbf{u} = (1, 1)$ and $\mathbf{v} = (1, -1)$ both norms are $\sqrt{5}$ , so $\cos \theta = \frac{3}{5}$ and the angle is acute, about $0.927$ radians. Their distance is $\lVert (0, 2) \rVert_G = 2$ , and the expansion with $t = -1$ agrees, since $5 - 2 \cdot 3 + 5 = 4$ .

Cauchy-Schwarz inequality

Definition For all vectors $\mathbf{u}$ and $\mathbf{v}$ in a real inner product space,

\lvert \langle \mathbf{u}, \mathbf{v} \rangle \rvert \le \lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert.

The proof uses only the axioms. If $\mathbf{v} = \mathbf{0}$ , both sides are $0$ . If $\mathbf{v} \ne \mathbf{0}$ , then $\lVert \mathbf{v} \rVert^2 > 0$ , and the expansion above with $-t$ in place of $t$ shows that for every real $t$

\begin{aligned} 0 &\le \lVert \mathbf{u} - t\mathbf{v} \rVert^2 \\[0.4em] &= \lVert \mathbf{u} \rVert^2 - 2t\langle \mathbf{u}, \mathbf{v} \rangle + t^2 \lVert \mathbf{v} \rVert^2. \end{aligned}

The value $t_0 = \langle \mathbf{u}, \mathbf{v} \rangle / \lVert \mathbf{v} \rVert^2$ is where this quadratic in $t$ is smallest. Choosing it makes the last two terms combine, and

\begin{aligned} 0 &\le \lVert \mathbf{u} - t_0\mathbf{v} \rVert^2 \\[0.4em] &= \lVert \mathbf{u} \rVert^2 - \frac{\langle \mathbf{u}, \mathbf{v} \rangle^2}{\lVert \mathbf{v} \rVert^2}. \end{aligned}

Multiplying by $\lVert \mathbf{v} \rVert^2$ gives $\langle \mathbf{u}, \mathbf{v} \rangle^2 \le \lVert \mathbf{u} \rVert^2 \lVert \mathbf{v} \rVert^2$ , and taking nonnegative square roots gives the inequality. Since the proof never uses coordinates, it holds for the dot product, for every weighted inner product and for the coefficient inner product on $P_2$ alike.

In plain words The inner product of two vectors is never larger in size than the product of their lengths. That is exactly what the cosine formula needs to produce a number between $-1$ and $1$ , so every inner product gives well-defined angles between nonzero vectors.

Example For $G = \operatorname{diag}(4, 1)$ , $\mathbf{u} = (1, 1)$ and $\mathbf{v} = (1, -1)$ the inequality reads $3 \le \sqrt{5} \cdot \sqrt{5} = 5$ . The proof’s choice is $t_0 = \frac{3}{5}$ , and the vector

\mathbf{u} - \tfrac{3}{5}\mathbf{v} = \left( \tfrac{2}{5}, \tfrac{8}{5} \right)

has weighted squared norm $4 \cdot \frac{4}{25} + \frac{64}{25} = \frac{16}{5}$ , which equals $5 - \frac{9}{5}$ as the formula predicts.

Equality case of Cauchy-Schwarz

Definition For vectors $\mathbf{u}$ and $\mathbf{v}$ in a real inner product space,

\lvert \langle \mathbf{u}, \mathbf{v} \rangle \rvert = \lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert

holds if and only if $\mathbf{u}$ and $\mathbf{v}$ are linearly dependent, that is, one of them is a scalar multiple of the other. If $\mathbf{v} = \mathbf{0}$ , the pair is dependent and both sides are $0$ . If $\mathbf{v} \ne \mathbf{0}$ , equality means $\lVert \mathbf{u} - t_0\mathbf{v} \rVert^2 = 0$ in the proof above, and positive definiteness turns this into $\mathbf{u} = t_0\mathbf{v}$ . Conversely, if $\mathbf{u} = c\mathbf{v}$ , both sides equal $\lvert c \rvert \, \lVert \mathbf{v} \rVert^2$ , and the case $\mathbf{v} = c\mathbf{u}$ is the same with the roles exchanged. For nonzero vectors the cosine is $1$ when the multiple is positive and $-1$ when it is negative.

In plain words The bound is reached exactly when the two vectors lie on one line through the origin. For two nonzero vectors this means they point the same way or opposite ways. For every other pair the inequality is strict.

Example With $G = \operatorname{diag}(4, 1)$ take $\mathbf{u} = (2, -2)$ and $\mathbf{v} = (-1, 1)$ , so $\mathbf{u} = -2\mathbf{v}$ . Then $\langle \mathbf{u}, \mathbf{v} \rangle_G = -8 - 2 = -10$ , while $\lVert \mathbf{u} \rVert_G = \sqrt{20}$ and $\lVert \mathbf{v} \rVert_G = \sqrt{5}$ have product $10$ . Equality holds and the cosine is $-1$ . The pair $(1, 1)$ and $(1, -1)$ from before is independent, and correspondingly $3 < 5$ . For that pair the smallest value of $\lVert (1, 1) - t(1, -1) \rVert_G^2$ is $\frac{16}{5}$ , not $0$ .

Triangle inequality

Definition For all vectors $\mathbf{u}$ and $\mathbf{v}$ in a real inner product space,

\lVert \mathbf{u} + \mathbf{v} \rVert \le \lVert \mathbf{u} \rVert + \lVert \mathbf{v} \rVert.

Using $\langle \mathbf{u}, \mathbf{v} \rangle \le \lvert \langle \mathbf{u}, \mathbf{v} \rangle \rvert \le \lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert$ from Cauchy-Schwarz,

\begin{aligned} &\lVert \mathbf{u} + \mathbf{v} \rVert^2 \\[0.4em] &\quad = \lVert \mathbf{u} \rVert^2 + 2\langle \mathbf{u}, \mathbf{v} \rangle + \lVert \mathbf{v} \rVert^2 \\[0.4em] &\quad \le \lVert \mathbf{u} \rVert^2 + 2\lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert + \lVert \mathbf{v} \rVert^2 \\[0.4em] &\quad = \left( \lVert \mathbf{u} \rVert + \lVert \mathbf{v} \rVert \right)^2, \end{aligned}

and taking nonnegative square roots gives the inequality. Equality holds exactly when $\langle \mathbf{u}, \mathbf{v} \rangle = \lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert$ , which happens exactly when one of the vectors is a nonnegative multiple of the other. Applied to $\mathbf{u} - \mathbf{v}$ and $\mathbf{v} - \mathbf{w}$ , the inequality gives the distance form

\begin{aligned} \lVert \mathbf{u} - \mathbf{w} \rVert &\le \lVert \mathbf{u} - \mathbf{v} \rVert \\[0.4em] &\quad + \lVert \mathbf{v} - \mathbf{w} \rVert. \end{aligned}

In plain words Walking along $\mathbf{u}$ and then along $\mathbf{v}$ ends at $\mathbf{u} + \mathbf{v}$ , and the straight route there is never longer than the two legs together. Two nonzero vectors give equality exactly when they point the same way. If they point opposite ways, Cauchy-Schwarz is an equality but the triangle inequality is strict.

Example With $G = \operatorname{diag}(4, 1)$ and $\mathbf{u} = (1, 1)$ , the vector $\mathbf{v} = (2, 2) = 2\mathbf{u}$ gives equality, since $\lVert (3, 3) \rVert_G = \sqrt{45} = 3\sqrt{5}$ and $\lVert \mathbf{u} \rVert_G + \lVert \mathbf{v} \rVert_G = \sqrt{5} + 2\sqrt{5}$ . The vector $\mathbf{w} = (-2, -2) = -2\mathbf{u}$ gives equality in Cauchy-Schwarz, because $\lvert \langle \mathbf{u}, \mathbf{w} \rangle_G \rvert = 10 = \sqrt{5} \cdot \sqrt{20}$ . Yet $\lVert \mathbf{u} + \mathbf{w} \rVert_G = \lVert (-1, -1) \rVert_G = \sqrt{5}$ , which is strictly less than $\sqrt{5} + \sqrt{20} = 3\sqrt{5}$ .

Check yourself

On $\mathbb{R}^2$ use the weighted inner product $\langle \mathbf{x}, \mathbf{y} \rangle = 2x_1 y_1 + 3x_2 y_2$ . For $\mathbf{a} = (1, 2)$ and $\mathbf{b} = (3, -1)$ , find $\langle \mathbf{a}, \mathbf{b} \rangle$ , $\lVert \mathbf{a} \rVert$ and $\lVert \mathbf{b} \rVert$ . Are the vectors orthogonal for this inner product? Are they orthogonal for the dot product?

Show the answer

The inner product is $2 \cdot 1 \cdot 3 + 3 \cdot 2 \cdot (-1) = 6 - 6 = 0$ , so the vectors are orthogonal for this inner product. The norms are $\lVert \mathbf{a} \rVert = \sqrt{2 + 12} = \sqrt{14}$ and $\lVert \mathbf{b} \rVert = \sqrt{18 + 3} = \sqrt{21}$ . As a check, $\mathbf{a} + \mathbf{b} = (4, 1)$ has squared norm $2 \cdot 16 + 3 \cdot 1 = 35 = 14 + 21$ , the Pythagorean identity for orthogonal vectors. The dot product is $3 - 2 = 1$ , so the vectors are not orthogonal in the ordinary sense.

Decide which of these rules on $\mathbb{R}^2$ are inner products. (a) $\langle \mathbf{x}, \mathbf{y} \rangle = x_1 y_1 + 5x_2 y_2$ . (b) $\langle \mathbf{x}, \mathbf{y} \rangle = x_1 y_1$ . (c) $\langle \mathbf{x}, \mathbf{y} \rangle = x_1 y_2 + x_2 y_1$ .

Show the answer

All three rules are symmetric and linear in each argument, so only positive definiteness needs a check.

Rule (a) is an inner product. Its weights $1$ and $5$ are positive, and $\langle \mathbf{x}, \mathbf{x} \rangle = x_1^2 + 5x_2^2 > 0$ for $\mathbf{x} \ne \mathbf{0}$ .

Rule (b) is not. The nonzero vector $(0, 1)$ pairs with itself to give $0$ , so a nonzero vector would have length $0$ .

Rule (c) is not. Here $\langle \mathbf{x}, \mathbf{x} \rangle = 2x_1 x_2$ , and $\mathbf{x} = (1, -1)$ gives $-2$ , a negative squared length.

Use $G = \operatorname{diag}(4, 1)$ , $\mathbf{u} = (2, k)$ and $\mathbf{v} = (1, 3)$ . For which $k$ is the Cauchy-Schwarz inequality an equality? For that $k$ , is the triangle inequality an equality too? What changes if $\mathbf{u}$ is replaced by $(-2, -6)$ ?

Show the answer

Equality in Cauchy-Schwarz holds exactly when the vectors are dependent. Since $\mathbf{v} \ne \mathbf{0}$ , that means $\mathbf{u} = c\mathbf{v}$ , and the first entries force $c = 2$ , so $k = 6$ . Check that $\langle \mathbf{u}, \mathbf{v} \rangle_G = 8 + 18 = 26$ , while $\lVert \mathbf{u} \rVert_G = \sqrt{52}$ and $\lVert \mathbf{v} \rVert_G = \sqrt{13}$ have product $\sqrt{676} = 26$ .

Because $\mathbf{u} = 2\mathbf{v}$ is a positive multiple, the triangle inequality is also an equality. Indeed $\lVert \mathbf{u} + \mathbf{v} \rVert_G = \lVert 3\mathbf{v} \rVert_G = 3\sqrt{13}$ , which equals $2\sqrt{13} + \sqrt{13}$ .

For $(-2, -6) = -2\mathbf{v}$ the pair is still dependent, so Cauchy-Schwarz is still an equality, with inner product $-26$ . But the sum is $-\mathbf{v}$ , of norm $\sqrt{13}$ , which is strictly less than $3\sqrt{13}$ . The triangle inequality is strict, because the multiple is negative.

Inner Products and Cauchy-Schwarz

Key terms

Inner product

Weighted inner product

Norm from an inner product

Cauchy-Schwarz inequality

Equality case of Cauchy-Schwarz

Triangle inequality

Check yourself

How this lesson connects

Builds on

Used later in

Where this leads in Linear Algebra II