Orthonormal Coordinates and Line Projection

Definition A list $\mathbf{v}_1, \dots, \mathbf{v}_k$ of vectors in a real inner product space is orthogonal if $\langle \mathbf{v}_i, \mathbf{v}_j \rangle = 0$ whenever $i \ne j$. It is orthonormal if, in addition, every vector has norm $1$, that is

An orthogonal list may contain the zero vector, while an orthonormal list cannot. An orthogonal list of nonzero vectors is linearly independent. Indeed, if $c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k = \mathbf{0}$, taking the inner product of both sides with $\mathbf{v}_j$ leaves $c_j \lVert \mathbf{v}_j \rVert^2 = 0$, so $c_j = 0$ for each $j$. Dividing each vector of such a list by its norm gives an orthonormal list with the same span.

Definition An orthonormal basis of a finite-dimensional real inner product space $V$ is an orthonormal list that is a basis of $V$. Since orthonormal lists are independent, every orthonormal list of $\dim V$ vectors in $V$ is an orthonormal basis. If $\mathcal{B} = (\mathbf{q}_1, \dots, \mathbf{q}_n)$ is an orthonormal basis of $V$, then every $\mathbf{v}$ in $V$ satisfies

so its coordinates are $[\mathbf{v}]_{\mathcal{B}} = (\langle \mathbf{q}_1, \mathbf{v} \rangle, \dots, \langle \mathbf{q}_n, \mathbf{v} \rangle)$. To see this, write $\mathbf{v} = a_1\mathbf{q}_1 + \cdots + a_n\mathbf{q}_n$ and take the inner product with $\mathbf{q}_j$. Every term but one vanishes, and $\langle \mathbf{q}_j, \mathbf{q}_j \rangle = 1$, which leaves $\langle \mathbf{q}_j, \mathbf{v} \rangle = a_j$.

Definition Let $\mathbf{u}$ be a nonzero vector in $\mathbb{R}^n$ and $L = \operatorname{span}\{\mathbf{u}\}$, the line through the origin in the direction of $\mathbf{u}$. The orthogonal projection of a vector $\mathbf{b}$ in $\mathbb{R}^n$ onto $L$ is

The scalar $\alpha = (\mathbf{u} \cdot \mathbf{b})/(\mathbf{u} \cdot \mathbf{u})$ is the projection coefficient. It is the only scalar for which $\mathbf{b} - \alpha\mathbf{u}$ is orthogonal to $\mathbf{u}$, because the equation

has exactly one solution, since $\mathbf{u} \cdot \mathbf{u} > 0$ for $\mathbf{u} \ne \mathbf{0}$. For a unit vector $\mathbf{q}$ the denominator is $1$, so the coefficient is $\mathbf{q} \cdot \mathbf{b}$ and the projection is $(\mathbf{q} \cdot \mathbf{b})\,\mathbf{q}$. Replacing $\mathbf{u}$ by $c\mathbf{u}$ with $c \ne 0$ divides the coefficient by $c$ and leaves the projected vector unchanged, so the projection depends only on the line. In any real inner product space the same formula holds with $\langle \mathbf{u}, \mathbf{b} \rangle / \langle \mathbf{u}, \mathbf{u} \rangle$ as the coefficient. If $\mathbf{u} = \mathbf{0}$, its span is $\{\mathbf{0}\}$ rather than a line, and the projection onto that subspace is $\mathbf{0}$, with no division involved.

Definition Let $\mathbf{u} \ne \mathbf{0}$, $L = \operatorname{span}\{\mathbf{u}\}$ and $\mathbf{p} = \operatorname{proj}_L \mathbf{b}$. The residual vector is $\mathbf{r} = \mathbf{b} - \mathbf{p}$. It is orthogonal to $\mathbf{u}$, hence to every vector of $L$, and

writes $\mathbf{b}$ as a vector on $L$ plus a vector orthogonal to $L$. This split is unique. If $\mathbf{b} = \beta\mathbf{u} + \mathbf{s}$ with $\mathbf{s}$ orthogonal to $\mathbf{u}$, then $\mathbf{b} - \beta\mathbf{u}$ is orthogonal to $\mathbf{u}$, so $\beta$ is the projection coefficient. For orthogonal vectors $\mathbf{x}$ and $\mathbf{y}$ the Pythagorean identity $\lVert \mathbf{x} + \mathbf{y} \rVert^2 = \lVert \mathbf{x} \rVert^2 + \lVert \mathbf{y} \rVert^2$ holds, so

Definition Let $\mathbf{u} \ne \mathbf{0}$ in $\mathbb{R}^n$, $L = \operatorname{span}\{\mathbf{u}\}$, $\mathbf{b}$ in $\mathbb{R}^n$ and $\mathbf{p} = \operatorname{proj}_L \mathbf{b}$. For every $\mathbf{y}$ in $L$,

Hence $\lVert \mathbf{b} - \mathbf{y} \rVert \ge \lVert \mathbf{b} - \mathbf{p} \rVert$, with equality only for $\mathbf{y} = \mathbf{p}$. So $\mathbf{p}$ is the unique point of $L$ closest to $\mathbf{b}$, and the distance from $\mathbf{b}$ to $L$ is the length $\lVert \mathbf{r} \rVert$ of the residual. To prove the identity, write $\mathbf{b} - \mathbf{y} = \mathbf{r} + (\mathbf{p} - \mathbf{y})$. Both $\mathbf{p}$ and $\mathbf{y}$ lie on $L$, so $\mathbf{p} - \mathbf{y}$ lies on $L$ and is orthogonal to $\mathbf{r}$. The Pythagorean identity gives the equation, and its last term is $0$ only when $\mathbf{y} = \mathbf{p}$.

Definition A real $n \times n$ matrix $Q$ is orthogonal if its columns $\mathbf{q}_1, \dots, \mathbf{q}_n$ form an orthonormal basis of $\mathbb{R}^n$. Since the $(i, j)$ entry of $Q^{\top}Q$ is $\mathbf{q}_i \cdot \mathbf{q}_j$, this is the same as

Because $Q$ is square, it follows that $Q$ is invertible with $Q^{-1} = Q^{\top}$, so $QQ^{\top} = I$ as well. For the orthonormal basis $\mathcal{B}$ formed by its columns, coordinates are read off and rebuilt by

Multiplication by $Q$ keeps dot products and lengths, since $(Q\mathbf{x}) \cdot (Q\mathbf{y}) = \mathbf{x}^{\top}Q^{\top}Q\mathbf{y} = \mathbf{x} \cdot \mathbf{y}$. Despite the name, the columns must be orthonormal, and orthogonal columns alone are not enough. An $m \times n$ matrix with orthonormal columns and $m > n$ still satisfies $Q^{\top}Q = I_n$. But it is not square, so $Q^{\top}$ undoes it only from the left, and $QQ^{\top} \ne I_m$. Such a matrix is not called an orthogonal matrix.