Orthogonal Complements and the Four Subspaces

Definition Let $W$ be a subspace of $\mathbb{R}^n$. The orthogonal complement of $W$, written $W^{\perp}$ and read ”$W$ perp”, is the set of vectors in the same space $\mathbb{R}^n$ that have zero dot product with every vector of $W$,

It is a subspace of $\mathbb{R}^n$, because it contains $\mathbf{0}$ and $(a\mathbf{v} + b\mathbf{z}) \cdot \mathbf{w} = a(\mathbf{v} \cdot \mathbf{w}) + b(\mathbf{z} \cdot \mathbf{w})$, which is $0$ when $\mathbf{v}$ and $\mathbf{z}$ lie in $W^{\perp}$. If $\mathbf{w}_1, \dots, \mathbf{w}_k$ span $W$, then $\mathbf{v}$ lies in $W^{\perp}$ exactly when $\mathbf{v} \cdot \mathbf{w}_i = 0$ for each $i = 1, \dots, k$, since a dot product with a combination of the $\mathbf{w}_i$ is the same combination of zeros. The only vector in both $W$ and $W^{\perp}$ is $\mathbf{0}$, because such a vector satisfies $\lVert \mathbf{v} \rVert^2 = \mathbf{v} \cdot \mathbf{v} = 0$. At the extremes, $\{\mathbf{0}\}^{\perp} = \mathbb{R}^n$ and $(\mathbb{R}^n)^{\perp} = \{\mathbf{0}\}$.

Definition Let $A$ be a real $m \times n$ matrix, a map from $\mathbb{R}^n$ to $\mathbb{R}^m$. Each row has $n$ entries and, written as a column, is a vector of $\mathbb{R}^n$. The row space $\operatorname{Row} A$ is the span of these vectors, which equals $\operatorname{Col} A^{\top}$, so it lies in the input space $\mathbb{R}^n$, and so does $\operatorname{Null} A = \{\mathbf{x} \in \mathbb{R}^n \mid A\mathbf{x} = \mathbf{0}\}$. These two subspaces are orthogonal complements,

The reason is that entry $i$ of $A\mathbf{x}$ is the dot product of row $i$ with $\mathbf{x}$. So $A\mathbf{x} = \mathbf{0}$ holds exactly when $\mathbf{x}$ is perpendicular to every row, which is the same as being perpendicular to their span. The identity holds for every real matrix, square or not, whether or not its rows are independent.

Definition The left null space of a real $m \times n$ matrix $A$ is

The name comes from the equivalent equation $\mathbf{y}^{\top}A = \mathbf{0}^{\top}$, where $\mathbf{y}^{\top}$ multiplies $A$ from the left, but the vectors $\mathbf{y}$ are still columns in the output space $\mathbb{R}^m$. The rows of $A^{\top}$ are the columns of $A$, so the previous result applied to $A^{\top}$ gives

an identity between two subspaces of $\mathbb{R}^m$. If $\mathbf{b} = A\mathbf{x}$ and $\mathbf{y}$ is in $\operatorname{Null} A^{\top}$, then $\mathbf{y} \cdot \mathbf{b} = \mathbf{y}^{\top}A\mathbf{x} = 0$. So every vector of the left null space gives a linear condition that each attainable output satisfies.

Definition Let $A$ be a real $m \times n$ matrix of rank $r$. The row space and the column space both have dimension $r$. Rank-nullity for $A$ gives $\dim \operatorname{Null} A = n - r$, and rank-nullity for $A^{\top}$, which also has rank $r$, gives $\dim \operatorname{Null} A^{\top} = m - r$. Hence

The first count takes place in the input space and the second in the output space. For a square matrix both totals are $n$, but the subspaces involved still differ in general, because $\operatorname{Row} A$ and $\operatorname{Col} A$ need not be equal. More generally, every subspace $W$ of $\mathbb{R}^n$ satisfies

For $W = \{\mathbf{0}\}$ this reads $0 + n = n$. Otherwise, write a basis of $W$ with $k$ vectors as the rows of a $k \times n$ matrix $M$. Its rank is $k$, its null space is $W^{\perp}$ by the previous result, and rank-nullity gives $\dim W^{\perp} = n - k$.

Definition Let $W$ be a subspace of $\mathbb{R}^n$. Every $\mathbf{v}$ in $\mathbb{R}^n$ can be written in exactly one way as

with $\mathbf{w}$ in $W$ and $\mathbf{z}$ in $W^{\perp}$, and the two pieces are perpendicular. This is written $\mathbb{R}^n = W \oplus W^{\perp}$ and called an orthogonal direct sum. For existence, put a basis of $W$ and a basis of $W^{\perp}$ together. A combination of the first list that cancels a combination of the second gives a vector in $W \cap W^{\perp} = \{\mathbf{0}\}$, so both combinations are zero and all their coefficients vanish. The joined list is therefore independent, and by the dimension count it has $n$ vectors, so it is a basis of $\mathbb{R}^n$. For uniqueness, the difference of two such decompositions gives a vector in $W \cap W^{\perp}$, which is $\mathbf{0}$. Applied to a real $m \times n$ matrix $A$,

It also follows that $(W^{\perp})^{\perp} = W$, because every vector of $W$ is perpendicular to all of $W^{\perp}$, so $W$ lies inside $(W^{\perp})^{\perp}$, and both subspaces have dimension $n - \dim W^{\perp}$. In particular $\operatorname{Col} A = (\operatorname{Null} A^{\top})^{\perp}$, so $A\mathbf{x} = \mathbf{b}$ has a solution exactly when $\mathbf{b}$ is perpendicular to every vector of the left null space.