Vectors, Matrices, and Linear Predictions

Definition A vector in $\mathbb{R}^p$ is an ordered list of $p$ real numbers. On this page every vector is a column, and its transpose, written with $^{\top}$, is the same numbers written as a row. In a data set with $p$ features listed in a fixed order, the feature vector of one observation is

where $x_j$ is the value of feature $j$ in its stated unit. A linear model pairs it with a coefficient vector $\mathbf{w} = (w_1, \dots, w_p)^{\top}$ in the same order, so $w_j$ belongs to feature $j$ and is measured in units of the prediction per unit of feature $j$. The same coefficients give meaningful predictions only for feature vectors that list the same features in the same order and units.

Definition For two vectors $\mathbf{u}$ and $\mathbf{v}$ in $\mathbb{R}^p$, the dot product is

It is defined only when both vectors have the same number of entries. It is symmetric, $\mathbf{u}^{\top}\mathbf{v} = \mathbf{v}^{\top}\mathbf{u}$, and linear in each vector. As a matrix product, the $1 \times p$ row $\mathbf{u}^{\top}$ times the $p \times 1$ column $\mathbf{v}$ is a $1 \times 1$ matrix, which is treated as a single number.

Definition A real $n \times p$ matrix is a rectangular array of real numbers with $n$ rows and $p$ columns. Its entry in row $i$ and column $j$ is written $X_{ij}$, and its transpose $X^{\top}$ is the $p \times n$ matrix with entries $(X^{\top})_{ji} = X_{ij}$. For a data set of $n$ observations with $p$ features, the design matrix $X$ is the $n \times p$ matrix whose row $i$ is $\mathbf{x}_i^{\top}$, the feature vector of observation $i$ written as a row. Column $j$ then holds the values of feature $j$ for all $n$ observations. The targets form a vector $\mathbf{y}$ in $\mathbb{R}^n$ listed in the same row order. On this page $X$ holds only measured features, and the intercept column of a later card is kept separate.

Definition Let $X$ be an $n \times p$ matrix with rows $\mathbf{x}_1^{\top}, \dots, \mathbf{x}_n^{\top}$, and let $\mathbf{w}$ be a vector in $\mathbb{R}^p$. The product $X\mathbf{w}$ is the vector in $\mathbb{R}^n$ whose entry $i$ is the dot product of row $i$ with $\mathbf{w}$,

The product is defined only when the number of columns of $X$ equals the number of entries of $\mathbf{w}$, and the shapes combine as $(n \times p)(p \times 1) = n \times 1$. Equivalently, if $\mathbf{c}_j$ denotes column $j$ of $X$, then $X\mathbf{w} = w_1 \mathbf{c}_1 + \dots + w_p \mathbf{c}_p$, a combination of the feature columns weighted by the coefficients.

Definition A linear model with intercept $b$ adds the same number $b$ to every prediction,

where $\mathbf{1}$ is the vector of $n$ ones. The same predictions come from the augmented design matrix and the augmented coefficient vector

since $\tilde{X}\boldsymbol{\theta} = b\,\mathbf{1} + X\mathbf{w}$. The matrix $\tilde{X}$ is $n \times (p + 1)$ and $\boldsymbol{\theta}$ has $p + 1$ entries. The first column of $\tilde{X}$, all ones, is the intercept column. It is bookkeeping for the constant term, not a measured feature, and it stays labeled apart from the features because fitting procedures often treat it differently. For example, the intercept is usually left out of a regularization penalty, and the ones column is not standardized like a feature.

Definition For a feature vector $\mathbf{x}$ in $\mathbb{R}^p$, coefficients $\mathbf{w}$ in $\mathbb{R}^p$ and an intercept $b$, the linear prediction is

For $n$ observations the predictions are $\hat{\mathbf{y}} = X\mathbf{w} + b\,\mathbf{1}$, with $X$ of shape $n \times p$, $\mathbf{w}$ with $p$ entries and $\hat{\mathbf{y}}$ with $n$ entries. As a function of $\mathbf{x}$ the map is linear in the strict sense only when $b = 0$. With $b \ne 0$ it is affine, a linear map plus a constant, and “linear model” is the usual name that includes the intercept. For a fixed $\mathbf{x}$ the prediction is linear in the parameters $b$ and $\mathbf{w}$. Raising feature $j$ by one unit while every other feature stays fixed changes $\hat{y}$ by exactly $w_j$.