Regression and statistical identification

Definition Let $y \in \mathbb{R}^n$ be a vector of responses and $X$ an $n \times k$ design matrix whose columns are the regressors, often including a column of ones for an intercept. An ordinary least squares estimate is any $\hat\beta \in \mathbb{R}^k$ that minimizes, over $b \in \mathbb{R}^k$, the sum of squared residuals

The fitted vector $\hat y = X\hat\beta$ is the orthogonal projection of $y$ onto the column space $\operatorname{Col}(X)$, so a minimizer always exists and the fitted vector is unique. If $X$ has full column rank $k$, which requires $n \ge k$, the minimizer itself is unique and

If the columns are linearly dependent, the minimizers form the set $\hat\beta + \operatorname{Null}(X)$, and the pseudoinverse solution $X^{+}y$ is the one with the smallest Euclidean norm. With an intercept and one regressor, so that $X$ has rows $(1, x_j)$, the slope and intercept are

Nothing in this definition involves probability.

Definition The normal equations of the least squares problem are

The objective $S(b) = \lVert y - Xb \rVert^2$ has gradient $-2X^\top(y - Xb)$ and Hessian $2X^\top X$, which is positive semidefinite, so $S$ is convex and $b$ minimizes $S$ if and only if it solves the normal equations. Equivalently, the residual $e = y - X\hat\beta$ is orthogonal to every column of $X$, that is, $X^\top e = 0$. Writing $y - Xb = e + X(\hat\beta - b)$ and using $X^\top e = 0$ gives, for every $b \in \mathbb{R}^k$,

which proves the minimum directly. The normal equations always have a solution. The matrix $X^\top X$ is invertible exactly when $X$ has full column rank, and then it is positive definite and the solution is unique. When $X$ contains a column of ones, the corresponding equation says that the residuals sum to zero, and with one regressor the fitted line passes through the point of means $(\bar x, \bar y)$.

Definition Consider the linear model

with a random $n \times k$ design $X$, an unknown coefficient vector $\beta \in \mathbb{R}^k$ and an error vector $\varepsilon$ whose entries are integrable. The regressors are strictly exogenous if $\mathbb{E}[\varepsilon \mid X] = 0$, so every error has conditional mean zero given the regressors of all observations. If $X$ has full column rank, substituting the model into the formula for $\hat\beta$ gives

and therefore $\mathbb{E}[\hat\beta \mid X] = \beta$. This is conditional unbiasedness, and it gives $\mathbb{E}[\hat\beta] = \beta$ whenever $\hat\beta$ is integrable. A weaker condition, contemporaneous exogeneity, asks only that the error of each observation have mean zero and be uncorrelated with the regressors of the same observation. It does not by itself give unbiasedness. Together with independent and identically distributed observations, finite second moments and a nonsingular second-moment matrix of the regressors, it gives consistency of $\hat\beta$ through the law of large numbers, a result stated here without the details. In an autoregression the regressor $y_{t-1}$ depends on earlier errors, so strict exogeneity fails while contemporaneous exogeneity can still hold.

Definition A parameter is identified in a statistical model if the distribution of the observable data determines its value uniquely, so that two different parameter values allowed by the model never produce the same distribution of observables. For a pair $(x, y)$ with a random regressor vector $x \in \mathbb{R}^k$ and a response $y$, both with finite second moments, and a nonsingular matrix $\mathbb{E}[x x^\top]$, the coefficient of the best linear predictor of $y$ given $x$,

is identified. If $y = x^\top\beta + \varepsilon$, then

so the projection coefficient equals the model coefficient exactly when $\mathbb{E}[x\varepsilon] = 0$. If $\mathbb{E}[x x^\top]$ is singular, some regressor is almost surely a linear combination of the others, and only certain combinations of the coefficients are identified. Reading $\beta$ as the change in $y$ caused by an intervention on $x$ needs further assumptions or a research design that rules out confounding, reverse causation and selection. Using it for forecasting needs regressors that are known when the forecast is made.

Definition Suppose scalar random variables satisfy

where $X$, $Z$ and $\varepsilon$ have finite second moments, $\operatorname{Var}(X) > 0$ and $\operatorname{Cov}(X, \varepsilon) = 0$. The population slope of the regression of $Y$ on $X$ alone, with an intercept, is $\tilde\beta = \operatorname{Cov}(X, Y)/\operatorname{Var}(X)$. Covariance with a constant is zero, so

and dividing by $\operatorname{Var}(X)$ gives

The second term is the omitted variable bias. It vanishes exactly when $\gamma = 0$ or $\operatorname{Cov}(X, Z) = 0$. Under independent and identically distributed sampling with finite second moments, the slope of the short regression converges in probability to $\tilde\beta$, not to $\beta$. The same algebra holds exactly in every sample. The short regression slope equals the long regression’s coefficient on $X$ plus its coefficient on $Z$ times the slope from regressing $Z$ on $X$.

Definition In the linear model with strict exogeneity and full column rank, let $\Omega = \operatorname{Var}(\varepsilon \mid X) = \mathbb{E}[\varepsilon\varepsilon^\top \mid X]$, assumed finite. From $\hat\beta - \beta = (X^\top X)^{-1}X^\top\varepsilon$,

called the sandwich covariance. The errors are homoskedastic when $\Omega = \sigma^2 I_n$, for example when they are uncorrelated with a common conditional variance. The sandwich then reduces to $\sigma^2(X^\top X)^{-1}$, and with $n > k$ the estimate $\hat\sigma^2 = \lVert e \rVert^2/(n - k)$ is conditionally unbiased for $\sigma^2$. The errors are heteroskedastic when the conditional variances $\operatorname{Var}(\varepsilon_j \mid X)$ differ across observations. Heteroskedasticity leaves the normal equations, the fitted vector and conditional unbiasedness unchanged. It changes only the covariance, and the classical formula can then be too small or too large. The heteroskedasticity-consistent estimator replaces $\Omega$ by $\operatorname{diag}(e_1^2, \dots, e_n^2)$ built from the residuals. Its justification is asymptotic, for independent observations under moment conditions, and is stated here without proof. With serially correlated errors $\Omega$ also has off-diagonal entries, which a diagonal correction does not cover, and estimators that add cross-date terms are needed.