Expectation, Variance, and Covariance

Definition Let $Y$ be a discrete random variable with PMF $p_Y$. If $\sum_y \lvert y \rvert\, p_Y(y)$ is finite, the expected value of $Y$ is

with the sum over the support. For a function $g$, the expected value $E[g(Y)]$ is $\sum_y g(y)\, p_Y(y)$, provided $\sum_y \lvert g(y) \rvert\, p_Y(y)$ is finite. For an absolutely continuous $Y$ with density $f$, the sum becomes an integral,

provided the same integral with $\lvert y \rvert$ in place of $y$ is finite. The expected value is a property of the model. The average of $n$ observed values is the expected value of the empirical distribution, which gives each observation the weight $1/n$. It is a different quantity, which can serve as an estimate of $E[Y]$ when the observations come from the population the model describes.

Definition Let $X$ and $Y$ be random variables on the same sample space with finite expected values, and let $a$, $b$ and $c$ be constants. Then

The rule extends to every finite sum, $E[X_1 + \dots + X_n] = E[X_1] + \dots + E[X_n]$, and it holds whether or not the variables are independent. For a random vector $\mathbf{Z}$ whose components have finite expected values, $E[\mathbf{Z}]$ is the vector of those expected values, and for a constant vector $\mathbf{w}$ linearity gives $E[\mathbf{w}^{\top}\mathbf{Z}] = \mathbf{w}^{\top}E[\mathbf{Z}]$. The rule covers linear combinations only. For a nonlinear function $g$, the value $E[g(X)]$ need not equal $g(E[X])$.

Definition Let $Y$ have a finite second moment $E[Y^2]$, and write $\mu = E[Y]$. The variance of $Y$ is

and the standard deviation $\sigma_Y$ is its nonnegative square root. Expanding the square gives $\operatorname{Var}(Y) = E[Y^2] - \mu^2$. For constants $a$ and $b$, $\operatorname{Var}(aY + b) = a^2 \operatorname{Var}(Y)$, and the standard deviation is multiplied by $\lvert a \rvert$. For observed values $y_1, \dots, y_n$ with average $\bar{y}$, the variance of the empirical distribution divides $\sum_{i=1}^{n} (y_i - \bar{y})^2$ by $n$. For $n \ge 2$ the usual sample variance divides by $n - 1$ instead,

When the observations are independent and identically distributed with finite variance, the expected value of $s^2$ is exactly $\operatorname{Var}(Y)$. The divisor $n - 1$ compensates for measuring the deviations from $\bar{y}$, which is computed from the same observations, rather than from $\mu$.

Definition For random variables $X$ and $Y$ with finite second moments and means $\mu_X$ and $\mu_Y$, the covariance is

It is symmetric in $X$ and $Y$, it satisfies $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$, and $\operatorname{Cov}(aX + b, Y) = a \operatorname{Cov}(X, Y)$ for constants $a$ and $b$, so its value depends on the units of both variables. When both standard deviations are positive, the correlation is

It has no units, it lies in $[-1, 1]$, and it equals $1$ or $-1$ exactly when $Y = \alpha + \beta X$ with probability $1$ for some constants $\alpha$ and $\beta \ne 0$. If either standard deviation is $0$, the correlation is undefined. Independent variables with finite second moments have covariance $0$, but covariance $0$ does not imply independence.

Definition Let $\mathbf{Z} = (Z_1, \dots, Z_d)^{\top}$ be a random vector whose components have finite second moments, with mean vector $\boldsymbol{\mu} = E[\mathbf{Z}]$. Its covariance matrix is the $d \times d$ matrix

whose entry in row $j$ and column $k$ is $\operatorname{Cov}(Z_j, Z_k)$, with the variances on the diagonal. It is symmetric and positive semidefinite, because for every constant vector $\mathbf{v}$

For data, center the columns of an $n \times p$ design matrix $X$ by subtracting each column’s average, call the result $X_c$, and let $n \ge 2$. The sample covariance matrix $X_c^{\top} X_c/(n - 1)$ is an estimate of the covariance matrix of the feature vector when the rows are independent and identically distributed draws of it.

Definition Let $T$ be a discrete random variable, let $t$ be a value with $P(T = t) > 0$, and let $Y$ be a discrete random variable with a finite expected value. Write $p(y \mid t) = P(Y = y \mid T = t)$. The conditional expectation of $Y$ given $T = t$ is the expected value of this conditional distribution,

The law of total expectation says that $E[Y]$ is the sum of $E[Y \mid T = t]\,P(T = t)$ over the support of $T$. If $Y$ also has a finite second moment, then for every number $a$

where $\operatorname{Var}(Y \mid T = t)$ is the variance of the conditional distribution. So the conditional mean is the single prediction with the smallest expected squared error given $T = t$. Read as a function of $t$, it is the prediction rule with the smallest expected squared error, which is why regression under squared loss aims at it. When $T$ is continuous, $P(T = t) = 0$, so conditional probabilities given $T = t$ cannot be defined as ratios, and a more general construction is needed.