Covariance, Correlation and Sums

Definition Let $X$ and $Y$ be random variables on the same probability model with finite second moments, $E[X^2] < \infty$ and $E[Y^2] < \infty$. Then the means $\mu_X = E[X]$ and $\mu_Y = E[Y]$ are finite, and so is $E[XY]$, because $\lvert xy \rvert \le \tfrac12 (x^2 + y^2)$ for all real $x$ and $y$. The covariance of $X$ and $Y$ is the expected product of their deviations from their means,

The second form comes from expanding the product and using linearity of expectation. Its subtracted term is the product of the two means, not the mean of the product. Covariance is symmetric, $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$, and its unit is the product of the units of $X$ and $Y$. It can be positive, negative or zero. Positive covariance says that the centered products are positive on average, so the products from deviations with the same sign outweigh those from deviations with opposite signs.

Definition Random variables $X$ and $Y$ with finite second moments are uncorrelated if $\operatorname{Cov}(X, Y) = 0$, or equivalently $E[XY] = E[X]\,E[Y]$. Independent variables with finite second moments are uncorrelated. For discrete variables the product law turns the double sum for $E[XY]$ into the product of the sums for $E[X]$ and $E[Y]$. The converse is false. Uncorrelated variables can be dependent, even when one of them is a function of the other.

Definition If $X$ and $Y$ have finite second moments, then

with or without independence. The formula comes from expanding $\bigl((X - \mu_X) + (Y - \mu_Y)\bigr)^2$ and taking expectations. For constants $a$ and $b$ the same expansion gives

so in particular $\operatorname{Var}(X - Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) - 2\operatorname{Cov}(X, Y)$. For $n$ variables with finite second moments,

The variances add exactly when the covariance terms add to $0$. This holds when the variables are pairwise uncorrelated, and in particular when every two of them are independent.

Definition For random variables with finite second moments and constants $a$, $b$, $c$ and $d$,

and covariance distributes over sums in each argument, as in $\operatorname{Cov}(X + Y, Z) = \operatorname{Cov}(X, Z) + \operatorname{Cov}(Y, Z)$. Together with the symmetry $\operatorname{Cov}(X, Y) = \operatorname{Cov}(Y, X)$, these rules say that covariance is linear in each argument when the other one is held fixed, which is what bilinear means. Added constants drop out because centering removes them, and $\operatorname{Cov}(X, c) = 0$ for every constant $c$. With $Y = X$ the rules give $\operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X)$, and expanding $\operatorname{Cov}(X + Y, X + Y)$ term by term gives the variance of a sum again.

Definition Let $X$ and $Y$ have finite second moments and positive variances. Their correlation coefficient is

It has no unit. For constants $a \ne 0$, $c \ne 0$, $b$ and $d$, the correlation of $aX + b$ and $cY + d$ equals $\rho_{X,Y}$ when $ac > 0$ and $-\rho_{X,Y}$ when $ac < 0$, so a change of units with positive scale factors leaves it unchanged. If one of the variances is $0$, that variable equals a constant with probability $1$, its covariance with every variable is $0$, and the correlation is undefined, because the formula divides by $0$. Such a case is reported as undefined, not as zero correlation. Uncorrelated variables with positive variances have correlation $0$.

Definition For random variables $U$ and $V$ with finite second moments, the Cauchy-Schwarz inequality states

Applied to the centered variables $X - \mu_X$ and $Y - \mu_Y$, it gives $\operatorname{Cov}(X, Y)^2 \le \operatorname{Var}(X)\operatorname{Var}(Y)$, so $-1 \le \rho_{X,Y} \le 1$ whenever the correlation is defined. A direct argument standardizes the variables. With $\rho = \rho_{X,Y}$, the variables $U = (X - \mu_X)/\operatorname{SD}(X)$ and $V = (Y - \mu_Y)/\operatorname{SD}(Y)$ have mean $0$, variance $1$ and $E[UV] = \rho$, so

Perfect correlation means $\rho_{X,Y} = 1$ or $\rho_{X,Y} = -1$. For $X$ and $Y$ with positive variances, it happens exactly when $Y = aX + b$ with probability $1$ for constants $a \ne 0$ and $b$, with $\rho_{X,Y} = 1$ when $a > 0$ and $\rho_{X,Y} = -1$ when $a < 0$. For example, $\rho_{X,Y} = 1$ forces $E[(U - V)^2] = 2 - 2\rho_{X,Y} = 0$, so $U = V$ with probability $1$.