Expected Value, Variance and Sums of Random Variables, statistics lesson

A product is often assembled from parts whose sizes vary, and what matters is the total. If each part has a known average and a known spread, what are the average and the spread of the total? The average of the total follows from a rule that does not depend on how the parts are related, while its spread depends on how the parts vary together. This lesson defines expected value and variance for a probability model and shows exactly where independence and covariance enter.

Key terms

Expected value

Definition Let $X$ be a discrete random variable with PMF $p_X$ . If $\sum_x \lvert x \rvert \, p_X(x)$ is finite, the expected value of $X$ is

E(X) = \sum_{x} x \, p_X(x),

with the sum running over the support of $X$ . The condition holds automatically when the support is finite. More generally, for a function $g$ the expected value of $g(X)$ is $E(g(X)) = \sum_x g(x) \, p_X(x)$ , provided $\sum_x \lvert g(x) \rvert \, p_X(x)$ is finite. The expected value is also called the mean of $X$ and written $\mu$ or $\mu_X$ .

In plain words The expected value is a probability-weighted average of the possible values, the balance point of the PMF. Over many independent repetitions of the same model, the average of the observed values tends to settle near it. It need not be a value that $X$ can take, and a single trial need not come out near it.

Example Let $X$ be $-1$ with probability $1/4$ and $1$ with probability $3/4$ . Then

\begin{aligned} E(X) &= (-1) \cdot \tfrac14 + 1 \cdot \tfrac34 \\[0.4em] &= \tfrac12 . \end{aligned}

No single trial gives $1/2$ , since $X$ is always $-1$ or $1$ . The value $1/2$ is the point where the two masses balance, closer to $1$ because that value carries more probability.

Variance of a random variable

Definition Let $X$ be a discrete random variable whose second moment

E(X^2) = \sum_x x^2 \, p_X(x)

is finite. Then the mean $\mu = E(X)$ is finite too, and the variance of $X$ is the expected squared deviation from the mean,

\operatorname{Var}(X) = E\bigl((X - \mu)^2\bigr).

The standard deviation is $\operatorname{SD}(X) = \sqrt{\operatorname{Var}(X)}$ . Expanding the square gives the identity

\operatorname{Var}(X) = E(X^2) - \bigl(E(X)\bigr)^2 .

The variance is never negative, and it equals $0$ exactly when $X = \mu$ with probability $1$ . For constants $a$ and $b$ ,

\begin{gathered} \operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X), \\[0.4em] \operatorname{SD}(aX + b) = \lvert a \rvert \operatorname{SD}(X). \end{gathered}

In plain words Variance measures spread as an average squared distance from the mean, so its unit is the square of the unit of $X$ , and the standard deviation returns to the original unit. The second moment measures squared distance from $0$ instead of from the mean, which is why the two are equal exactly when the mean is $0$ . Shifting $X$ moves the mean but not the spread, and multiplying $X$ by $a$ multiplies the spread by $\lvert a \rvert$ .

Example For the variable $X$ above, the deviations from $\mu = 1/2$ are $-3/2$ and $1/2$ , with squares $9/4$ and $1/4$ , so

\begin{aligned} \operatorname{Var}(X) &= \tfrac94 \cdot \tfrac14 + \tfrac14 \cdot \tfrac34 \\[0.4em] &= \tfrac{3}{4} . \end{aligned}

Both values of $X$ square to $1$ , so the second moment is $E(X^2) = 1$ , and the identity gives $1 - (1/2)^2 = 3/4$ again. The standard deviation is $\sqrt{3}/2 \approx 0.866$ .

Linearity of expectation

Definition Let $X$ and $Y$ be random variables defined on the same sample space, with finite expected values, and let $a$ , $b$ and $c$ be constants. Then

\begin{aligned} &E(aX + bY + c) \\[0.4em] &\quad = aE(X) + bE(Y) + c . \end{aligned}

The rule extends to any finite sum, $E(X_1 + \dots + X_n) = E(X_1) + \dots + E(X_n)$ , and it holds whether or not the variables are independent. It applies to linear combinations. For a function $g$ that is not linear, $E(g(X))$ need not equal $g(E(X))$ .

In plain words The mean of a total is the total of the means, whatever the relation between the parts, as long as each part has a finite mean. Doubling a quantity doubles its mean, and adding a constant adds the same constant to the mean.

Example Let $Y$ have the same distribution as the variable $X$ above. Then $E(X + Y) = \tfrac12 + \tfrac12 = 1$ , whether $Y$ is chosen independently of $X$ or $Y = X$ on every trial. For $U = 2X + 1$ , linearity gives $E(U) = 2 \cdot \tfrac12 + 1 = 2$ . Squaring is not linear, and indeed $E(X^2) = 1$ while $\bigl(E(X)\bigr)^2 = \tfrac14$ .

Independent random variables

Definition Discrete random variables $X$ and $Y$ are independent if

\begin{aligned} &P(X = x, Y = y) \\[0.4em] &\quad = P(X = x)\,P(Y = y) \end{aligned}

for every pair of values $x$ and $y$ . Equivalently, for every $x$ with $P(X = x) > 0$ , the conditional distribution of $Y$ given $X = x$ is the same as the distribution of $Y$ . Several variables $X_1, \dots, X_n$ are independent if their joint probabilities factor in the same way for every list of values. If $X$ and $Y$ are independent and have finite expected values, then $E(XY) = E(X)\,E(Y)$ . An independent copy of $X$ is a random variable that is independent of $X$ and has the same distribution.

In plain words Independent variables carry no information about each other, so learning the value of one leaves the probabilities for the other unchanged. Having the same distribution is a different property. A variable and itself have the same distribution, but they are not independent unless the variable is constant with probability $1$ .

Example Let $Y$ be an independent copy of $X$ . The four pairs of values have the probabilities in the table, each the product of the two separate probabilities.

Joint probability	$Y = -1$	$Y = 1$
$X = -1$	$1/16$	$3/16$
$X = 1$	$3/16$	$9/16$

If instead $Y = X$ , the pair $X = -1$ and $Y = 1$ is impossible. Its probability is $0$ , while $P(X = -1)\,P(Y = 1) = 3/16$ , so these two variables are not independent.

Covariance of random variables

Definition For random variables $X$ and $Y$ with finite second moments and means $\mu_X$ and $\mu_Y$ , the covariance is

\begin{aligned} &\operatorname{Cov}(X, Y) \\[0.4em] &\quad = E\bigl((X - \mu_X)(Y - \mu_Y)\bigr) \\[0.4em] &\quad = E(XY) - \mu_X \mu_Y . \end{aligned}

In particular $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$ . Independent variables with finite second moments have covariance $0$ , because $E(XY) = E(X)\,E(Y)$ . The converse is false. Covariance $0$ does not imply independence, as the example below shows. When both standard deviations are positive, the correlation

\begin{aligned} &\operatorname{Corr}(X, Y) \\[0.4em] &\quad = \frac{\operatorname{Cov}(X, Y)}{\operatorname{SD}(X)\operatorname{SD}(Y)} \end{aligned}

lies between $-1$ and $1$ . These are quantities of the model. The sample covariance $s_{xy}$ and the sample correlation $r$ are their counterparts computed from observed data.

In plain words Covariance averages the product of the two deviations from the means, weighting by probability. Deviations with the same sign give a positive product and deviations with opposite signs a negative one, so the covariance is positive when the same-sign products outweigh the others, negative when they are outweighed, and $0$ when the two balance. Both the sizes of the deviations and how often each case occurs count. It reflects linear association, so even a strong curved relation can have covariance $0$ .

Example For $Y = X$ with $X$ as above, $\operatorname{Cov}(X, X) = \operatorname{Var}(X) = 3/4$ and the correlation is $1$ . For an independent copy, the table gives $E(XY) = \tfrac{1}{16} - \tfrac{3}{16} - \tfrac{3}{16} + \tfrac{9}{16} = \tfrac14 = E(X)\,E(Y)$ , so the covariance is $0$ . To see that covariance $0$ does not imply independence, let $V$ take the values $-1$ , $0$ and $1$ with probability $1/3$ each, and compare $V$ with its square $V^2$ . Here $E(V) = 0$ and $E(V \cdot V^2) = E(V^3) = 0$ , so $\operatorname{Cov}(V, V^2) = 0$ . Yet $V^2$ is determined by $V$ , and $P(V = 0, V^2 = 0) = 1/3$ differs from $P(V = 0)\,P(V^2 = 0) = 1/9$ , so the two are not independent.

Variance of a sum

Definition For random variables $X$ and $Y$ with finite second moments and constants $a$ and $b$ ,

\begin{aligned} &\operatorname{Var}(aX + bY) \\[0.4em] &\quad = a^2 \operatorname{Var}(X) \\[0.4em] &\qquad + b^2 \operatorname{Var}(Y) \\[0.4em] &\qquad + 2ab \operatorname{Cov}(X, Y). \end{aligned}

When $\operatorname{Cov}(X, Y) = 0$ , in particular when $X$ and $Y$ are independent, both $X + Y$ and $X - Y$ have variance $\operatorname{Var}(X) + \operatorname{Var}(Y)$ . For a sum of $n$ variables, the variance is the sum of the $n$ variances plus twice the sum of the covariances over all pairs. Standard deviations do not add in general. The equality $\operatorname{SD}(X + Y) = \operatorname{SD}(X) + \operatorname{SD}(Y)$ holds exactly when $\operatorname{Cov}(X, Y) = \operatorname{SD}(X)\operatorname{SD}(Y)$ , which for two variables with positive standard deviations means correlation $1$ .

In plain words Variances add for uncorrelated parts, and the covariance term corrects for parts that move together or against each other. For two independent parts that both vary, their deviations from the means have the same sign in some trials and opposite signs in others, so the standard deviation of the total is smaller than the sum of the two standard deviations.

Example For an independent copy $Y$ of $X$ , the sum $X + Y$ takes the values $-2$ , $0$ and $2$ with probabilities $1/16$ , $6/16$ and $9/16$ . Its mean is $1$ , and its variance is $3/4 + 3/4 = 3/2$ , which a direct computation from these three probabilities confirms. Its standard deviation is $\sqrt{3/2} \approx 1.225$ , less than $\operatorname{SD}(X) + \operatorname{SD}(Y) = \sqrt{3} \approx 1.732$ . If instead $Y = X$ , the sum is $2X$ , with the same mean $1$ and variance

\begin{aligned} \operatorname{Var}(2X) &= \tfrac34 + \tfrac34 + 2 \cdot \tfrac34 \\[0.4em] &= 3, \end{aligned}

twice as large. Its standard deviation $\sqrt{3}$ now equals the sum of the two standard deviations, because the correlation is $1$ .

Check yourself

For the variable $X$ of this lesson, which is $-1$ with probability $1/4$ and $1$ with probability $3/4$ , let $T = 3X - 2$ . Find $E(T)$ , $\operatorname{Var}(T)$ and $\operatorname{SD}(T)$ . Why does the $-2$ change the mean but not the spread?

Show the answer

By linearity, $E(T) = 3 \cdot \tfrac12 - 2 = -\tfrac12$ . The variance is multiplied by $3^2$ , so $\operatorname{Var}(T) = 9 \cdot \tfrac34 = \tfrac{27}{4} = 6.75$ , and the standard deviation is multiplied by $3$ , so $\operatorname{SD}(T) = 3\sqrt{3}/2 \approx 2.598$ . Subtracting $2$ moves every value and the mean by the same amount, so every deviation from the mean stays the same. Multiplying by $3$ triples every deviation, which multiplies every squared deviation by $9$ .

Let $Z$ be an independent copy of $X$ and $W = X - Z$ . Someone argues that $\operatorname{Var}(W) = \operatorname{Var}(X) - \operatorname{Var}(Z) = 0$ . Find $E(W)$ and $\operatorname{Var}(W)$ , and say when the variance of $X - Z$ would be $0$ .

Show the answer

By linearity, $E(W) = \tfrac12 - \tfrac12 = 0$ . The coefficient of $Z$ is $-1$ , and it enters the variance squared, so with covariance $0$

\begin{aligned} \operatorname{Var}(W) &= \tfrac34 + (-1)^2 \cdot \tfrac34 \\[0.4em] &= \tfrac32 . \end{aligned}

Directly, $W$ is $-2$ or $2$ with probability $3/16$ each and $0$ with probability $10/16$ , so $E(W^2) = 4 \cdot \tfrac{6}{16} = \tfrac32$ , which is $\operatorname{Var}(W)$ because $E(W) = 0$ . The argument fails because variances of independent parts add even in a difference. If instead $Z = X$ on every trial, then $W = 0$ always, and the formula gives $\tfrac34 + \tfrac34 - 2 \cdot \tfrac34 = 0$ , the covariance term canceling both variances. More generally, the variance of $X - Z$ is $0$ exactly when $X - Z$ is constant with probability $1$ , and for a $Z$ with the same distribution as $X$ that constant must be $0$ .

A product’s total length is $T = A + B + C$ , where each of the three parts has a standard deviation of $0.2\ \text{mm}$ . A colleague reports $\operatorname{SD}(T) = 0.6\ \text{mm}$ . Find $\operatorname{SD}(T)$ when the three lengths are independent, and say when the colleague’s value would be right.

Show the answer

For independent parts all covariances are $0$ , so

\begin{aligned} \operatorname{Var}(T) &= 3 \cdot (0.2\ \text{mm})^2 \\[0.4em] &= 0.12\ \text{mm}^2, \end{aligned}

and $\operatorname{SD}(T) = \sqrt{0.12}\ \text{mm} \approx 0.346\ \text{mm}$ . Adding the standard deviations gives $0.6\ \text{mm}$ , which is correct exactly when every pair of lengths has correlation $1$ , for example when the three parts always deviate from their means by the same amount. Then $\operatorname{Var}(T) = 9 \cdot 0.04\ \text{mm}^2 = 0.36\ \text{mm}^2$ and $\operatorname{SD}(T) = 0.6\ \text{mm}$ . The expected total length is the sum of the three expected lengths in either case.

Expected Value, Variance and Sums of Random Variables

Key terms

Expected value

Variance of a random variable

Linearity of expectation

Independent random variables

Covariance of random variables

Variance of a sum

Check yourself

How this lesson connects

Builds on

Used later in