Linearity of Expectation, introduction to probability lesson

A repair visit has two parts, finding the fault and fixing it, and a quick diagnosis often comes with a long repair. To plan for the expected length of a visit, do you need the distribution of the total time, or a guarantee that the two parts are independent? You need neither. This lesson shows that the expectation of a sum is the sum of the expectations whenever those expectations are finite, whatever the dependence between the terms, and it marks where the shortcut stops, at products and squares.

Key terms

Finite expectation

Definition A random variable $X$ has finite expectation when $E[\lvert X \rvert] < \infty$ . For a discrete variable with PMF $p_X$ this means

\sum_{x} \lvert x \rvert \, p_X(x) < \infty,

and then $E[X] = \sum_x x \, p_X(x)$ is a real number that does not depend on the order in which the terms are added. A variable with finitely many possible values has finite expectation, and so does every bounded variable. When $E[\lvert X \rvert] = \infty$ , the expectation is $+\infty$ , $-\infty$ or undefined, and the rules on this page are stated only for finite expectations.

In plain words Before an expected value is used in arithmetic, check that it is an actual number. A valid distribution on infinitely many values can still have a weighted sum of values that grows without bound, and two infinite expectations of opposite sign cannot be combined into a meaningful answer.

Example Let $X$ take the value $n$ with probability $\frac{1}{n(n+1)}$ for $n = 1, 2, 3, \dots$ . These probabilities add to $1$ , because $\frac{1}{n(n+1)} = \frac{1}{n} - \frac{1}{n+1}$ and the sum telescopes. Yet

\begin{aligned} E[X] &= \sum_{n=1}^{\infty} \frac{n}{n(n+1)} \\[0.4em] &= \sum_{n=1}^{\infty} \frac{1}{n+1} = \infty . \end{aligned}

Now let $Y = -X$ . The sum $X + Y$ is the constant $0$ , so $E[X + Y] = 0$ , but the expression $E[X] + E[Y]$ would read $\infty + (-\infty)$ , which has no value.

Expectation of a sum

Definition Let $X$ and $Y$ be random variables defined on the same probability model, each with finite expectation. Then $X + Y$ has finite expectation and

E[X + Y] = E[X] + E[Y].

No independence is assumed. To see why in a model with finitely many outcomes $\omega_1, \dots, \omega_m$ , write $p_i = P(\{\omega_i\})$ , $x_i = X(\omega_i)$ and $y_i = Y(\omega_i)$ . An expectation can be computed outcome by outcome, since grouping the outcomes that give the same value returns the PMF formula. So

\begin{aligned} E[X + Y] &= \sum_{i} p_i (x_i + y_i) \\[0.4em] &= \sum_{i} p_i x_i + \sum_{i} p_i y_i \\[0.4em] &= E[X] + E[Y]. \end{aligned}

The same splitting works with countably many outcomes, because finite expectations make both series absolutely convergent, and the rule holds for random variables in general, discrete or not.

In plain words To average a total, average each part and add. Which values of $X$ occur together with which values of $Y$ matters for the distribution of $X + Y$ , but not for its mean.

Example Three equally likely job types give the task durations $(X, Y) = (1, 4)$ , $(2, 2)$ and $(5, 0)$ in hours. The separate means are $E[X] = \frac{1 + 2 + 5}{3} = \frac{8}{3}$ and $E[Y] = \frac{4 + 2 + 0}{3} = 2$ . The totals $X + Y$ are $5$ , $4$ and $5$ , so the direct calculation gives $E[X + Y] = \frac{14}{3}$ , which equals $\frac{8}{3} + 2$ . The durations are strongly dependent, since the value of $X$ identifies the job type and therefore fixes $Y$ , and the rule holds anyway.

Linearity of expectation

Definition A linear combination of random variables $X_1, \dots, X_n$ is a random variable $a_1X_1 + \cdots + a_nX_n$ , where the number of terms $n$ is fixed and the coefficients $a_1, \dots, a_n$ are fixed real numbers. Linearity of expectation states that if $X_1, \dots, X_n$ are defined on the same probability model and each has finite expectation, then for every fixed real number $c$ the variable $a_1X_1 + \cdots + a_nX_n + c$ has finite expectation and

\begin{aligned} &E\Bigl[\,\sum_{i=1}^{n} a_i X_i + c\Bigr] \\[0.4em] &\qquad = \sum_{i=1}^{n} a_i E[X_i] + c . \end{aligned}

It follows from the sum rule applied repeatedly, together with $E[aX] = aE[X]$ , since every value is multiplied by $a$ , and $E[c] = c$ , since the probabilities add to $1$ . Negative coefficients are allowed, so $E[X - Y] = E[X] - E[Y]$ . The number of terms has to be fixed. A sum with a random number of terms needs its own argument about how the count and the terms are related.

In plain words Expectation passes through sums, constant multiples and added constants. Combine the means in the same way the variables are combined, and ignore any dependence between them.

Example In the job table above, a fixed preparation time of half an hour adds $\frac12$ to every total, so the expected total becomes $\frac{14}{3} + \frac{1}{2} = \frac{31}{6}$ hours. A charge of $3$ tokens per hour of task $X$ , less a discount of $2$ tokens, is $C = 3X - 2$ , and linearity gives $E[C] = 3 \cdot \frac{8}{3} - 2 = 6$ tokens. Directly, the three charges are $1$ , $4$ and $13$ tokens, whose average is $\frac{18}{3} = 6$ .

Expectation of a product

Definition For random variables $X$ and $Y$ on the same probability model, $E[XY]$ is the expectation of the single random variable $XY$ , whose value on each outcome is the product of the values of $X$ and $Y$ . In a finite outcome table with the notation above,

E[XY] = \sum_{i} p_i \, x_i y_i ,

which depends on which values occur together. In general $E[XY]$ need not equal $E[X]E[Y]$ . If $X$ and $Y$ are independent and both have finite expectations, then $E[XY] = E[X]E[Y]$ , a separate product rule with its own hypothesis. The converse is false, so the equation $E[XY] = E[X]E[Y]$ does not show that $X$ and $Y$ are independent. Finite expectations of $X$ and $Y$ also do not guarantee a finite $E[XY]$ , for example when $Y = X$ and $E[X^2] = \infty$ .

In plain words Linearity covers adding and scaling, not multiplying. The average of a product depends on whether large values of one variable come with large or with small values of the other, and multiplying two separate averages throws that information away.

Example In the job table, the products $XY$ are $4$ , $4$ and $0$ square hours, so $E[XY] = \frac{8}{3}$ . The product of the means is $\frac{8}{3} \cdot 2 = \frac{16}{3}$ , twice as large. Long first tasks come with short second tasks here, and that pairing makes the average product smaller than the product of the averages.

Expectation of a square

Definition The expectation of the square of $X$ is $E[X^2]$ , the expectation of the random variable $X^2$ . For a discrete variable,

E[X^2] = \sum_x x^2 \, p_X(x),

a sum of nonnegative terms that is either a finite number or $+\infty$ . It should not be confused with the square of the expectation, $(E[X])^2$ . More generally, $E[g(X)]$ averages the values $g(x)$ , while $g(E[X])$ applies $g$ to a single number. The two agree for every $X$ with finite expectation when $g(x) = ax + b$ , by linearity, but they can differ for other functions, and $g(x) = x^2$ is the standard example. For $X$ with finite expectation, the difference $E[X^2] - (E[X])^2$ is the variance, the subject of a later lesson in this unit.

In plain words Read the brackets. Squaring each outcome and then averaging is a different instruction from averaging and then squaring. Squaring first lets outcomes far from zero count more heavily, and when $X$ has finite expectation the two results agree only if $X$ equals one value with probability one.

Example In the job table, $X^2$ takes the values $1$ , $4$ and $25$ , so $E[X^2] = \frac{30}{3} = 10$ . Squaring the mean gives $(\frac{8}{3})^2 = \frac{64}{9}$ , about $7.11$ . Both numbers are well defined, and they answer different questions.

Check yourself

Three scenarios have probabilities $0.2$ , $0.5$ and $0.3$ and give the paired durations $(A, B) = (0, 5)$ , $(4, 1)$ and $(6, 3)$ in hours. Find $E[2A + B]$ with linearity, then check it directly from the rows.

Show the answer

The separate means are $E[A] = 0 \cdot 0.2 + 4 \cdot 0.5 + 6 \cdot 0.3 = 3.8$ hours and $E[B] = 5 \cdot 0.2 + 1 \cdot 0.5 + 3 \cdot 0.3 = 2.4$ hours. Linearity gives $E[2A + B] = 2 \cdot 3.8 + 2.4 = 10$ hours. Directly, the rows give $2A + B = 5$ , $9$ and $15$ , and the weighted average is $5 \cdot 0.2 + 9 \cdot 0.5 + 15 \cdot 0.3 = 1 + 4.5 + 4.5 = 10$ hours. The two durations are not assumed independent, and neither calculation needs that.

A fair die is rolled. Let $X$ be the number shown and $Y = 7 - X$ the number on the opposite face. Find $E[X + Y]$ , $E[XY]$ and $E[X]E[Y]$ .

Show the answer

The sum $X + Y$ equals $7$ on every outcome, so $E[X + Y] = 7$ , in agreement with $E[X] + E[Y] = \frac72 + \frac72$ . For the product, write $XY = 7X - X^2$ . The second moment is $E[X^2] = \frac{1 + 4 + 9 + 16 + 25 + 36}{6} = \frac{91}{6}$ , so linearity gives

\begin{aligned} E[XY] &= 7E[X] - E[X^2] \\[0.4em] &= \frac{49}{2} - \frac{91}{6} = \frac{28}{3}. \end{aligned}

Directly, the six products are $6$ , $10$ , $12$ , $12$ , $10$ and $6$ , with average $\frac{56}{6} = \frac{28}{3}$ . The product of the means is $(\frac72)^2 = \frac{49}{4} = 12.25$ , larger than $\frac{28}{3} \approx 9.33$ . Linearity applies to the sum and to $7X - X^2$ , but not to the product $X \cdot Y$ itself.

Let $X$ take the values $-1$ , $0$ and $1$ with probability $\frac13$ each, and let $Y = X^2$ . Show that $E[XY] = E[X]E[Y]$ . Are $X$ and $Y$ independent?

Show the answer

The product $XY = X^3$ equals $X$ on every outcome, because each possible value satisfies $x^3 = x$ . So $E[XY] = E[X] = 0$ , and also $E[X]E[Y] = 0 \cdot \frac23 = 0$ . The two sides agree. The variables are still dependent, since $Y$ is determined by $X$ and is not constant. For instance, $P(X = 0, Y = 0) = \frac13$ , while $P(X = 0)P(Y = 0) = \frac13 \cdot \frac13 = \frac19$ . Independence implies the product rule, but the product rule alone does not imply independence.

Linearity of Expectation

Key terms

Finite expectation

Expectation of a sum

Linearity of expectation

Expectation of a product

Expectation of a square

Check yourself

How this lesson connects

Builds on

Used later in