Sums of Discrete Random Variables, introduction to probability lesson

Two tasks produce random counts $X$ and $Y$ , and the question is how their total $X + Y$ is distributed. The mean of the total is the sum of the two means whenever those are finite, but the distribution asks for more. It asks which pairs of counts produce each total and how likely those pairs are. This lesson builds the distribution of a sum from ordered pairs, writes the collecting step as one formula, and shows exactly where independence enters, first in the pair probabilities and then in the variance.

Key terms

Ordered-pair table

Definition Let $X$ and $Y$ be discrete random variables on the same sample space, each with finitely many possible values. Their ordered-pair table has one row for each possible value $x$ of $X$ and one column for each possible value $y$ of $Y$ . The cell in row $x$ and column $y$ stands for the event $\{X = x, Y = y\}$ and holds its probability $P(X = x, Y = y)$ , together with the total $x + y$ . Different cells are disjoint events, and their probabilities add to $1$ . If $X$ and $Y$ are independent, every cell probability is the product $p_X(x)\,p_Y(y)$ .

In plain words The table records which task produced which count before anything is added. The pair with $X = 0$ and $Y = 2$ and the pair with $X = 2$ and $Y = 0$ give the same total, but they are different outcomes and can have different probabilities.

Example In a constructed model, let $X$ equal $0$ , $1$ or $2$ with probabilities $\tfrac14$ , $\tfrac12$ and $\tfrac14$ , and let $Y$ equal $0$ or $2$ with probabilities $\tfrac34$ and $\tfrac14$ , independently of $X$ . Each cell below shows its probability and its total.

	$y = 0$	$y = 2$
$x = 0$	$\tfrac{3}{16}$ , total $0$	$\tfrac{1}{16}$ , total $2$
$x = 1$	$\tfrac{6}{16}$ , total $1$	$\tfrac{2}{16}$ , total $3$
$x = 2$	$\tfrac{3}{16}$ , total $2$	$\tfrac{1}{16}$ , total $4$

For instance, the top left cell has probability $\tfrac14 \cdot \tfrac34 = \tfrac{3}{16}$ . The six cell probabilities add to $\tfrac{16}{16} = 1$ .

Distribution of a sum

Definition Let $X$ and $Y$ be discrete random variables on the same sample space. Their sum $S = X + Y$ is a discrete random variable, and its PMF is

\begin{aligned} p_S(s) &= P(X + Y = s) \\[0.4em] &= \sum_{x + y = s} P(X = x, Y = y), \end{aligned}

where the sum runs over all pairs $(x, y)$ with $x$ a possible value of $X$ , $y$ a possible value of $Y$ and $x + y = s$ . The events $\{X = x, Y = y\}$ for different pairs are disjoint, so their probabilities add. The formula uses the joint probabilities of the pairs. In general the two separate distributions of $X$ and $Y$ do not determine $p_S$ , and more must be known about how the variables are related, for example that they are independent.

In plain words Group the cells of the table by their total and add the probabilities inside each group. Counting the cells in a group instead of adding their probabilities works when all cells are equally likely, and it can mislead otherwise.

Example In the table above, the total $2$ comes from the cells $(0, 2)$ and $(2, 0)$ , so $P(S = 2) = \tfrac{1}{16} + \tfrac{3}{16} = \tfrac14$ . Collecting every total gives the PMF of $S$ .

$s$	$0$	$1$	$2$	$3$	$4$
$p_S(s)$	$\tfrac{3}{16}$	$\tfrac{6}{16}$	$\tfrac{4}{16}$	$\tfrac{2}{16}$	$\tfrac{1}{16}$

The numerators add to $16$ , as they must when every cell is used exactly once. Two cells give the total $2$ , but the total $1$ is more likely, since its one cell has probability $\tfrac{6}{16}$ .

Discrete convolution

Definition If $X$ and $Y$ are independent discrete random variables, the PMF of $S = X + Y$ is

p_S(s) = \sum_{x} p_X(x)\, p_Y(s - x),

where $x$ runs over the possible values of $X$ , and $p_Y(s - x) = 0$ whenever $s - x$ is not a possible value of $Y$ . This operation on two PMFs is called discrete convolution, and $p_S$ is the convolution of $p_X$ and $p_Y$ . Once $x$ and the total $s$ are fixed, the partner value $y = s - x$ is forced, so each compatible pair is counted exactly once.

In plain words For each possible value of the first variable, find the value of the second that completes the total, multiply the two probabilities and add over the first variable. It is the table calculation written as one formula, and the product inside it relies on independence.

Example For the variables of the table, $Y$ takes only the values $0$ and $2$ , so

\begin{aligned} p_S(2) &= p_X(0)\,p_Y(2) \\[0.4em] &\quad + p_X(1)\,p_Y(1) \\[0.4em] &\quad + p_X(2)\,p_Y(0) \\[0.4em] &= \tfrac{1}{16} + 0 + \tfrac{3}{16} = \tfrac14 . \end{aligned}

The middle term is $0$ because $Y$ never equals $1$ , and the result matches the two cells collected from the table.

Variance of an independent sum

Definition Let $X$ and $Y$ be random variables on the same sample space with finite variances. Then $E[X + Y] = E[X] + E[Y]$ , with or without independence. If in addition $X$ and $Y$ are independent, then

\begin{aligned} \operatorname{Var}(X + Y) &= \operatorname{Var}(X) \\[0.4em] &\quad + \operatorname{Var}(Y). \end{aligned}

For independent $X$ and $Y$ the standard deviation of the sum is therefore $\operatorname{SD}(X + Y) = \sqrt{\operatorname{Var}(X) + \operatorname{Var}(Y)}$ , which is at most $\operatorname{SD}(X) + \operatorname{SD}(Y)$ and is strictly smaller unless one of the two variances is $0$ .

In plain words Means add for any two variables with finite means. Variances add for independent variables, because for them the cross term in the next card averages to zero. To combine standard deviations, add the variances first and take the square root at the end.

Example For the independent variables of the table, $E[X] = 1$ and $E[Y] = 0 \cdot \tfrac34 + 2 \cdot \tfrac14 = \tfrac12$ , so $E[S] = \tfrac32$ . The deviations of $X$ are $-1$ , $0$ and $1$ , so $\operatorname{Var}(X) = \tfrac14 + 0 + \tfrac14 = \tfrac12$ , and $\operatorname{Var}(Y) = 4 \cdot \tfrac14 - \big(\tfrac12\big)^2 = \tfrac34$ , so the rule gives $\operatorname{Var}(S) = \tfrac54$ . The PMF of $S$ confirms this, since $E[S] = \tfrac{24}{16} = \tfrac32$ , $E[S^2] = \tfrac{56}{16} = \tfrac72$ and $\tfrac72 - \tfrac94 = \tfrac54$ . The standard deviation of $S$ is $\tfrac{\sqrt{5}}{2} \approx 1.118$ , smaller than $\operatorname{SD}(X) + \operatorname{SD}(Y) \approx 0.707 + 0.866 = 1.573$ .

Cross term

Definition Let $X$ and $Y$ have finite variances, and write $D_X = X - E[X]$ and $D_Y = Y - E[Y]$ for their deviations from their means. The deviation of the sum is $D_X + D_Y$ , and

\begin{aligned} (D_X + D_Y)^2 &= D_X^2 + D_Y^2 \\[0.4em] &\quad + 2D_X D_Y . \end{aligned}

Taking expectations gives

\begin{aligned} &\operatorname{Var}(X + Y) \\[0.4em] &\quad = \operatorname{Var}(X) + \operatorname{Var}(Y) \\[0.4em] &\qquad + 2E[D_X D_Y]. \end{aligned}

The term $2E[D_X D_Y]$ is the cross term. If $X$ and $Y$ are independent, then $E[D_X D_Y] = E[D_X]\,E[D_Y] = 0 \cdot 0 = 0$ , and the variances add. Without independence the cross term can be positive, negative or zero. The average product $E[D_X D_Y]$ is the covariance of $X$ and $Y$ , the subject of the lesson Covariance, Correlation and Sums.

In plain words The cross term is twice the average product of the two deviations. A product is positive when both variables fall on the same side of their means and negative when they fall on opposite sides. When the positive products outweigh the negative ones on average, the deviations reinforce each other and the sum spreads more than the two variances alone suggest. When the negative products outweigh the positive ones, the deviations partly cancel and the sum spreads less.

Example Keep $X$ as before, but now let $Y = 2$ exactly when $X = 2$ , and $Y = 0$ otherwise. Then $P(Y = 2) = \tfrac14$ and $P(Y = 0) = \tfrac34$ , the same distribution of $Y$ as before, but now $X$ and $Y$ are dependent. The sum equals $0$ , $1$ or $4$ with probabilities $\tfrac14$ , $\tfrac12$ and $\tfrac14$ . Its mean is still $\tfrac32$ , but $E[S^2] = \tfrac12 + 4 = \tfrac92$ , so $\operatorname{Var}(S) = \tfrac92 - \tfrac94 = \tfrac94$ instead of $\tfrac54$ . The difference $1$ is the cross term. Directly, with $D_X = X - 1$ and $D_Y = Y - \tfrac12$ , the product $D_X D_Y$ equals $\tfrac12$ when $X = 0$ , $0$ when $X = 1$ and $\tfrac32$ when $X = 2$ , so $E[D_X D_Y] = \tfrac18 + \tfrac38 = \tfrac12$ and the cross term is $2 \cdot \tfrac12 = 1$ . The two separate distributions are unchanged, yet the distribution and the variance of the sum are different.

Check yourself

Two independent selections are each uniform on $\{0, 1, 2\}$ , and $T$ is their sum. Find the PMF of $T$ , its mean and its variance.

Show the answer

The nine ordered pairs are equally likely, each with probability $\tfrac19$ , so here counting cells is valid. The totals $0$ , $1$ , $2$ , $3$ and $4$ come from $1$ , $2$ , $3$ , $2$ and $1$ pairs, so $T$ takes these values with probabilities $\tfrac19$ , $\tfrac29$ , $\tfrac39$ , $\tfrac29$ and $\tfrac19$ . The mean is $2$ by symmetry, or $1 + 1$ by linearity. One selection has second moment $\tfrac{0 + 1 + 4}{3} = \tfrac53$ and variance $\tfrac53 - 1 = \tfrac23$ , so independence gives $\operatorname{Var}(T) = \tfrac23 + \tfrac23 = \tfrac43$ . As a check from the PMF, $E[T^2] = \tfrac{0 + 2 + 12 + 18 + 16}{9} = \tfrac{16}{3}$ and $\tfrac{16}{3} - 2^2 = \tfrac43$ .

A count $X$ equals $0$ or $1$ with probabilities $0.7$ and $0.3$ , and an independent count $Y$ equals $0$ , $1$ or $2$ with probabilities $0.2$ , $0.5$ and $0.3$ . Use discrete convolution to find the PMF of $S = X + Y$ , say which total is most likely, and check that $\operatorname{Var}(S) = \operatorname{Var}(X) + \operatorname{Var}(Y)$ .

Show the answer

For each total, add $p_X(x)\,p_Y(s - x)$ over $x = 0$ and $x = 1$ . This gives $p_S(0) = 0.7 \cdot 0.2 = 0.14$ , $p_S(1) = 0.7 \cdot 0.5 + 0.3 \cdot 0.2 = 0.41$ , $p_S(2) = 0.7 \cdot 0.3 + 0.3 \cdot 0.5 = 0.36$ and $p_S(3) = 0.3 \cdot 0.3 = 0.09$ , which add to $1$ . The most likely total is $1$ . For the variances, $E[X] = E[X^2] = 0.3$ gives $\operatorname{Var}(X) = 0.3 - 0.3^2 = 0.21$ , and $E[Y] = 0.5 + 0.6 = 1.1$ with $E[Y^2] = 0.5 + 1.2 = 1.7$ gives $\operatorname{Var}(Y) = 1.7 - 1.1^2 = 0.49$ . From the PMF, $E[S] = 0.41 + 0.72 + 0.27 = 1.4$ , which equals $0.3 + 1.1$ , and $E[S^2] = 0.41 + 1.44 + 0.81 = 2.66$ , so $\operatorname{Var}(S) = 2.66 - 1.96 = 0.70 = 0.21 + 0.49$ .

Let $X$ equal $-1$ or $1$ with probability $\tfrac12$ each, so $\operatorname{Var}(X) = 1$ . In each case below $Y$ has the same distribution as $X$ . Find $\operatorname{Var}(X + Y)$ when $Y$ is independent of $X$ , when $Y = X$ , and when $Y = -X$ . What does the comparison show?

Show the answer

With independence, the variances add to $1 + 1 = 2$ . With $Y = X$ , the sum is $2X$ , whose variance is $2^2 \cdot 1 = 4$ , so the cross term is $2$ . With $Y = -X$ , the sum is $0$ with probability $1$ , whose variance is $0$ , so the cross term is $-2$ . In all three cases $X$ and $Y$ have the same separate distributions, yet the variance of the sum is $2$ , $4$ or $0$ . The variance of a sum depends on how the variables are related. The variances add exactly when the cross term is zero, and independence is one condition that guarantees this.

Sums of Discrete Random Variables

Key terms

Ordered-pair table

Distribution of a sum

Discrete convolution

Variance of an independent sum

Cross term

Check yourself

How this lesson connects

Builds on

Used later in