Marginal Probability Distributions, introduction to probability lesson

A joint table for a parcel gives the probability of each combination of route and inspection. Often the question involves only one of them, such as how likely express routing is, whatever happens with inspection. Answering it means keeping the condition you care about and letting the other variable take every value it can. This lesson shows how to get the distribution of each variable from the joint table in this way, and why those separate distributions cannot in general be turned back into the joint table.

Key terms

Marginal distribution

Definition Let $X$ and $Y$ be random variables on the same sample space. The marginal distribution of $X$ is the distribution of $X$ by itself, given by the probabilities $P(X \in A)$ for sets $A$ of real numbers, with no condition on $Y$ . The marginal distribution of $Y$ is defined in the same way. The word marginal records only that the distribution was obtained from a joint law. It is the ordinary distribution of that variable, so its CDF, and its mean and variance when they exist, are found exactly as for any single random variable.

In plain words A marginal distribution answers questions about one variable and places no restriction on the other.

Example In a constructed model, $X = 1$ when a parcel takes the express route and $X = 0$ for the standard route, and $Y = 1$ when it gets an extra inspection and $Y = 0$ when it does not. The joint table is

	$Y = 0$	$Y = 1$
$X = 0$	$0.3$	$0.2$
$X = 1$	$0.1$	$0.4$

Express routing, with or without inspection, is the event $\{X = 1, Y = 0\} \cup \{X = 1, Y = 1\}$ , so $P(X = 1) = 0.1 + 0.4 = 0.5$ . In the same way $P(Y = 1) = 0.2 + 0.4 = 0.6$ . Both variables are indicators, so $E[X] = 0.5$ and $E[Y] = 0.6$ . The CDF of $Y$ is the step function

F_Y(t) = \begin{cases} 0, & t < 0, \\ 0.4, & 0 \le t < 1, \\ 1, & t \ge 1. \end{cases}

The mean $0.6$ is the probability that a parcel is inspected. It does not describe any single parcel, which is either inspected or not.

Marginal probability mass function

Definition Let $X$ and $Y$ be discrete random variables with joint PMF $p_{X,Y}$ . The marginal PMFs of $X$ and $Y$ are

\begin{aligned} p_X(x) &= \sum_{y} p_{X,Y}(x, y), \\[0.4em] p_Y(y) &= \sum_{x} p_{X,Y}(x, y) \end{aligned}

for all real $x$ and $y$ , where each sum runs over the possible values of the other variable. The first formula holds because the event $\{X = x\}$ is, apart from an event of probability $0$ , the union of the disjoint events $\{X = x, Y = y\}$ over the possible values $y$ of $Y$ , and the second holds for the same reason. Each marginal PMF is nonnegative and adds to $1$ , because adding all of its masses adds every joint mass exactly once.

In plain words In a joint table, the marginal PMF of the row variable is the list of row totals, and the marginal PMF of the column variable is the list of column totals. They are written in the margins of the table, which gives them their name.

Example The parcel table with its margins is

	$Y = 0$	$Y = 1$	$p_X(x)$
$X = 0$	$0.3$	$0.2$	$0.5$
$X = 1$	$0.1$	$0.4$	$0.5$
$p_Y(y)$	$0.4$	$0.6$	$1$

The row totals give $p_X(0) = p_X(1) = 0.5$ , and the column totals give $p_Y(0) = 0.4$ and $p_Y(1) = 0.6$ . Each margin adds to $1$ , and both agree with the total of the four cells, which is a quick check on the arithmetic.

Summing out a variable

Definition Summing out $Y$ means adding the joint PMF over the values of $Y$ , which removes $Y$ and leaves the marginal PMF of $X$ . For a fixed $x$ , the sum must include every value $y$ that can occur together with $x$ , that is, every $y$ with $p_{X,Y}(x, y) > 0$ . Values of $Y$ that cannot occur with $x$ contribute $0$ , so the sum may also run over all possible values of $Y$ , but no pair of positive mass may be left out. When the joint support is not a rectangle, the values of $Y$ compatible with $x$ change with $x$ , and the limits of the sum change with them. Summing out $X$ works the same way.

In plain words Hold fixed the value you are asking about, and let the other variable run through everything it can be alongside it. The variable you sum over disappears from the answer.

Example A constructed joint PMF puts mass $\tfrac{1}{3x}$ on each pair of integers with $1 \le y \le x \le 3$ , and mass $0$ on every other pair.

	$Y = 1$	$Y = 2$	$Y = 3$
$X = 1$	$\tfrac13$	$0$	$0$
$X = 2$	$\tfrac16$	$\tfrac16$	$0$
$X = 3$	$\tfrac19$	$\tfrac19$	$\tfrac19$

Row $x$ holds $x$ cells of mass $\tfrac{1}{3x}$ , so $p_X(x) = \tfrac13$ for $x = 1, 2, 3$ . For $Y$ , the compatible values of $x$ are those with $x \ge y$ , so

p_Y(y) = \sum_{x = y}^{3} \frac{1}{3x}.

This gives $p_Y(1) = \tfrac13 + \tfrac16 + \tfrac19 = \tfrac{11}{18}$ , $p_Y(2) = \tfrac16 + \tfrac19 = \tfrac{5}{18}$ and $p_Y(3) = \tfrac19 = \tfrac{2}{18}$ , which add to $1$ . Using the formula $\tfrac{1}{3x}$ for every $x$ from $1$ to $3$ , without the restriction $x \ge y$ , would add values of the formula at pairs that cannot occur and make each of the three values $\tfrac{11}{18}$ , with total $\tfrac{33}{18}$ .

Information lost by marginalizing

Definition The joint PMF determines both marginal PMFs, but the two marginal PMFs do not in general determine the joint PMF. Different joint PMFs can have the same marginals and still assign different probabilities to events that involve both variables. The product $p_X(x)\,p_Y(y)$ equals $p_{X,Y}(x, y)$ for every pair exactly when $X$ and $Y$ are independent, the subject of a later lesson, so without independence the margins cannot simply be multiplied. In special cases the margins do fix the joint PMF, for example when one of the two variables is constant.

In plain words The margins tell how each variable behaves by itself. They can limit how the values pair up, but in general they do not fix it. That pairing information is in the joint table.

Example The tables with rows $(0.3, 0.2), (0.1, 0.4)$ and with rows $(0.2, 0.3), (0.2, 0.3)$ both have row totals $0.5$ and $0.5$ and column totals $0.4$ and $0.6$ . They agree on the probability of express routing and on the probability of inspection, but the probability of both is $0.4$ in the first and $0.3$ in the second. In fact every number $t$ with $0.1 \le t \le 0.5$ is possible for that cell. The table

\begin{pmatrix} t - 0.1 & 0.6 - t \\ 0.5 - t & t \end{pmatrix},

with rows for $X = 0, 1$ and columns for $Y = 0, 1$ , has these margins for every $t$ , and its entries are nonnegative exactly when $0.1 \le t \le 0.5$ . Only $t = 0.3$ makes every cell the product of its margins, since $0.5 \times 0.6 = 0.3$ .

Check yourself

A joint table has the row $(0.1, 0.2, 0.2)$ for $X = 0$ and the row $(0.2, 0.1, 0.2)$ for $X = 1$ , with columns for $Y = 0$ , $1$ and $2$ . Check that it is a joint PMF, find both marginal PMFs, and find $P(Y \ge 1)$ and $E[Y]$ .

Show the answer

All six entries are nonnegative and add to $1$ . The row totals give $p_X(0) = p_X(1) = 0.5$ . The column totals give $p_Y(0) = 0.1 + 0.2 = 0.3$ , $p_Y(1) = 0.2 + 0.1 = 0.3$ and $p_Y(2) = 0.2 + 0.2 = 0.4$ , which add to $1$ . From the marginal of $Y$ , the probability $P(Y \ge 1)$ is $0.3 + 0.4 = 0.7$ , and

\begin{aligned} E[Y] &= 0 \cdot 0.3 + 1 \cdot 0.3 \\[0.4em] &\quad + 2 \cdot 0.4 = 1.1. \end{aligned}

Two labels are drawn without replacement from a bag with one blue label and two gold labels, with each remaining label equally likely at each draw. Let $X = 1$ if the first label is blue and $X = 0$ otherwise, and let $Y = 1$ if the second label is blue and $Y = 0$ otherwise. Build the joint table, find both marginal PMFs, and compare $P(X = 1, Y = 1)$ with $p_X(1)\,p_Y(1)$ .

Show the answer

The first label is blue with probability $\tfrac13$ , and then the second is gold, so $p_{X,Y}(1, 0) = \tfrac13$ and $p_{X,Y}(1, 1) = 0$ . The first label is gold with probability $\tfrac23$ , and then the second is blue with probability $\tfrac12$ , so $p_{X,Y}(0, 1) = \tfrac13$ and $p_{X,Y}(0, 0) = \tfrac13$ .

	$Y = 0$	$Y = 1$	$p_X(x)$
$X = 0$	$\tfrac13$	$\tfrac13$	$\tfrac23$
$X = 1$	$\tfrac13$	$0$	$\tfrac13$
$p_Y(y)$	$\tfrac23$	$\tfrac13$	$1$

Both draws have the same marginal PMF, with $p_X(1) = p_Y(1) = \tfrac13$ . The product $p_X(1)\,p_Y(1) = \tfrac19$ , but the joint mass is $0$ , because the bag holds only one blue label. The margins do not show this, and multiplying them assumes an independence that the two draws do not have.

Two codes $X$ and $Y$ , each $0$ or $1$ , have $P(X = 1) = 0.3$ and $P(Y = 1) = 0.6$ . Which values can $P(X = 1, Y = 1)$ take? Give the joint table at the smallest and at the largest possible value, and the table in which every joint mass is the product of its margins.

Show the answer

Write $t = P(X = 1, Y = 1)$ . The margins then force the other three cells,

\begin{aligned} p_{X,Y}(1, 0) &= 0.3 - t, \\[0.4em] p_{X,Y}(0, 1) &= 0.6 - t, \\[0.4em] p_{X,Y}(0, 0) &= 0.1 + t. \end{aligned}

All four cells are nonnegative exactly when $0 \le t \le 0.3$ , so the joint probability can be any number from $0$ to $0.3$ . At $t = 0$ the rows for $X = 0$ and $X = 1$ are $(0.1, 0.6)$ and $(0.3, 0)$ . At $t = 0.3$ they are $(0.4, 0.3)$ and $(0, 0.3)$ . The product table has $t = 0.3 \times 0.6 = 0.18$ , with rows $(0.28, 0.42)$ and $(0.12, 0.18)$ . The same margins fit infinitely many joint tables, so by themselves they restrict $P(X = 1, Y = 1)$ to this interval but do not determine its value.

Marginal Probability Distributions

Key terms

Marginal distribution

Marginal probability mass function

Summing out a variable

Information lost by marginalizing

Check yourself

How this lesson connects

Builds on

Used later in