Independent Random Variables

Definition Let $X$ and $Y$ be discrete random variables on the same probability model. They are independent if every event described by $X$ alone is independent of every event described by $Y$ alone, that is, if

for all sets $A$ and $B$ of real numbers, where the left side is the probability that both events occur. Events with probability $0$ satisfy this product rule automatically, so independence is equivalent to $P(Y \in B \mid X \in A) = P(Y \in B)$ for every set $B$ and every set $A$ with $P(X \in A) > 0$. Independence is a property of the joint distribution of the pair. In general it cannot be read from the two marginal distributions, and it does not follow from the variables having different names or different descriptions.

Definition Discrete random variables $X$ and $Y$ satisfy the product law when their joint PMF factors at every pair of real numbers $x$ and $y$,

Discrete $X$ and $Y$ are independent if and only if the product law holds. Taking $A = \{x\}$ and $B = \{y\}$ in the definition of independence gives the product law. Conversely, when the product law holds, the cells with $x$ in $A$ and $y$ in $B$ add up to $P(X \in A, Y \in B)$, and their products $p_X(x)\,p_Y(y)$ add up to $P(X \in A)\,P(Y \in B)$, so the two are equal. A pair in which $x$ or $y$ is not a possible value satisfies the law automatically, because both sides are $0$, so for a table it is enough to check every cell against the margins of that same table. A single cell where the law fails proves dependence. A cell where it holds does not by itself settle the other cells, except in a table with two rows and two columns, where the margins then force all four cells to agree. In particular, if $p_X(x) > 0$ and $p_Y(y) > 0$ but $p_{X,Y}(x, y) = 0$, the variables are dependent. The joint support of independent variables is therefore the full grid of pairs of possible values, but a full grid does not prove independence.

Definition The joint cumulative distribution function of random variables $X$ and $Y$ on the same probability model is

for all real numbers $a$ and $b$. It is defined for variables of every type, discrete or not. For discrete variables it is the sum of $p_{X,Y}(x, y)$ over the pairs with $x \le a$ and $y \le b$, and both thresholds are included. The marginal CDFs are recovered by letting one threshold grow, since $F_X(a) = \lim_{b \to \infty} F_{X,Y}(a, b)$. When $Y$ has a largest possible value, $F_{X,Y}(a, b) = F_X(a)$ already for every $b$ at least that large.

Definition Random variables $X$ and $Y$ of any type are independent if their joint CDF factors at every pair of thresholds,

for all real $a$ and $b$. This is the general definition of independence for random variables, and for discrete variables it is equivalent to the product law. A failure at one pair of thresholds proves dependence, and agreement at one pair is only one check. When $X$ and $Y$ take finitely many values, both sides stay constant as a threshold moves between consecutive possible values, and both are $0$ below the smallest ones, so it is enough to check thresholds at pairs of possible values. Pairs with $a$ at least the largest value of $X$, or $b$ at least the largest value of $Y$, always agree, because there one factor equals $1$ and the joint CDF reduces to the other marginal CDF.

Definition Let $X$ and $Y$ be independent discrete random variables, and let $g$ and $h$ be real functions defined at their possible values. Then $g(X)$ and $h(Y)$ are independent. The reason is that an event $\{g(X) \in C\}$ is the event $\{X \in A\}$ for the set $A$ of values $x$ with $g(x) \in C$, and likewise for $h(Y)$, so the product rule for $X$ and $Y$ carries over. The result needs one function of each variable. It does not make $X$ independent of $X^2$, and it does not make $X$ independent of $X + Y$, because each of these pairs shares the input $X$. Such pairs have to be checked directly.

Definition Discrete random variables $X_1, \dots, X_n$ on the same probability model are mutually independent if

for every choice of real numbers $x_1, \dots, x_n$. Summing out any of the variables shows that every subcollection then satisfies the same product law, so this single condition on the full joint PMF covers every group of the variables. Equivalently, $P(X_1 \in A_1, \dots, X_n \in A_n) = P(X_1 \in A_1) \cdots P(X_n \in A_n)$ for all sets $A_1, \dots, A_n$ of real numbers. The variables are pairwise independent if every two of them are independent. Mutual independence implies pairwise independence, but for three or more variables pairwise independence does not imply mutual independence. Functions of disjoint groups of mutually independent variables are independent, for example $X_1 + X_2$ and $X_3^2$ when $X_1$, $X_2$ and $X_3$ are mutually independent. A sequence of random variables is IID, independent and identically distributed, if all of them have the same distribution and every finite collection of them is mutually independent.