Conditional Expectation and the Tower Rule

Definition Let $X$ and $Y$ be discrete random variables on the same probability model, and let $x$ be a value with $p_X(x) > 0$. The conditional expectation of $Y$ given $X = x$ is the mean of the conditional PMF $p_{Y \mid X}(y \mid x) = p_{X,Y}(x, y)/p_X(x)$,

provided the sum converges absolutely, which is the case when $E[\lvert Y \rvert] < \infty$, in particular when $Y$ has finitely many values. It is defined only for values with $p_X(x) > 0$, because the conditional PMF divides by $p_X(x)$. In the same way, for an event $B$ with $P(B) > 0$, the conditional expectation $E[Y \mid B]$ is the mean of $Y$ computed with the conditional probabilities $P(Y = y \mid B)$.

Definition The conditional mean function of $Y$ given $X$ is

defined for every $x$ with $p_X(x) > 0$ at which this conditional expectation exists, which is every such $x$ when $E[\lvert Y \rvert] < \infty$, in particular when $Y$ has finitely many values. Each $m(x)$ is a fixed number. When $Y$ is the indicator of an event, $m(x) = P(Y = 1 \mid X = x)$ is the conditional probability of that event. When $Y$ is measured in a unit, such as minutes, $m(x)$ is measured in the same unit. A value $x$ with $p_X(x) = 0$ gets no conditional mean from the table, since its row and its row total are both $0$.

Definition Putting the random variable $X$ into the conditional mean function gives the random variable

On an outcome where $X$ takes the value $x$, it takes the value $m(x)$. When $m(x)$ exists at every $x$ with $p_X(x) > 0$, for example when $E[\lvert Y \rvert] < \infty$, it is defined with probability $1$, because a discrete $X$ takes such a value with probability $1$. It is a function of $X$, so its value is known as soon as $X$ is observed, and its distribution comes from that of $X$. For each number $v$, the probability $P(m(X) = v)$ is the sum of $p_X(x)$ over the values $x$ with $m(x) = v$.

Definition Let $X$ and $Y$ be discrete random variables on the same probability model with $E[\lvert Y \rvert] < \infty$. Then

or, written out,

where $m(x) = E[Y \mid X = x]$ and the sum runs over the values $x$ with $p_X(x) > 0$. This is also called the tower rule. For a finite table the proof is a regrouping of one double sum. Each conditional probability is a cell divided by its row total, so the row total cancels,

Every cell contributes its value of $y$ times its probability exactly once, whether the sum is taken row by row or column by column. The same rule holds for groups $A_1, \dots, A_k$ that partition the sample space and have positive probabilities, as $E[Y] = \sum_{i} P(A_i)\, E[Y \mid A_i]$.

Definition A random variable $Y$ is integrable if $E[\lvert Y \rvert] < \infty$. For a discrete $Y$ this means $\sum_y \lvert y \rvert \, p_Y(y) < \infty$, and every variable with finitely many values is integrable. If $Y$ is integrable, then every conditional expectation $E[Y \mid X = x]$ with $p_X(x) > 0$ is finite, because $E[\lvert Y \rvert \mid X = x] \le E[\lvert Y \rvert]/p_X(x)$, and the double sum in the tower rule converges absolutely, so it may be regrouped by rows or by columns. Integrability is a sufficient condition for the law of total expectation. Without it, the conditional means can all exist while $E[Y]$ does not.

Definition Suppose $E[Y^2] < \infty$, and let $x$ be a value with $p_X(x) > 0$. For every real number $a$,

where $\operatorname{Var}(Y \mid X = x) = E\bigl[(Y - m(x))^2 \mid X = x\bigr]$ is the variance of the conditional distribution. The identity follows by writing $Y - a$ as $\bigl(Y - m(x)\bigr) + \bigl(m(x) - a\bigr)$ and expanding, and the cross term vanishes because $E[Y - m(x) \mid X = x] = 0$. So $a = m(x)$ is the unique number that minimizes the conditional expected squared error. Averaging over $x$ shows that for every function $g$ defined at the values $x$ with $p_X(x) > 0$,

with equality exactly when $g(x) = m(x)$ for every such $x$. The finite second moment makes the conditional variances and the expected squared error of $m(X)$ finite. The result is about squared error only. The conditional mean need not be the most likely value, and other losses have other best predictions, for example a median for absolute error.