Continuous Expectation and Variance, introduction to probability lesson

A measurement $X$ described by a density on $[0, 1]$ gives probability $0$ to each single value, so its average cannot be found by listing values and their probabilities. Cut the range into narrow strips instead. A strip of width $\Delta x$ near $x$ has probability close to $f(x)\,\Delta x$ , and it adds about $x f(x)\,\Delta x$ to the weighted average, so letting the strips shrink replaces the sum by an integral. This lesson defines continuous expectation and variance in this way, extends them to functions of $X$ and to mixed distributions, and shows why a valid density need not have a finite mean.

Key terms

Expected value of a continuous variable

Definition Let $X$ have density $f$ . If

\int_{-\infty}^{\infty} \lvert x \rvert f(x)\,dx < \infty,

the expected value of $X$ is the finite number

E[X] = \int_{-\infty}^{\infty} x f(x)\,dx.

It is also called the mean, often written $\mu$ , and it has the units of $X$ . When $f$ is $0$ outside a bounded interval $[a, b]$ , the condition holds automatically and $a \le E[X] \le b$ . When $f(x) = 0$ for all $x < 0$ , the integral of $x f(x)$ is either a finite number or $+\infty$ , and in the second case we write $E[X] = +\infty$ .

In plain words The mean is the same weighted average as for a discrete variable, with density times a small width in the role of a mass. It is the balance point of the distribution, not a forecast of one observation, and it need not equal the median.

Example Let $X$ have density $f(x) = 2x$ for $0 \le x \le 1$ and $f(x) = 0$ otherwise. The total area is $\int_0^1 2x\,dx = 1$ , and

\begin{aligned} E[X] &= \int_0^1 x \cdot 2x\,dx \\[0.4em] &= \Bigl[\tfrac23 x^3\Bigr]_0^1 = \frac23. \end{aligned}

The density rises to the right, so the mean lies to the right of $\tfrac12$ . The CDF is $F(x) = x^2$ on $[0, 1]$ , so $P(X \le \tfrac23) = \tfrac49$ , and the median is $1/\sqrt{2} \approx 0.707$ , not the mean.

Law of the unconscious statistician

Definition Let $X$ have density $f$ , and let $g$ be a function such as a polynomial or a piecewise continuous function. If $\int_{-\infty}^{\infty} \lvert g(x) \rvert f(x)\,dx < \infty$ , then

E[g(X)] = \int_{-\infty}^{\infty} g(x) f(x)\,dx.

If $g \ge 0$ , the formula holds without that condition, with both sides possibly equal to $+\infty$ . In particular, the $k$ th moment $E[X^k]$ is the integral of $x^k f(x)$ , finite when the integral of $\lvert x \rvert^k f(x)$ is finite, and $E[X^2]$ is the second moment. The rule is called the law of the unconscious statistician, or LOTUS, because it is easy to use without noticing that it needs proof.

In plain words To average a function of $X$ , weight the values of the function by the original density. There is no need to find the distribution of $g(X)$ first. The function is applied inside the integral, not to the answer, so $E[g(X)]$ is usually different from $g(E[X])$ .

Example For the density $2x$ on $[0, 1]$ ,

\begin{aligned} E[X^2] &= \int_0^1 x^2 \cdot 2x\,dx \\[0.4em] &= \Bigl[\tfrac12 x^4\Bigr]_0^1 = \frac12, \end{aligned}

which is not $(E[X])^2 = \tfrac49$ . As a check, $Y = X^2$ has $P(Y \le y) = P(X \le \sqrt{y}) = y$ for $0 \le y \le 1$ , so $Y \sim \operatorname{Uniform}(0, 1)$ , and its mean is $\int_0^1 y\,dy = \tfrac12$ . LOTUS gave the same answer without that detour.

Variance of a continuous variable

Definition Let $X$ have density $f$ and finite mean $\mu$ . The variance of $X$ is

\begin{aligned} &\operatorname{Var}(X) = E\bigl[(X - \mu)^2\bigr] \\[0.4em] &\quad = \int_{-\infty}^{\infty} (x - \mu)^2 f(x)\,dx, \end{aligned}

a number in $[0, \infty)$ or $+\infty$ . It is finite exactly when $E[X^2] < \infty$ , and then

\operatorname{Var}(X) = E[X^2] - \mu^2.

The standard deviation is $\operatorname{SD}(X) = \sqrt{\operatorname{Var}(X)}$ . Variance has the squared units of $X$ , and the standard deviation has the units of $X$ . When $\operatorname{Var}(X)$ is finite, the rules $\operatorname{Var}(a + bX) = b^2 \operatorname{Var}(X)$ and $\operatorname{SD}(a + bX) = \lvert b \rvert \operatorname{SD}(X)$ hold for all constants $a$ and $b$ . A variable with a density and a finite mean has $\operatorname{Var}(X) > 0$ , because variance $0$ would force $P(X = \mu) = 1$ .

In plain words Variance is the average squared distance from the mean, weighted by the density. Squaring removes signs and stresses large distances, so variance is not the average absolute distance. The shortcut $E[X^2] - \mu^2$ is convenient when both moments are finite, and it shows that the average of the squares exceeds the square of the average by exactly the variance.

Example For the density $2x$ on $[0, 1]$ ,

\begin{aligned} \operatorname{Var}(X) &= \frac12 - \left(\frac23\right)^2 \\[0.4em] &= \frac12 - \frac49 = \frac{1}{18}, \end{aligned}

and $\operatorname{SD}(X) = \sqrt{1/18} \approx 0.236$ . The direct integral $\int_0^1 (x - \tfrac23)^2 \, 2x\,dx$ gives the same $\tfrac{1}{18}$ . If $X$ is measured in hours, the variance is in squared hours and the standard deviation in hours. The recorded quantity $Y = 10 + 6X$ has $E[Y] = 10 + 6 \cdot \tfrac23 = 14$ and $\operatorname{Var}(Y) = 36 \cdot \tfrac{1}{18} = 2$ , so $\operatorname{SD}(Y) = \sqrt{2} \approx 1.414$ . Adding $10$ moves the mean and leaves the spread alone.

Mean of a mixed distribution

Definition Let $X$ have a mixed distribution with atoms $a_i$ of masses $m_i$ and a continuous part $h \ge 0$ whose total area is $1 - M$ , where $M = \sum_i m_i$ . For a function $g$ such that $\sum_i \lvert g(a_i) \rvert m_i$ and $\int \lvert g(x) \rvert h(x)\,dx$ are finite,

\begin{aligned} &E[g(X)] = \sum_i g(a_i)\, m_i \\[0.4em] &\quad + \int_{-\infty}^{\infty} g(x) h(x)\,dx. \end{aligned}

In particular, $E[X]$ is $M$ times the mean of the discrete part plus $1 - M$ times the mean of the continuous part, a weighted average of the two conditional means. The variance is $E[X^2] - (E[X])^2$ with both moments found this way. It is at least the weighted average of the two conditional variances, and larger whenever the two conditional means differ.

In plain words Each piece contributes in proportion to its own weight. The atoms contribute their values times their masses, and the continuous part contributes an integral against $h$ , whose total area is less than $1$ . Leaving out an atom, or treating the continuous part as the whole distribution, describes a different model.

Example Let $T$ equal $0$ with probability $0.2$ and otherwise be uniform on $(0, 1)$ , so $h(t) = 0.8$ on $(0, 1)$ . The atom contributes $0 \cdot 0.2 = 0$ to each moment, so

\begin{gathered} E[T] = \int_0^1 t \cdot 0.8\,dt = \frac25, \\[0.4em] E[T^2] = \int_0^1 t^2 \cdot 0.8\,dt = \frac{4}{15}. \end{gathered}

Then $\operatorname{Var}(T) = \tfrac{4}{15} - \tfrac{4}{25} = \tfrac{8}{75} \approx 0.107$ . Ignoring the atom would give the mean $\tfrac12$ and the variance $\tfrac{1}{12} \approx 0.083$ of the continuous part alone. The atom pulls the mean down, and the gap between the conditional means $0$ and $\tfrac12$ adds spread.

Infinite mean

Definition Let $X$ have a density $f$ with $f(x) = 0$ for all $x < 0$ . If the integral of $x f(x)$ over $[0, \infty)$ diverges, then $X$ has infinite mean, written $E[X] = +\infty$ . The density still has total area $1$ . Normalization fixes the area under $f$ and says nothing about the integral of $x f(x)$ . An infinite mean requires $X$ to be unbounded, meaning $P(X > t) > 0$ for every $t$ .

In plain words Every single value of $X$ is finite, but large values keep so much probability that the weighted average grows without bound as more of the range is included. The mean is then not some very large number. No finite number serves as the balance point.

Example Let $f(x) = 1/x^2$ for $x \ge 1$ and $f(x) = 0$ for $x < 1$ . For $b > 1$ , $\int_1^b x^{-2}\,dx = 1 - 1/b$ , which tends to $1$ , so $f$ is a density. But

\begin{aligned} \int_1^b x \cdot \frac{1}{x^2}\,dx &= \int_1^b \frac{dx}{x} \\[0.4em] &= \ln b, \end{aligned}

which grows without bound. Without logarithms, on each interval from $2^k$ to $2^{k+1}$ the integrand $1/x$ is at least $1/2^{k+1}$ over a width $2^k$ , so each such interval adds at least $\tfrac12$ , and infinitely many halves have no finite sum. Hence $E[X] = +\infty$ , even though $P(X > t) = 1/t$ for $t \ge 1$ , which tends to $0$ , and the median is $2$ .

Moment existence

Definition Let $X$ have density $f$ . The $k$ th moment $E[X^k]$ exists as a finite number when

\int_{-\infty}^{\infty} \lvert x \rvert^k f(x)\,dx < \infty.

For $k = 1$ this condition is called absolute integrability, and it is exactly what makes $E[X]$ a finite number. If the positive part $\int_0^\infty x f(x)\,dx$ and the negative part $\int_{-\infty}^0 \lvert x \rvert f(x)\,dx$ are both infinite, $E[X]$ is undefined. If exactly one of them is infinite, $E[X]$ is $+\infty$ or $-\infty$ . A finite second moment gives a finite mean, because $\lvert x \rvert \le 1 + x^2$ , and the variance is finite exactly when the second moment is. A density that is $0$ outside a bounded interval has finite moments of every order.

In plain words Before using a moment, check that the integral of its absolute size is finite. Symmetry alone does not supply a mean. When both tails carry infinite weight, canceling them against each other over $[-R, R]$ produces a number that depends on how the limits are taken, so it is not an expectation.

Example The density $f(x) = 1/(2x^2)$ for $\lvert x \rvert \ge 1$ , and $0$ for $\lvert x \rvert < 1$ , has total area $1$ and is symmetric about $0$ . Every symmetric integral $\int_{-R}^{R} x f(x)\,dx$ is $0$ , but

\begin{aligned} \int_{-\infty}^{\infty} \lvert x \rvert f(x)\,dx &= \int_1^\infty \frac{dx}{x} \\[0.4em] &= \infty, \end{aligned}

and integrating over $[-R, 2R]$ instead gives $\tfrac12 \ln 2$ for every $R \ge 1$ . So $E[X]$ is undefined, although $0$ is the center of symmetry and a median. For a finite mean with infinite variance, take $f(x) = 2/x^3$ for $x \ge 1$ and $0$ otherwise. Its total area is $1$ , its mean is $\int_1^\infty 2x^{-2}\,dx = 2$ , and its second moment is $\int_1^\infty 2x^{-1}\,dx = +\infty$ , so $\operatorname{Var}(X) = +\infty$ .

Check yourself

Let $X$ have density $f(x) = 3x^2$ for $0 \le x \le 1$ and $0$ otherwise. Find $E[X]$ , $E[X^2]$ , $\operatorname{Var}(X)$ and $\operatorname{SD}(X)$ .

Show the answer

The total area is $\int_0^1 3x^2\,dx = 1$ , and the support is bounded, so every moment is finite. By LOTUS,

\begin{gathered} E[X] = \int_0^1 3x^3\,dx = \frac34, \\[0.4em] E[X^2] = \int_0^1 3x^4\,dx = \frac35. \end{gathered}

With the common denominator $80$ ,

\begin{aligned} \operatorname{Var}(X) &= \frac35 - \frac{9}{16} \\[0.4em] &= \frac{48}{80} - \frac{45}{80} = \frac{3}{80}, \end{aligned}

and $\operatorname{SD}(X) = \sqrt{3/80} \approx 0.194$ .

For the same $X$ , find the mean, variance and standard deviation of $Y = 5 - 2X$ . A classmate writes $\operatorname{Var}(Y) = -2 \operatorname{Var}(X)$ . What went wrong?

Show the answer

By linearity, $E[Y] = 5 - 2 \cdot \tfrac34 = \tfrac72$ . Adding $5$ does not change the spread, and multiplying by $-2$ multiplies the variance by $(-2)^2 = 4$ , so

\operatorname{Var}(Y) = 4 \cdot \frac{3}{80} = \frac{3}{20},

and $\operatorname{SD}(Y) = \lvert -2 \rvert \operatorname{SD}(X) = \sqrt{3/20} \approx 0.387$ . The classmate’s value $-2 \cdot \tfrac{3}{80} = -\tfrac{3}{40}$ is negative, which no average of squared distances can be. The factor $-2$ reverses the direction of the scale, but distances from the mean are doubled and their squares are multiplied by $4$ .

Let $X$ have density $f(x) = 3x^{-4}$ for $x \ge 1$ and $0$ for $x < 1$ . Check that $f$ is a density, find $E[X]$ and $\operatorname{Var}(X)$ , and decide whether $E[X^3]$ is finite.

Show the answer

The function is nonnegative, and for $b > 1$ , $\int_1^b 3x^{-4}\,dx = 1 - b^{-3}$ , which tends to $1$ . So $f$ is a density. Since $X \ge 1$ , every moment integral has a nonnegative integrand and is either finite or $+\infty$ . Here

\begin{gathered} E[X] = \int_1^\infty 3x^{-3}\,dx = \frac32, \\[0.4em] E[X^2] = \int_1^\infty 3x^{-2}\,dx = 3, \end{gathered}

so $\operatorname{Var}(X) = 3 - \tfrac94 = \tfrac34$ . The third moment is $\int_1^\infty 3x^{-1}\,dx$ , and since $\int_1^b 3x^{-1}\,dx = 3 \ln b$ grows without bound, $E[X^3] = +\infty$ . This variable has a finite mean and a finite variance but an infinite third moment, so finite lower moments do not guarantee a finite higher one.

Continuous Expectation and Variance

Key terms

Expected value of a continuous variable

Law of the unconscious statistician

Variance of a continuous variable

Mean of a mixed distribution

Infinite mean

Moment existence

Check yourself

How this lesson connects

Builds on

Used later in