Probability Densities and CDFs, introduction to probability lesson

A constructed model for a waiting time, measured in hours, uses a density curve that reaches the height $2$ . Does that give some waiting time a probability of $2$ , which no probability can be? It does not, because a density height is not a probability at all. Keeping three things apart settles the question, namely the possible values, the density at those values, and the probability of an interval. This lesson states exactly what a density must satisfy, what its height and units mean, how to write a complete CDF and read interval probabilities from it, and why each single value has probability zero.

Key terms

Probability density function

Definition A probability density function, or density, is a function $f$ defined for every real number $x$ with $f(x) \ge 0$ for every $x$ and

\int_{-\infty}^{\infty} f(x)\,dx = 1.

A random variable $X$ has density $f$ if, for all real numbers $a < b$ ,

\begin{aligned} &P(a < X \le b) \\[0.4em] &\quad = \int_a^b f(x)\,dx. \end{aligned}

The distribution of $X$ is then called absolutely continuous. Every absolutely continuous distribution has a continuous CDF, but a continuous CDF alone does not guarantee a density. In the models of this lesson, the probability is spread over an interval, called the support, and $f$ is $0$ outside it. A density of a distribution is not unique, since changing $f$ at finitely many points, to other nonnegative values, changes no integral and so gives another density of the same distribution.

In plain words A density describes a distribution by how thickly probability is spread along the axis. The probability of an interval is the area under the curve above it, and the heights of the curve are rates, not probabilities.

Example Let the waiting time $X$ , in hours, have density

f(x) = \begin{cases} 2x, & 0 \le x \le 1, \\ 0, & \text{otherwise}. \end{cases}

The support is $[0, 1]$ , the function is nonnegative, and the region under it is a triangle with base $1$ and height $2$ , so its area is $1$ . Writing only $f(x) = 2x$ would not define a distribution, because that formula is negative for every $x < 0$ .

Normalization of a density

Definition A function $f$ is a density exactly when it passes two separate checks. It must be nonnegative, $f(x) \ge 0$ for every $x$ , and it must be normalized, with total integral $1$ . If $g$ is a nonnegative function whose total area $A$ is finite and positive, then $f = g/A$ is a density, and finding this $f$ is called normalizing $g$ . In particular, a density of the form $cg(x)$ requires $c = 1/A$ .

In plain words The total probability must be $1$ , so the heights of a density are scaled to fit the width of its support. Spreading the same total over a wider interval calls for a lower average height.

Example Suppose $f(x) = cx$ for $0 \le x \le 2$ and $f(x) = 0$ otherwise. Normalization requires

\int_0^2 cx\,dx = c \cdot \frac{2^2}{2} = 2c = 1,

so $c = \tfrac12$ and $f(x) = x/2$ on $[0, 2]$ , which is nonnegative. The constant height $2$ on $[0, 2]$ is nonnegative but encloses area $4$ , so it is not a density, while the constant height $\tfrac12$ on $[0, 2]$ is one. In the other direction, $4x - 1$ on $[0, 1]$ has total area $2 - 1 = 1$ but is negative for $0 \le x < \tfrac14$ , and it would give the interval $[0, \tfrac14]$ the impossible value

\int_0^{1/4} (4x - 1)\,dx = -\tfrac18.

Density units

Definition If $X$ is measured in some unit, then a density value $f(x)$ is measured in probability per unit of $X$ , the reciprocal unit, such as per hour. A probability comes from a density only as an area, where heights are multiplied by widths and added up. A density value can be any nonnegative number, including numbers larger than $1$ . If the variable is rescaled to $Y = cX$ with a constant $c > 0$ , as in a change of units, then $P(Y \le y) = P(X \le y/c)$ , and the substitution $x = t/c$ in the integral of $f_X$ up to $y/c$ shows that $Y$ has density

f_Y(y) = \frac{1}{c}\, f_X\!\left(\frac{y}{c}\right).

In plain words A density height says how much probability there is per unit of width. Measuring the same quantity in a smaller unit spreads the same probability over more units, so every height is divided by the conversion factor, while every probability stays the same.

Example For the waiting time $X$ with density $2x$ on $[0, 1]$ , the height at $1$ hour is $2$ per hour. The same wait in minutes is $M = 60X$ , with density

\begin{aligned} f_M(m) &= \frac{1}{60} \cdot 2 \cdot \frac{m}{60} \\[0.4em] &= \frac{m}{1800} \end{aligned}

for $0 \le m \le 60$ , so the height at $60$ minutes is $\tfrac{1}{30}$ per minute. The event of waiting at most half an hour has the same probability in both units, $P(X \le \tfrac12) = \tfrac14$ and $P(M \le 30) = 30^2/3600 = \tfrac14$ . In the same way, the density $x/2$ on $[0, 2]$ from the previous card is the density of $2X$ .

CDF of a continuous variable

Definition If $X$ has density $f$ , its CDF is

\begin{aligned} F(x) &= P(X \le x) \\[0.4em] &= \int_{-\infty}^{x} f(t)\,dt \end{aligned}

for every real number $x$ . This $F$ is continuous everywhere and nondecreasing, it tends to $0$ as $x \to -\infty$ and to $1$ as $x \to \infty$ , and $F'(x) = f(x)$ at every $x$ where $f$ is continuous. A complete CDF gives a value at every real threshold. For a density that is $0$ outside $[c, d]$ , it equals $0$ for $x < c$ , equals $\int_c^x f(t)\,dt$ for $c \le x \le d$ , and equals $1$ for $x > d$ .

In plain words The CDF records the area to the left of each threshold. It is flat on every interval where the density is $0$ , rises over every interval where the density is positive, and never exceeds $1$ . The horizontal pieces outside the support are part of the answer, not decoration.

Example For the density $2x$ on $[0, 1]$ , integrating from $0$ to $x$ gives $x^2$ , so

F(x) = \begin{cases} 0, & x < 0, \\ x^2, & 0 \le x \le 1, \\ 1, & x > 1. \end{cases}

The formula $x^2$ on its own is not the CDF, since it exceeds $1$ for $x > 1$ and decreases for $x < 0$ , which no CDF can do. The pieces meet at $0$ and at $1$ , so $F$ has no jumps. For the density $x/2$ on $[0, 2]$ , the middle piece is $x^2/4$ for $0 \le x \le 2$ , with $0$ to its left and $1$ to its right.

Interval probability from a CDF

Definition For any random variable $X$ with CDF $F$ and real numbers $a < b$ ,

\begin{aligned} P(a < X \le b) &= F(b) - F(a), \\[0.4em] P(X > a) &= 1 - F(a). \end{aligned}

When $X$ has a density, $F$ is continuous, so including or excluding endpoints changes nothing. The probabilities $P(a \le X \le b)$ , $P(a < X < b)$ and $P(a \le X < b)$ also equal $F(b) - F(a)$ , and $P(X \ge a) = 1 - F(a)$ .

In plain words Take the probability accumulated up to the right end and remove what had accumulated by the left end. For an upper tail, remove the accumulated probability from the total $1$ .

Example For the density $2x$ on $[0, 1]$ ,

\begin{aligned} &P(0.2 < X < 0.8) \\[0.4em] &\quad = F(0.8) - F(0.2) \\[0.4em] &\quad = 0.64 - 0.04 = 0.6, \end{aligned}

and $P(X > 0.8) = 1 - 0.64 = 0.36$ . For the density $x/2$ on $[0, 2]$ , the probability of a value above $1$ is $1 - F(1) = 1 - \tfrac14 = \tfrac34$ .

Zero probability at a point

Definition If $X$ has a density, then $P(X = c) = 0$ for every real number $c$ . Indeed, for every $h > 0$ ,

\begin{aligned} &P(X = c) \\[0.4em] &\quad \le P(c - h < X \le c) \\[0.4em] &\quad = F(c) - F(c - h), \end{aligned}

and the right side tends to $0$ as $h$ decreases to $0$ , because $F$ is continuous. Probability $0$ does not mean impossible. Every value of $X$ has probability $0$ , yet $X$ always takes some value. A random variable that gives some single value positive probability, such as any discrete random variable, has no density.

In plain words In a density model, probability lives on intervals, not on single points. Shrinking an interval around a point shrinks its area to $0$ , even where the curve is tall.

Example For the density $2x$ on $[0, 1]$ ,

\begin{aligned} &P(0.8 - h < X \le 0.8) \\[0.4em] &\quad = 0.64 - (0.8 - h)^2 \\[0.4em] &\quad = 1.6h - h^2, \end{aligned}

which is $0.0159$ for $h = 0.01$ and tends to $0$ . So $P(X = 0.8) = 0$ , even though $0.8$ lies inside the support and $f(0.8) = 1.6$ . This is why $P(0.2 \le X \le 0.8)$ is also $0.6$ . A fair die roll $D$ is different. It has $P(D = 3) = \tfrac16$ , a jump of $\tfrac16$ in its CDF at $3$ , and no density.

Check yourself

A random variable $X$ has density $f(x) = \tfrac13$ for $0 \le x \le 1$ , $f(x) = \tfrac23$ for $1 < x \le 2$ , and $f(x) = 0$ otherwise. Check that $f$ is a density, and find $P(0.5 < X < 1.5)$ from areas.

Show the answer

Both heights are nonnegative, and the two rectangles have areas $1 \cdot \tfrac13$ and $1 \cdot \tfrac23$ , which add to $1$ . The interval $(0.5, 1.5)$ covers half of each rectangle, so

\begin{aligned} &P(0.5 < X < 1.5) \\[0.4em] &\quad = 0.5 \cdot \tfrac13 + 0.5 \cdot \tfrac23 \\[0.4em] &\quad = \tfrac16 + \tfrac13 = \tfrac12. \end{aligned}

Write the complete CDF of the same $X$ , and use it to check the answer to question 1.

Show the answer

Below $0$ nothing has accumulated. On $[0, 1]$ the area is $x/3$ . Beyond $1$ the first rectangle contributes $\tfrac13$ and the second adds $\tfrac23(x - 1)$ , which together give $(2x - 1)/3$ . So

F(x) = \begin{cases} 0, & x < 0, \\ \frac{x}{3}, & 0 \le x < 1, \\ \frac{2x - 1}{3}, & 1 \le x < 2, \\ 1, & x \ge 2. \end{cases}

The piece $x/3$ is $0$ at $x = 0$ , both middle pieces give $\tfrac13$ at $x = 1$ , and the second of them gives $1$ at $x = 2$ , so $F$ is continuous even though the density jumps at $0$ , $1$ and $2$ . Then $F(1.5) - F(0.5) = \tfrac23 - \tfrac16 = \tfrac12$ , which matches.

A random variable $X$ has density $f(x) = c(2 - x)$ for $0 \le x \le 2$ and $f(x) = 0$ otherwise. Find $c$ , then find $P(X \le 1)$ and $P(X = 1)$ . A classmate answers $P(X = 1) = f(1)$ . What is wrong with that?

Show the answer

Normalization requires

\begin{aligned} \int_0^2 c(2 - x)\,dx &= c(4 - 2) \\[0.4em] &= 2c = 1, \end{aligned}

so $c = \tfrac12$ , and then $f(x) = 1 - x/2$ is nonnegative on $[0, 2]$ . The CDF on $[0, 2]$ is $F(x) = x - x^2/4$ , so $P(X \le 1) = 1 - \tfrac14 = \tfrac34$ . Because $X$ has a density, $P(X = 1) = 0$ . The classmate’s value $f(1) = \tfrac12$ is a probability per unit of $x$ , not a probability. It says that a short interval of width $h$ near $1$ has probability close to $\tfrac12 h$ .

Probability Densities and CDFs

Key terms

Probability density function

Normalization of a density

Density units

CDF of a continuous variable

Interval probability from a CDF

Zero probability at a point

Check yourself

How this lesson connects

Builds on

Used later in