Exponential Distributions, introduction to probability lesson

A uniform waiting time has a last possible moment, but many waits have no natural cutoff. A model for such a wait can make long waits unlikely without ruling out any finite wait. The exponential distribution is one of the simplest models of this kind, with one positive parameter called the rate. This lesson finds its tail probabilities, mean and quantiles, separates the rate from the scale, and explains the memoryless property together with the situations it does not fit.

Key terms

Exponential distribution

Definition Let $\lambda > 0$ . A random variable $T$ has the exponential distribution with rate $\lambda$ , written $T \sim \operatorname{Exp}(\lambda)$ , if it has the density

f(t) = \begin{cases} \lambda e^{-\lambda t}, & t \ge 0, \\ 0, & t < 0. \end{cases}

Here $e \approx 2.718$ is the base of the natural exponential function, and $e^{-\lambda t}$ falls from $1$ at $t = 0$ toward $0$ as $t$ grows. The support is $[0, \infty)$ , and the density has total area $1$ , since

\begin{aligned} &\int_0^{\infty} \lambda e^{-\lambda t}\,dt \\[0.4em] &\quad = \Bigl[-e^{-\lambda t}\Bigr]_0^{\infty} \\[0.4em] &\quad = 0 - (-1) = 1. \end{aligned}

In plain words An exponential wait is never negative, and every interval of positive length inside $[0, \infty)$ has positive probability, however far out it lies. The density is highest at $0$ and decreases, so of two intervals of the same positive length inside $[0, \infty)$ , the one closer to $0$ carries more probability. A density value is not a probability. Each single time, including $0$ , has probability $0$ , and the height $\lambda$ at $t = 0$ is larger than $1$ whenever $\lambda > 1$ .

Example A constructed model measures a waiting time $T$ in minutes with rate $\lambda = 0.5$ per minute, so $f(t) = 0.5\,e^{-0.5t}$ for $t \ge 0$ . The probability that the wait is over within $3$ minutes is

\begin{aligned} &P(T \le 3) \\[0.4em] &\quad = \int_0^{3} 0.5\,e^{-0.5t}\,dt \\[0.4em] &\quad = 1 - e^{-1.5} \approx 0.7769. \end{aligned}

A model with rate $3$ per minute has density $3$ per minute at $t = 0$ . That height is larger than $1$ and still valid, because only areas under the density are probabilities.

Rate and scale parameter

Definition The parameter $\lambda > 0$ of $\operatorname{Exp}(\lambda)$ is the rate. It is measured in reciprocal time units, such as per minute, so the exponent $\lambda t$ has no units. The scale parameter is $\theta = 1/\lambda$ , measured in time units, and in terms of it the density is $f(t) = \theta^{-1} e^{-t/\theta}$ for $t \ge 0$ . The mean and variance are

\begin{gathered} E[T] = \frac{1}{\lambda} = \theta, \\[0.4em] \operatorname{Var}(T) = \frac{1}{\lambda^{2}} = \theta^{2}, \end{gathered}

so $\operatorname{SD}(T) = 1/\lambda$ as well. For the mean, the product rule shows that $-(t + 1/\lambda)e^{-\lambda t}$ has derivative $\lambda t e^{-\lambda t}$ , and this expression tends to $0$ as $t \to \infty$ . Hence

\begin{aligned} E[T] &= \int_0^{\infty} t \lambda e^{-\lambda t}\,dt \\[0.4em] &= \Bigl[-\bigl(t + \tfrac{1}{\lambda}\bigr)e^{-\lambda t}\Bigr]_0^{\infty} \\[0.4em] &= 0 - \Bigl(-\frac{1}{\lambda}\Bigr) = \frac{1}{\lambda}. \end{aligned}

In the same way $-(t^{2} + 2t/\lambda + 2/\lambda^{2})e^{-\lambda t}$ is an antiderivative of $\lambda t^{2} e^{-\lambda t}$ , which gives $E[T^{2}] = 2/\lambda^{2}$ and $\operatorname{Var}(T) = 2/\lambda^{2} - 1/\lambda^{2} = 1/\lambda^{2}$ . A positive rate is required. With $\lambda = 0$ the formula gives the zero function, whose total area is $0$ , and a negative $\lambda$ would give negative density values.

In plain words The rate and the scale carry the same information in reciprocal form. A larger rate means shorter waits on average, and a larger scale means longer ones. The mean wait equals the scale, and so does the standard deviation. The name rate comes from counting models. If the number of events in the time interval $(0, t]$ is $\operatorname{Pois}(\lambda t)$ for every $t > 0$ , as in the constant-rate model of Poisson Distributions, then the first event comes after time $t$ exactly when no event falls in $(0, t]$ , which has probability $e^{-\lambda t}$ . So the wait for the first event is $\operatorname{Exp}(\lambda)$ , with the same rate as the count. The rate is an average, not a promise of a fixed number of events in every minute, and a single wait is not forced to equal the mean. Books and software differ on whether “exponential with parameter $2$ ” means a rate or a scale, so state which one is meant before substituting numbers.

Example For the rate $0.5$ per minute, the scale is $\theta = 2$ minutes. The mean wait is $2$ minutes, the variance is $4$ minutes squared and the standard deviation is $2$ minutes. Reading the same number $0.5$ as a scale describes a different model, with scale $0.5$ minutes, rate $2$ per minute and a mean wait of $0.5$ minutes, a quarter of the first mean.

Survival function

Definition The survival function of a random variable $T$ is $S(t) = P(T > t) = 1 - F(t)$ , where $F$ is the CDF of $T$ . For $T \sim \operatorname{Exp}(\lambda)$ and $t \ge 0$ , integrating the density gives

S(t) = \int_t^{\infty} \lambda e^{-\lambda u}\,du = e^{-\lambda t},

while $S(t) = 1$ for $t < 0$ . So $F(t) = 1 - e^{-\lambda t}$ for $t \ge 0$ and $F(t) = 0$ for $t < 0$ . For $0 \le a < b$ ,

\begin{aligned} &P(a < T \le b) \\[0.4em] &\quad = S(a) - S(b) \\[0.4em] &\quad = e^{-\lambda a} - e^{-\lambda b}, \end{aligned}

and because $T$ has a density, including or excluding either endpoint does not change the probability.

In plain words The survival function gives the probability that the wait is still going on at time $t$ . For the exponential it is a single exponential factor, so tail and interval probabilities need no new integral once the formula is known. The formula $e^{-\lambda t}$ is used only for $t \ge 0$ . A negative threshold is certainly exceeded, so $S(t) = 1$ and $F(t) = 0$ there. Continuing $1 - e^{-\lambda t}$ to negative $t$ would give negative values, which no probability can have.

Example In the model with rate $0.5$ per minute, the probability of waiting more than $3$ minutes is $S(3) = e^{-1.5} \approx 0.2231$ , and the probability of being done by $3$ minutes is $F(3) \approx 0.7769$ . These two events split all possible waits, so their probabilities add to $1$ . A wait between $1$ and $3$ minutes has probability

\begin{aligned} &P(1 < T < 3) \\[0.4em] &\quad = e^{-0.5} - e^{-1.5} \\[0.4em] &\quad \approx 0.6065 - 0.2231 \\[0.4em] &\quad \approx 0.3834. \end{aligned}

At $t = -1$ the CDF is $0$ , while the expression $1 - e^{0.5} \approx -0.6487$ is negative.

Memorylessness

Definition A random variable $T$ with values in $[0, \infty)$ is memoryless if

\begin{aligned} &P(T > s + t \mid T > s) \\[0.4em] &\quad = P(T > t) \end{aligned}

for all $s, t \ge 0$ with $P(T > s) > 0$ . For $T \sim \operatorname{Exp}(\lambda)$ every probability $P(T > s) = e^{-\lambda s}$ is positive, and for $t \ge 0$ the event $\{T > s + t\}$ lies inside $\{T > s\}$ , so

\begin{aligned} &P(T > s + t \mid T > s) \\[0.4em] &\quad = \frac{e^{-\lambda (s + t)}}{e^{-\lambda s}} = e^{-\lambda t}. \end{aligned}

So every exponential distribution is memoryless. Among random variables with values in $[0, \infty)$ that have a density, the exponential distributions are the only memoryless ones. For waits counted in whole trials, the geometric distribution has the same property when $s$ and $t$ are whole numbers.

In plain words Given that the wait has already lasted $s$ , the remaining wait $T - s$ has the same $\operatorname{Exp}(\lambda)$ distribution as a fresh wait. Within the model, the time already spent does not change the distribution of the time still to come. This is a property of the model, not a law about real waits. A task with a fixed deadline, or a part that wears out with age, has an outlook that changes as time passes, and such waits may need a different model.

Example In the model with rate $0.5$ per minute, suppose the wait has already lasted $2$ minutes. The probability of more than $3$ further minutes is

\begin{aligned} &P(T > 5 \mid T > 2) \\[0.4em] &\quad = \frac{e^{-2.5}}{e^{-1}} \\[0.4em] &\quad = e^{-1.5} \approx 0.2231, \end{aligned}

the same as $P(T > 3)$ for a fresh wait. The expected remaining wait is still $2$ minutes, so the expected total wait given $T > 2$ is $4$ minutes. A uniform wait behaves differently. If $T \sim \operatorname{Uniform}(0, 10)$ in minutes, then $P(T > 3) = 0.7$ , but after $5$ minutes of waiting

\begin{aligned} &P(T > 8 \mid T > 5) \\[0.4em] &\quad = \frac{0.2}{0.5} = 0.4, \end{aligned}

because the end of the support at $10$ minutes is getting closer.

Natural logarithm and exponential quantile

Definition For $y > 0$ , the natural logarithm $\ln y$ is the unique real number $x$ with $e^{x} = y$ . Hence $\ln(e^{x}) = x$ for every real $x$ and $e^{\ln y} = y$ for every $y > 0$ . The logarithm is increasing, $\ln 1 = 0$ , and $\ln y < 0$ when $0 < y < 1$ . For $0 < p < 1$ , the $p$ -quantile of $\operatorname{Exp}(\lambda)$ is the unique time $Q(p)$ with $F(Q(p)) = p$ . It exists and is unique because $F$ is continuous and strictly increasing on $[0, \infty)$ , starting at $0$ and approaching $1$ . Solving $1 - e^{-\lambda t} = p$ gives $e^{-\lambda t} = 1 - p$ , then $-\lambda t = \ln(1 - p)$ , so

\begin{aligned} Q(p) &= -\frac{\ln(1 - p)}{\lambda} \\[0.4em] &= \frac{1}{\lambda}\ln\frac{1}{1 - p}. \end{aligned}

The median is $Q(0.5) = (\ln 2)/\lambda$ .

In plain words The logarithm undoes the exponential, so it recovers the exponent $-\lambda t$ from the tail probability $e^{-\lambda t}$ , and dividing by $-\lambda$ gives the time. The remaining tail $1 - p$ lies strictly between $0$ and $1$ , its logarithm is negative, and the minus sign in the formula makes the time positive. A negative quantile signals a sign error. The logarithm acts on a probability, which has no units, and dividing by the rate returns time units. The median is below the mean, because $\ln 2 \approx 0.693$ is less than $1$ , and the long right tail pulls the mean upward.

Example For the rate $0.5$ per minute, the $0.9$ -quantile is the time by which the wait is over with probability $0.9$ . It solves $e^{-0.5t} = 0.1$ , so

\begin{aligned} Q(0.9) &= \frac{\ln 10}{0.5} = 2\ln 10 \\[0.4em] &\approx 4.605 \end{aligned}

minutes. The median is $2\ln 2 \approx 1.386$ minutes, below the mean of $2$ minutes. Every exponential model gives $P(T \le 1/\lambda) = 1 - e^{-1} \approx 0.632$ , so about $63$ percent of the probability lies at or below the mean, whatever the rate.

Check yourself

A constructed model says a waiting time $T$ in minutes is exponential with mean $4$ minutes. Find the rate, the probability of waiting more than $6$ minutes, and the median.

Show the answer

The rate is the reciprocal of the mean, $\lambda = 1/4 = 0.25$ per minute, and the scale is $4$ minutes. Then

\begin{aligned} P(T > 6) &= e^{-6/4} = e^{-1.5} \\[0.4em] &\approx 0.2231. \end{aligned}

The median is $Q(0.5) = (\ln 2)/0.25 = 4\ln 2 \approx 2.773$ minutes, below the mean of $4$ minutes.

In the same model, the wait has already lasted $10$ minutes. Find the probability that it lasts at least $2$ more minutes. Why is $P(T > 12) \approx 0.0498$ not the answer?

Show the answer

The question asks for $P(T \ge 12 \mid T > 10)$ . Since $T$ has a density, this equals $P(T > 12 \mid T > 10)$ , and memorylessness gives

\begin{aligned} &P(T > 12 \mid T > 10) \\[0.4em] &\quad = e^{-0.25 \cdot 2} = e^{-0.5} \\[0.4em] &\quad \approx 0.6065. \end{aligned}

The value $P(T > 12) = e^{-3} \approx 0.0498$ is the probability, judged at the start, that the total wait exceeds $12$ minutes. It ignores the information that $10$ minutes have already passed. The answer $0.6065$ holds under the exponential model, not for every real wait that has lasted ten minutes.

A constructed report says a repair time $T$ , in hours, is “exponential with parameter $2$ ” and does not say whether $2$ is a rate or a scale. Find the mean and $P(T > 1)$ under each reading.

Show the answer

Read as a rate, $\lambda = 2$ per hour, the mean is $0.5$ hours and $P(T > 1) = e^{-2} \approx 0.1353$ . Read as a scale, $\theta = 2$ hours, the rate is $\lambda = 0.5$ per hour, the mean is $2$ hours and $P(T > 1) = e^{-0.5} \approx 0.6065$ . The means differ by a factor of $4$ and the tail probabilities by a factor of about $4.5$ , so the convention has to be settled before any number is used.

Exponential Distributions

Key terms

Exponential distribution

Rate and scale parameter

Survival function

Memorylessness

Natural logarithm and exponential quantile

Check yourself

How this lesson connects

Builds on

Used later in