The Bernoulli Distribution, introduction to probability lesson

Many questions about a random experiment have a yes or no answer. Does this package need a manual label check? Does the lamp pass its test? Recording yes as $1$ and no as $0$ turns the answer into a random variable that takes only the values $0$ and $1$ , and a single probability then describes its whole distribution. This lesson builds that distribution, derives its mean and variance, and separates what it says about one trial from what it leaves open about several.

Key terms

Bernoulli trial and success

Definition A Bernoulli trial is one run of a random experiment together with one event $A$ whose occurrence is recorded. The event $A$ is called success and its complement $A^{c}$ is called failure. Coding success as $1$ and failure as $0$ gives the indicator $I_A$ , which equals $1$ on outcomes in $A$ and $0$ on all other outcomes. Many texts use the plural phrase “Bernoulli trials” only for independent trials with a common success probability. Independence and a common success probability are extra assumptions about the whole sequence, and they do not follow from each trial having two recorded results.

In plain words Success is a name for the event you chose to record, not a judgment. A defect, a delay or a failed test can be the success if that is the event being counted. The experiment itself may have many outcomes, as long as each one is sorted into the event or its complement.

Example Roll a fair die and call a six a success. The die has six outcomes, but the trial records only whether $A = \{6\}$ occurred, so $I_A = 1$ with probability $\tfrac{1}{6}$ . For a package on a sorting line, the success can be the event that it needs a manual label check, even though nobody hopes for that. Now draw two tokens without replacement from a bag with $4$ marked tokens among $10$ , and record a marked token as success. Each draw is a Bernoulli trial with success probability $\tfrac{4}{10}$ before anything is seen, yet after a marked first draw the second is marked with probability $\tfrac{3}{9}$ . The two trials are dependent.

Bernoulli distribution

Definition Let $0 \le p \le 1$ . A random variable $B$ has the Bernoulli distribution with parameter $p$ , written $B \sim \operatorname{Bernoulli}(p)$ , if its PMF has

\begin{gathered} p_B(1) = p, \\[0.4em] p_B(0) = 1 - p, \end{gathered}

and $p_B(b) = 0$ for every other $b$ . The two masses are nonnegative and add to $1$ for every $p$ in $[0, 1]$ , so this is a valid distribution. Every indicator $I_A$ is Bernoulli with parameter $P(A)$ . Some books write the PMF as $p^{b}(1 - p)^{1 - b}$ for $b \in \{0, 1\}$ , which relies on the convention $0^0 = 1$ when $p$ is $0$ or $1$ .

In plain words A Bernoulli variable is a yes or no answer written as a number. Its whole distribution is fixed by one probability, the chance of a $1$ . Questions about $B$ are answered by translating the event into the values $0$ and $1$ .

Example Let $B = 1$ if a package needs a manual check and $B = 0$ otherwise, with $P(B = 1) = \tfrac{3}{20}$ in this constructed model. Then $B \sim \operatorname{Bernoulli}(3/20)$ and $P(B = 0) = \tfrac{17}{20}$ . The event $B \ge 1$ holds only at the value $1$ , so $P(B \ge 1) = \tfrac{3}{20}$ , while $P(B > 1) = 0$ because no possible value exceeds $1$ .

Parameter of a distribution

Definition A parameter is a number, or a list of numbers, that selects one distribution from a family described by a common formula. The set of allowed values is the parameter space. The Bernoulli family has one parameter $p$ with parameter space $[0, 1]$ , and each value of $p$ gives exactly one PMF. If $B \sim \operatorname{Bernoulli}(p)$ , the reversed coding $C = 1 - B$ satisfies $C \sim \operatorname{Bernoulli}(1 - p)$ , because it exchanges the two masses.

In plain words The family gives the formula, and the parameter fills in what the formula leaves open, which for the Bernoulli family is the one number $p$ . The parameter belongs to the model. The fraction of checked packages in one batch is an observed frequency, and it can differ from $p$ . Different trials can belong to the same family with different parameters.

Example For the package model, $B \sim \operatorname{Bernoulli}(3/20)$ . The indicator of no manual check is $C = 1 - B$ , which equals $1$ exactly when $B = 0$ , so $C \sim \operatorname{Bernoulli}(17/20)$ . A fragile label type with a higher check probability, say $\tfrac{3}{10}$ , is described by another member of the same family, and treating both label types as one trial type with a single $p$ would misdescribe them.

Mean and variance of a Bernoulli variable

Definition If $B \sim \operatorname{Bernoulli}(p)$ , then

\begin{aligned} E[B] &= 1 \cdot p + 0 \cdot (1 - p) \\[0.4em] &= p. \end{aligned}

Because $0^2 = 0$ and $1^2 = 1$ , the variable satisfies $B^2 = B$ , so $E[B^2] = p$ and

\begin{aligned} \operatorname{Var}(B) &= E[B^2] - (E[B])^2 \\[0.4em] &= p - p^2 \\[0.4em] &= p(1 - p). \end{aligned}

The standard deviation is $\operatorname{SD}(B) = \sqrt{p(1 - p)}$ . For an indicator these give $E[I_A] = P(A)$ and $\operatorname{Var}(I_A) = P(A)\bigl(1 - P(A)\bigr)$ .

In plain words The mean of a variable that is $0$ or $1$ is the probability of a $1$ . It is an average over the model, and when $0 < p < 1$ it is not a value the variable can take. Squaring a $0$ or a $1$ changes nothing, so the second moment equals the mean, and subtracting the squared mean leaves $p(1 - p)$ .

Example For the package indicator with $p = \tfrac{3}{20}$ , the mean is $E[B] = \tfrac{3}{20} = 0.15$ . No package receives $0.15$ of a check. Each package gives $0$ or $1$ , and $0.15$ is the probability-weighted average of those values. The variance is

\begin{aligned} \operatorname{Var}(B) &= \frac{3}{20} \cdot \frac{17}{20} \\[0.4em] &= \frac{51}{400} = 0.1275, \end{aligned}

and $\operatorname{SD}(B) = \sqrt{51}/20 \approx 0.357$ . If ordinary processing costs $2$ units and a manual check adds $8$ more, the cost is $2 + 8B$ , and linearity gives the expected cost $2 + 8 \cdot \tfrac{3}{20} = \tfrac{16}{5} = 3.2$ units. The table of costs agrees, since $2 \cdot \tfrac{17}{20} + 10 \cdot \tfrac{3}{20} = \tfrac{64}{20}$ .

Degenerate distribution

Definition A random variable $X$ has a degenerate distribution if $P(X = c) = 1$ for some constant $c$ . Its variance is then $0$ , and conversely a variable with variance $0$ satisfies $P(X = E[X]) = 1$ . The Bernoulli distributions with $p = 0$ and $p = 1$ are degenerate, at $0$ and at $1$ . For $0 < p < 1$ both values $0$ and $1$ have positive probability and $\operatorname{Var}(B) = p(1 - p) > 0$ .

In plain words A degenerate variable takes one value with probability $1$ , so there is nothing left to predict. It is still a valid distribution. At the endpoints it is clearer to read the PMF directly than to evaluate $p^{b}(1 - p)^{1 - b}$ with a $0^0$ in it.

Example In a model where a lamp passes its test with probability $1$ , the indicator of failure is $\operatorname{Bernoulli}(0)$ . It equals $0$ with probability $1$ , its mean is $0$ and its variance is $0 \cdot 1 = 0$ . The indicator of passing is $\operatorname{Bernoulli}(1)$ , with mean $1$ and variance $1 \cdot 0 = 0$ .

Maximum Bernoulli variance

Definition For every $p$ in $[0, 1]$ ,

\begin{aligned} p(1 - p) &= \frac{1}{4} - \Bigl(p - \frac{1}{2}\Bigr)^2 \\[0.4em] &\le \frac{1}{4}, \end{aligned}

with equality exactly when $p = \tfrac{1}{2}$ . So a Bernoulli variance is at most $\tfrac{1}{4}$ , a Bernoulli standard deviation is at most $\tfrac{1}{2}$ , and both maxima occur only at $p = \tfrac{1}{2}$ . The variance takes the same value at $p$ and at $1 - p$ .

In plain words The result is hardest to predict when $0$ and $1$ are equally likely. As $p$ moves away from $\tfrac{1}{2}$ toward $0$ or $1$ , one value dominates and the variance shrinks, reaching $0$ at the endpoints. Coding the other event as success replaces $p$ by $1 - p$ and leaves the variance unchanged.

Example The package indicator with $p = 0.15$ has variance $0.1275$ . A lamp indicator with $p = 0.3$ or with $p = 0.7$ has variance $0.21$ , and a fair coin has the largest possible value $0.25$ . No Bernoulli variable has variance $0.3$ , since that exceeds $\tfrac{1}{4}$ . A variance of $0.21$ does not determine $p$ , because both $0.3$ and $0.7$ give it.

Check yourself

A lamp works on its test with probability $0.7$ . Let $W = 1$ if it works and $W = 0$ if it fails, and let $F = 1$ if it fails and $F = 0$ if it works. Give the distributions of $W$ and $F$ with their means and variances.

Show the answer

The variable $W$ is $\operatorname{Bernoulli}(0.7)$ , with mean $0.7$ and variance $0.7 \cdot 0.3 = 0.21$ . The failure indicator is $F = 1 - W$ , which is $\operatorname{Bernoulli}(0.3)$ with mean $0.3$ and the same variance $0.3 \cdot 0.7 = 0.21$ . These are two codings of the same test, not two independent observations. Whenever one of them is $1$ , the other is $0$ .

Each lamp test costs $3$ units, and a failure adds $5$ more. Find the expected cost of one test in two ways.

Show the answer

The cost is $3$ when the lamp works and $8$ when it fails, so the table gives $3 \cdot 0.7 + 8 \cdot 0.3 = 2.1 + 2.4 = 4.5$ units. The cost can also be written as $3 + 5F$ , and linearity gives $3 + 5 \cdot 0.3 = 4.5$ units.

Two packages each need a manual check with probability $\tfrac{3}{20}$ . Is the probability that both need checks equal to $\bigl(\tfrac{3}{20}\bigr)^2 = \tfrac{9}{400}$ ?

Show the answer

Not necessarily. The product rule holds exactly when the two check indicators are independent, and their Bernoulli laws say nothing about that. Suppose a shared printing fault makes the two packages either both need a check or both pass, with the check happening with probability $\tfrac{3}{20}$ . Each indicator is still $\operatorname{Bernoulli}(3/20)$ , but the probability that both need checks is $\tfrac{3}{20}$ , not $\tfrac{9}{400}$ . In general these two marginal laws allow any value from $0$ to $\tfrac{3}{20}$ for that probability, so the way the packages are linked has to be modeled separately.

The Bernoulli Distribution

Key terms

Bernoulli trial and success

Bernoulli distribution

Parameter of a distribution

Mean and variance of a Bernoulli variable

Degenerate distribution

Maximum Bernoulli variance

Check yourself

How this lesson connects

Builds on

Used later in