Bayes’ Theorem and Base Rates, introduction to probability lesson

An inspection system raises an alert for 90 percent of faulty items. If an item triggers an alert, is it faulty with probability $0.9$ ? In general it is not, because that rate conditions on the fault while the question conditions on the alert, and the answer also depends on how common faults are before any alert. This lesson derives Bayes’ theorem from the definition of conditional probability and the law of total probability, and uses it to combine a prior probability with the likelihoods of the evidence into a posterior probability. All rates on this page are constructed for illustration.

Key terms

Bayes’ theorem

Definition Let $H$ and $E$ be events with $P(H) > 0$ and $P(E) > 0$ . Then

P(H \mid E) = \frac{P(E \mid H) P(H)}{P(E)}.

The proof uses the definition of conditional probability twice, since $P(H \mid E)\,P(E)$ and $P(E \mid H)\,P(H)$ both equal $P(H \cap E)$ . If also $P(H) < 1$ , the law of total probability over the partition $H$ , $H^{c}$ gives the denominator

\begin{aligned} P(E) &= P(E \mid H) P(H) \\[0.4em] &\quad + P(E \mid H^{c}) P(H^{c}). \end{aligned}

More generally, if $H_1, \dots, H_k$ partition $\Omega$ and every $P(H_i) > 0$ , then $P(E) = \sum_{i=1}^{k} P(E \mid H_i)\,P(H_i)$ , and the same formula gives each $P(H_j \mid E)$ . The requirement $P(E) > 0$ cannot be dropped, because the calculation conditions on $E$ .

In plain words To find how probable $H$ is once $E$ is known, take the probability that $H$ and $E$ happen together and divide it by the total probability of $E$ , counting every way that $E$ can happen. Bayes’ theorem builds that joint probability from the prior and the likelihood, which are often the numbers a model actually provides.

Example Let $F$ be the event that an item is faulty and $A$ the event that it triggers an alert, with $P(F) = 0.02$ , $P(A \mid F) = 0.9$ and $P(A \mid F^{c}) = 0.05$ . True alerts contribute $0.9 \cdot 0.02 = 0.018$ and false alerts contribute $0.05 \cdot 0.98 = 0.049$ , so $P(A) = 0.067$ and

P(F \mid A) = \frac{0.018}{0.067} = \frac{18}{67},

about $0.269$ . An imagined group of 10,000 items with exactly these proportions shows the same calculation as counts.

	Alert	No alert	Total
Faulty	180	20	200
Sound	490	9,310	9,800
Total	670	9,330	10,000

The alert column holds 670 items, of which 180 are faulty, and $180/670 = 18/67$ . These counts are the model’s probabilities written as frequencies, not measured results, and a random batch of 10,000 items would generally show somewhat different counts.

Prior and posterior probability

Definition In a Bayes calculation for a hypothesis $H$ and evidence $E$ , the prior probability is $P(H)$ , the probability of $H$ before $E$ is taken into account. The posterior probability is $P(H \mid E)$ , the probability of $H$ after conditioning on $E$ , which requires $P(E) > 0$ . Both are probabilities of the same event in the same model, and they differ only in the information they condition on. When the prior describes how common the hypothesis is in the relevant population, it is also called the base rate.

In plain words The prior is the starting point, and the posterior is where the evidence moves it. The names describe roles in one calculation, not two different kinds of probability.

Example In the alert model the prior fault probability is $0.02$ , and after an alert the posterior is $18/67 \approx 0.269$ . No alert has probability $0.1$ given a fault, so $P(F \cap A^{c}) = 0.02 \cdot 0.1 = 0.002$ , while $P(A^{c}) = 1 - 0.067 = 0.933$ . Hence

P(F \mid A^{c}) = \frac{0.002}{0.933} = \frac{2}{933},

about $0.0021$ . In the table this is $20/9{,}330$ , the faulty share of the whole no-alert column. The two posteriors average back to the prior when weighted by the probabilities of the evidence, since $\tfrac{18}{67} \cdot 0.067 + \tfrac{2}{933} \cdot 0.933 = 0.018 + 0.002 = 0.02$ , as the law of total probability requires.

Likelihood

Definition For a hypothesis $H$ with $P(H) > 0$ and an observed event $E$ , the likelihood of $E$ under $H$ is the conditional probability $P(E \mid H)$ . It conditions on the hypothesis, the opposite direction from the posterior $P(H \mid E)$ . When also $P(E) > 0$ , Bayes’ theorem shows that the two are equal exactly when $P(E \mid H) = 0$ or $P(H) = P(E)$ , so in general they differ. The likelihoods of one piece of evidence under competing hypotheses, such as $P(E \mid H)$ and $P(E \mid H^{c})$ , condition on different events, so they need not add to $1$ .

In plain words A likelihood asks how expected the evidence is if the hypothesis is true. A likelihood alone is not a posterior, because it leaves out how probable the hypothesis was to begin with and how expected the evidence is under the alternatives.

Example In the alert model the likelihood of an alert is $0.9$ under a fault and $0.05$ under a sound item, and these add to $0.95$ , not $1$ . Their ratio $0.9/0.05 = 18$ says that an alert is 18 times as likely for a faulty item as for a sound one. This ratio gives a quick form of the update in odds. The prior odds of a fault are $0.02$ to $0.98$ , or $1$ to $49$ , and multiplying by $18$ gives posterior odds of $18$ to $49$ , which is the probability $18/(18 + 49) = 18/67$ again. Multiplying the odds by 18 did not multiply the probability by 18, which would have given $0.36$ instead of about $0.269$ . The lesson Probability and Odds develops this form.

Sensitivity and specificity

Definition Let $F$ be the event that the condition being tested for is present, with $0 < P(F) < 1$ , and let $A$ be the event of a positive result, here an alert. The sensitivity, or true positive rate, is $P(A \mid F)$ . Its complement $P(A^{c} \mid F) = 1 - P(A \mid F)$ is the false negative rate. The specificity, or true negative rate, is $P(A^{c} \mid F^{c})$ , and its complement is the false positive rate. All of these rates condition on the true state, so none of them is a posterior probability of the condition.

In plain words Sensitivity asks how often the test catches the condition when it is present. Specificity asks how often the test stays negative when the condition is absent. Each looks at one true state at a time.

Example In the alert model the sensitivity is $0.9$ , so the false negative rate is $0.1$ , and the specificity is $1 - 0.05 = 0.95$ . Sensitivity is not the probability of a fault after an alert, which is $18/67 \approx 0.269$ . Likewise a false negative rate of $0.1$ does not make the fault probability after no alert equal to $0.1$ . That posterior is $2/933 \approx 0.0021$ , the faulty share of all 9,330 no-alert items in the table, not a share of the faulty items.

False positive rate

Definition The false positive rate of a test is $P(A \mid F^{c})$ , the probability of a positive result for an item without the condition, which requires $P(F^{c}) > 0$ . It equals $1$ minus the specificity. It is different from the proportion of positive results that are false, which is the posterior probability $P(F^{c} \mid A)$ and requires $P(A) > 0$ . By Bayes’ theorem that proportion depends on the base rate as well as on the test.

In plain words The false positive rate looks only at items without the condition and asks how often they set off the test. The share of false alarms looks only at alarms and asks how many of them come from such items.

Example In the alert model the false positive rate is $0.05$ , yet among alerts the share from sound items is

P(F^{c} \mid A) = \frac{0.049}{0.067} = \frac{49}{67},

about $0.731$ . In the table, 490 of the 9,800 sound items raise an alert, which is 5 percent, while 490 of the 670 alerts, about 73 percent, come from sound items. Sound items outnumber faulty ones 49 to 1, while the alert rate for faulty items is only 18 times the rate for sound ones, so here the small rate applied to the large group produces more alerts than the large rate applied to the small group.

Base rate neglect

Definition Base rate neglect is the error of judging a posterior probability $P(H \mid E)$ from the likelihoods alone while ignoring or underweighting the prior probability $P(H)$ , the base rate of the hypothesis in the relevant population. A common form is to report a likelihood such as the sensitivity $P(E \mid H)$ as if it were $P(H \mid E)$ . Bayes’ theorem shows that when the likelihoods $P(E \mid H)$ and $P(E \mid H^{c})$ are held fixed and both are positive, the posterior increases strictly as the prior increases, so the same evidence can support very different conclusions in different populations.

In plain words Unless the evidence rules the hypothesis in or out completely, how probable the hypothesis is after the evidence depends on how common it was before the evidence arrived, not only on the evidence. When the hypothesis is rare, even a small false positive rate applied to the many cases where it is false can produce more false alarms than true ones.

Example Keep the alert rates $0.9$ and $0.05$ , but suppose the items come from a population in which 20 percent are faulty. True alerts contribute $0.9 \cdot 0.2 = 0.18$ and false alerts $0.05 \cdot 0.8 = 0.04$ , so

P(F \mid A) = \frac{0.18}{0.22} = \frac{9}{11},

about $0.818$ . With the prior $0.02$ the same alert gave only $18/67 \approx 0.269$ . The alert system did not change. The mix of faulty and sound items changed, and with it the makeup of the alert group. The prior and the likelihoods must describe the population and selection process the item actually comes from, or the calculation is precise but answers a different question.

Check yourself

In another constructed system, 10 percent of items are faulty. An alert occurs for 80 percent of faulty items and for 20 percent of sound items. Find the probability that an item with an alert is faulty, and say how much of the alert probability comes from false alerts. Why is the answer neither $0.8$ nor $0.08$ ?

Show the answer

True alerts contribute $P(A \mid F)\,P(F) = 0.8 \cdot 0.1 = 0.08$ and false alerts contribute $P(A \mid F^{c})\,P(F^{c}) = 0.2 \cdot 0.9 = 0.18$ , so $P(A) = 0.26$ and

P(F \mid A) = \frac{0.08}{0.26} = \frac{4}{13},

about $0.308$ . False alerts supply $0.18$ of the $0.26$ , more than two thirds of the alert probability. The number $0.8$ is the likelihood $P(A \mid F)$ , which conditions in the wrong direction. The number $0.08$ is the joint probability $P(F \cap A)$ , which has not yet been divided by the probability of an alert. The alert raises the fault probability from $0.1$ to about $0.308$ because alerts are four times as likely under a fault, but the larger sound group still supplies most of the alerts.

A screening test for a condition with prevalence $0.05$ has sensitivity $0.95$ and specificity $0.90$ . These rates are illustrative. Find the probability of the condition after a positive result and after a negative result.

Show the answer

Let $D$ be the event that a person has the condition and $T$ the event of a positive result. The false positive rate is $1 - 0.90 = 0.10$ and the false negative rate is $1 - 0.95 = 0.05$ . In an imagined group of 2,000 people with exactly these proportions, 100 have the condition and 95 of them test positive, while 1,900 do not and 190 of them test positive. So

P(D \mid T) = \frac{95}{95 + 190} = \frac{1}{3}.

The formula agrees. True positives contribute $0.95 \cdot 0.05 = 0.0475$ and false positives $0.10 \cdot 0.95 = 0.095$ , so $P(T) = 0.1425$ and $P(D \mid T) = 0.0475/0.1425 = 1/3$ . Among negative results, 5 come from people with the condition and 1,710 from people without it, so

P(D \mid T^{c}) = \frac{5}{1715} = \frac{1}{343},

about $0.0029$ . A positive result raises the probability from $0.05$ to $1/3$ , yet two of every three positive results are false. A negative result lowers it to well under 1 percent.

Return to the first alert model, with prior $0.02$ , sensitivity $0.9$ and false positive rate $0.05$ . Which change raises $P(F \mid A)$ more, a perfect sensitivity of $1$ or a false positive rate of $0.01$ ? Explain the result with base rates.

Show the answer

With sensitivity $1$ , true alerts contribute $0.02$ and false alerts still contribute $0.049$ , so

\begin{aligned} P(F \mid A) &= \frac{0.02}{0.069} \\[0.4em] &= \frac{20}{69} \approx 0.290. \end{aligned}

With false positive rate $0.01$ , true alerts contribute $0.018$ and false alerts $0.01 \cdot 0.98 = 0.0098$ , so

\begin{aligned} P(F \mid A) &= \frac{0.018}{0.0278} \\[0.4em] &= \frac{90}{139} \approx 0.647. \end{aligned}

The lower false positive rate helps far more. Faults are rare, so most alerts come from the large sound group. Perfect sensitivity can add at most $0.002$ to the true-alert term, while the lower false positive rate cuts the false-alert term from $0.049$ to $0.0098$ .

Bayes’ Theorem and Base Rates

Key terms

Bayes’ theorem

Prior and posterior probability

Likelihood

Sensitivity and specificity

False positive rate

Base rate neglect

Check yourself

How this lesson connects

Builds on

Used later in

Where this leads in Statistics