Conditional Probability and Bayes' Rule, machine learning lesson

A signal flags 80 percent of the faulty items it inspects. When it flags an item, is that item 80 percent likely to be faulty? The two questions sound alike, but they measure the same overlap against different groups of items, and in the constructed example on this page the second answer is only about 31 percent. This lesson defines events and conditional probability, derives Bayes’ rule, shows how the base rate shapes a posterior probability, and separates independence from conditional independence.

Key terms

Events and joint probability

Definition A probability model starts from a sample space $\Omega$ , the set of all possible outcomes. An event is a set of outcomes, a subset $A \subseteq \Omega$ to which the model assigns a probability, and it occurs when the outcome lies in $A$ . In a finite model every subset can be an event. A probability $P$ gives each event a number with $0 \le P(A) \le 1$ and $P(\Omega) = 1$ , and it adds over disjoint events, so $P(A \cup B) = P(A) + P(B)$ whenever $A \cap B = \varnothing$ , and likewise for any finite or countable list of pairwise disjoint events. The complement $A^c$ contains the outcomes not in $A$ , and $P(A^c) = 1 - P(A)$ . The joint probability of events $A$ and $B$ is $P(A \cap B)$ , the probability that both occur. When $\Omega$ is finite and all its outcomes are equally likely,

P(A) = \frac{\lvert A \rvert}{\lvert \Omega \rvert},

the number of outcomes in $A$ divided by the number of all outcomes.

In plain words An event matches a yes or no question about the outcome, answered once the outcome is known. A joint probability asks for two answers to be yes at the same time. Every probability refers to a stated population or process, such as one item drawn at random from a stated set, and changing that reference can change the number.

Example A constructed set of $100$ items has been inspected. Each item is faulty or sound, and a signal either flagged it or did not. Draw one item at random, with all $100$ equally likely.

	Flagged	Not flagged	Total
Faulty	8	2	10
Sound	18	72	90
Total	26	74	100

Let $D$ be the event that the drawn item is faulty and $S$ the event that it was flagged. Then $P(D) = 0.10$ , its complement has $P(D^c) = 0.90$ , the flag event has $P(S) = 0.26$ , and the joint probability is $P(D \cap S) = 0.08$ . These are exact probabilities for a draw from this table. Counts from a real inspection study would only estimate the probabilities of a larger population, with sampling and measurement error.

Conditional probability

Definition Let $A$ and $B$ be events with $P(B) > 0$ . The conditional probability of $A$ given $B$ is

P(A \mid B) = \frac{P(A \cap B)}{P(B)}.

When $P(B) = 0$ this ratio is undefined. For a fixed event $B$ with $P(B) > 0$ , the rule that sends each event $A$ to $P(A \mid B)$ is itself a probability, so for example $P(A \mid B) + P(A^c \mid B) = 1$ . The two conditional probabilities $P(A \mid B)$ and $P(B \mid A)$ share the numerator $P(A \cap B)$ but divide it by different probabilities. When $P(A)$ and $P(B)$ are both positive, they are equal exactly when $P(A) = P(B)$ or $P(A \cap B) = 0$ .

In plain words Conditioning on $B$ means keeping only the outcomes in $B$ and asking what share of that group, measured by probability, also lies in $A$ . The event after the bar names the reference group and supplies the denominator. Swapping the two events swaps the reference group, and so it changes the question being asked.

Example In the inspection table, $8$ of the $10$ faulty items are flagged, so $P(S \mid D) = 0.08/0.10 = 0.8$ . Reversing the condition, $8$ of the $26$ flagged items are faulty, so

P(D \mid S) = \frac{0.08}{0.26} = \frac{4}{13},

about $0.308$ . Among the $74$ unflagged items, $2$ are faulty, so $P(D \mid S^c) = 2/74 = 1/37$ , about $0.027$ . A flag raises the probability of a fault from $0.1$ and the absence of a flag lowers it, but neither one settles the state of a particular item.

Bayes’ rule

Definition Let $D$ and $S$ be events with $P(D) > 0$ and $P(S) > 0$ . The joint probability can be written in two ways, $P(D \cap S) = P(D \mid S)\,P(S)$ and $P(D \cap S) = P(S \mid D)\,P(D)$ . Setting them equal and dividing by $P(S)$ gives Bayes’ rule,

\begin{aligned} &P(D \mid S) \\[0.4em] &\quad = \frac{P(S \mid D)\,P(D)}{P(S)}. \end{aligned}

When also $P(D) < 1$ , the law of total probability splits $S$ by whether $D$ occurs,

\begin{aligned} P(S) &= P(S \mid D)\,P(D) \\[0.4em] &\quad + P(S \mid D^c)\,P(D^c). \end{aligned}

In this form $P(D)$ is the prior probability of the state $D$ , the conditional probabilities $P(S \mid D)$ and $P(S \mid D^c)$ are the likelihoods of the observed evidence $S$ under each state, and $P(D \mid S)$ is the posterior probability. A likelihood is a probability of the evidence given a state, and it is not a probability of the state.

In plain words Bayes’ rule turns the probability of the evidence given the state, which is what a detector’s rates describe, into the probability of the state given the evidence, which is what a decision about one flagged item needs. The denominator counts every way the evidence can arise. Leaving out the flags that come from sound items would treat every flag as a true fault.

Example For the inspection table, $P(D) = 0.1$ , $P(S \mid D) = 0.8$ and $P(S \mid D^c) = 18/90 = 0.2$ . The two routes to a flag have probabilities $0.8 \times 0.1 = 0.08$ and $0.2 \times 0.9 = 0.18$ , so $P(S) = 0.26$ and

\begin{aligned} P(D \mid S) &= \frac{0.08}{0.08 + 0.18} \\[0.4em] &= \frac{4}{13}, \end{aligned}

the same answer that the table gives directly.

Base rate and posterior probability

Definition The base rate of a state is its probability in the population where a prediction is made, before the evidence at hand is taken into account. In Bayes’ rule it is the prior $P(D)$ . The posterior probability $P(D \mid S)$ is the probability of the state after conditioning on the observed evidence $S$ . If the two likelihoods $a = P(S \mid D)$ and $b = P(S \mid D^c)$ are positive and held fixed, the posterior

P(D \mid S) = \frac{a\,p}{a\,p + b\,(1 - p)}

is a strictly increasing function of the base rate $p = P(D)$ for $0 < p < 1$ .

In plain words The base rate is what is known before the signal, and the posterior is what is known after it. A detector with fixed rates means less when the condition it detects is rare, because the false flags from the large group of sound items can outnumber the true flags from the small group of faulty ones. A probabilistic classifier aims to report posterior probabilities of the classes given the features, so the right outputs depend on the class base rates of the population where it is used. A posterior does not by itself choose an action, since that also depends on the costs of the possible mistakes.

Example Keep $P(S \mid D) = 0.8$ and $P(S \mid D^c) = 0.2$ , but move to a population where only $1$ percent of items are faulty. The routes to a flag now have probabilities $0.8 \times 0.01 = 0.008$ and $0.2 \times 0.99 = 0.198$ , so

\begin{aligned} P(D \mid S) &= \frac{0.008}{0.008 + 0.198} \\[0.4em] &= \frac{4}{103} \approx 0.039. \end{aligned}

The posterior fell from about $0.308$ to about $0.039$ , although the detector’s rates did not change. The comparison assumes that those rates carry over to the new population. If conditions there also change how the signal behaves, new likelihoods are needed as well as a new base rate.

Independence

Definition Events $A$ and $B$ are independent if

P(A \cap B) = P(A)\,P(B).

When $P(B) > 0$ this is equivalent to $P(A \mid B) = P(A)$ . If $A$ and $B$ are independent, then so are $A$ and $B^c$ . An event with probability $0$ or $1$ is independent of every event. Disjoint events that both have positive probability are never independent, because their joint probability is $0$ while the product of their probabilities is positive.

In plain words For events with positive probability, independence means that learning one of them occurred leaves the probability of the other unchanged. It is a property of the probability model. It should be justified by how the outcomes are produced or checked against the model’s numbers, and it never follows from what the events are called. Disjoint events with positive probabilities are strongly dependent, since each one rules out the other.

Example In the inspection table, $P(D)\,P(S) = 0.1 \times 0.26 = 0.026$ , while $P(D \cap S) = 0.08$ . The two differ, so $D$ and $S$ are not independent, as the comparison $P(D \mid S) \approx 0.308 > 0.1 = P(D)$ also shows. That dependence is exactly what makes the signal useful. A signal independent of the state would leave the probability of a fault at $0.1$ whether or not it flagged the item.

Conditional independence

Definition Let $C$ be an event with $P(C) > 0$ . Events $A$ and $B$ are conditionally independent given $C$ if

\begin{aligned} &P(A \cap B \mid C) \\[0.4em] &\quad = P(A \mid C)\,P(B \mid C). \end{aligned}

Conditional independence given a variable, such as the true state of an item, means that this equation holds given each value of the variable that has positive probability, here given $D$ and given $D^c$ . In general neither independence nor conditional independence implies the other.

In plain words Conditional independence is an independence statement made inside each group defined by the condition. Two sensors that both respond to faults can be independent among the faulty items and among the sound items, yet flag together more often than independence would predict across the whole population, because both follow the same hidden state. An independence claim should always name what it conditions on, and dropping the condition changes the claim.

Example Two sensors each flag a faulty item with probability $0.8$ and a sound item with probability $0.2$ , and $P(D) = 0.1$ as before. Let $U$ and $V$ be the events that the first and the second sensor flag, and assume they are conditionally independent given the state. Given a faulty item both flag with probability $0.8^2 = 0.64$ , and given a sound item with probability $0.2^2 = 0.04$ , so

\begin{aligned} P(U \cap V) &= 0.1 \times 0.64 \\[0.4em] &\quad + 0.9 \times 0.04 \\[0.4em] &= 0.1. \end{aligned}

Each sensor flags with probability $0.1 \times 0.8 + 0.9 \times 0.2 = 0.26$ , and $0.26^2 = 0.0676$ is not $0.1$ , so $U$ and $V$ are not independent. Two flags give the posterior $P(D \mid U \cap V) = 0.064/0.1 = 0.64$ , well above the $4/13$ from one flag. The conditional independence is an assumption about how the sensors work. It does not follow from their equal rates.

Check yourself

A constructed set of $200$ messages contains $50$ spam messages. A filter flags $40$ of the spam messages and $15$ of the $150$ other messages. One message is chosen at random, with all $200$ equally likely. Find the probability that a spam message is flagged, the probability that a flagged message is spam, and the probability that an unflagged message is spam.

Show the answer

Let $A$ be the event that the message is spam and $F$ the event that it is flagged. Within the $50$ spam messages, $P(F \mid A) = 40/50 = 0.8$ . The filter flags $40 + 15 = 55$ messages, so within that group

P(A \mid F) = \frac{40}{55} = \frac{8}{11},

about $0.727$ . The $145$ unflagged messages contain $10$ spam messages, so $P(A \mid F^c) = 10/145 = 2/29$ , about $0.069$ . The first two answers share the numerator $40$ , the count of flagged spam, but divide it by different groups, which is why reversing the condition changes the answer.

In a constructed model, $20$ percent of items are faulty. A signal flags $75$ percent of faulty items and $10$ percent of sound items. Find the probability that a flagged item is faulty, and explain why the answer is not $0.75$ . Then find the same posterior in a population where $2$ percent of items are faulty, assuming the signal’s rates carry over.

Show the answer

Let $D$ be the event that an item is faulty and $S$ the event that it is flagged. The routes to a flag have probabilities $0.75 \times 0.2 = 0.15$ and $0.1 \times 0.8 = 0.08$ , so $P(S) = 0.23$ and

P(D \mid S) = \frac{0.15}{0.23} = \frac{15}{23},

about $0.652$ . In expected counts per $100$ items, the model gives $20$ faulty items with $15$ flags and $80$ sound items with $8$ flags, and $15$ of the $23$ flags belong to faulty items. The number $0.75$ is $P(S \mid D)$ , the share of faulty items that get flagged, which answers a different question with a different reference group.

With a base rate of $0.02$ the routes have probabilities $0.75 \times 0.02 = 0.015$ and $0.1 \times 0.98 = 0.098$ , so

\begin{aligned} P(D \mid S) &= \frac{0.015}{0.113} \\[0.4em] &= \frac{15}{113} \approx 0.133. \end{aligned}

The same flag is now much weaker evidence, because sound items supply most of the flags.

Two sensors each flag a faulty item with probability $0.9$ and a sound item with probability $0.1$ , independently of each other given the state, and $30$ percent of items are faulty. Let $U$ and $V$ be the events that the first and the second sensor flag. A colleague says the probability that both flag is $P(U)\,P(V)$ . Find $P(U)$ , the correct $P(U \cap V)$ and the posterior probability of a fault when both flag.

Show the answer

Let $D$ be the event that the item is faulty. By the law of total probability, $P(U) = 0.3 \times 0.9 + 0.7 \times 0.1 = 0.34$ , and the same holds for $V$ . Conditional independence lets the probabilities multiply only within each state, so

\begin{aligned} P(U \cap V) &= 0.3 \times 0.81 \\[0.4em] &\quad + 0.7 \times 0.01 \\[0.4em] &= 0.25. \end{aligned}

The colleague’s product $0.34^2 = 0.1156$ is less than half of this. The sensors are not independent across the whole population, because both follow the same hidden state, and indeed $P(V \mid U) = 0.25/0.34 \approx 0.735$ , far above $P(V) = 0.34$ . When both flag,

\begin{aligned} P(D \mid U \cap V) &= \frac{0.243}{0.25} \\[0.4em] &= 0.972, \end{aligned}

compared with $P(D \mid U) = 0.27/0.34 \approx 0.794$ after a single flag.

Conditional Probability and Bayes' Rule

Key terms

Events and joint probability

Conditional probability

Bayes’ rule

Base rate and posterior probability

Independence

Conditional independence

Check yourself

How this lesson connects

Builds on

Used later in