Conditional Probability and Independence, statistics lesson

A clinic reports that 22 percent of its visits involve a long wait. If you book a morning visit, is 22 percent the right figure for you? Not necessarily, because the rate among morning visits can differ from the rate among all visits. This lesson shows how to compute a probability within a subgroup, how these conditional probabilities combine into the overall rate, and what it means for two events to be independent. All counts on this page are constructed for illustration.

Key terms

Joint and marginal probability

Definition For two events $A$ and $B$ , the joint probability $P(A \cap B)$ is the probability that both occur. A marginal probability is the probability of one event on its own, such as $P(A)$ , without regard to the other classification. Marginal probabilities are sums of joint probabilities. The events $A \cap B$ and $A \cap B^c$ are disjoint with union $A$ , so

\begin{aligned} P(A) &= P(A \cap B) \\[0.4em] &\quad + P(A \cap B^c). \end{aligned}

When a model is given as a two-way table of probabilities, the joint probabilities are the cells and the marginal probabilities are the row and column totals in the margins of the table.

In plain words Joint means both at once, and marginal means one classification while ignoring the other. Both are measured against the whole sample space, so in a table of counts both divide by the grand total.

Example A constructed set of $100$ clinic visits is classified by time of day and length of wait. Choose one visit at random, with all $100$ equally likely, so that each count divided by $100$ is a probability.

	Long wait	Short wait	Total
Morning	10	30	40
Not morning	12	48	60
Total	22	78	100

Let $M$ be the event of a morning visit and $L$ the event of a long wait. The joint probability is $P(M \cap L) = 10/100 = 0.10$ , and the marginal probabilities are $P(M) = 0.40$ and $P(L) = 0.22$ . The second one sums the joint probabilities in the long-wait column, $0.10 + 0.12 = 0.22$ .

Conditional probability

Definition Let $A$ and $B$ be events with $P(B) > 0$ . The conditional probability of $A$ given $B$ is

P(A \mid B) = \frac{P(A \cap B)}{P(B)}.

The vertical bar is read “given” and is not a division sign. When $P(B) = 0$ this formula does not define $P(A \mid B)$ . For a fixed event $B$ with $P(B) > 0$ , the conditional probabilities given $B$ obey all the rules of a probability model, so for example $P(A^c \mid B) = 1 - P(A \mid B)$ .

In plain words Once you know that $B$ has happened, the outcomes outside $B$ are out of play. Restrict attention to $B$ and ask what share of it, measured by probability, also lies in $A$ . With equally likely outcomes this is the number of outcomes in both $A$ and $B$ divided by the number in $B$ . The order matters, because $P(A \mid B)$ divides the joint probability by $P(B)$ , while $P(B \mid A)$ divides it by $P(A)$ , and these two usually differ.

Example In the visit table, $P(L \mid M) = 0.10/0.40 = 0.25$ , the share of long waits among the $40$ morning visits. Among the $60$ other visits it is $P(L \mid M^c) = 12/60 = 0.20$ . Reversing the condition gives

P(M \mid L) = \frac{0.10}{0.22} = \frac{5}{11},

about $0.455$ , the share of morning visits among the $22$ long waits. The joint probability $0.10$ is the same in both directions, but the denominators $0.40$ and $0.22$ differ. A higher rate of long waits among morning visits in these counts describes an association. On its own it does not show that the time of day causes the longer waits.

Multiplication rule

Definition For events $A$ and $B$ with $P(A) > 0$ ,

P(A \cap B) = P(A)\,P(B \mid A),

and likewise $P(A \cap B) = P(B)\,P(A \mid B)$ when $P(B) > 0$ . The rule is the definition of conditional probability multiplied out, and it holds whether or not the events are independent. It extends to more events, for example $P(A \cap B \cap C) = P(A)\,P(B \mid A)\,P(C \mid A \cap B)$ when $P(A \cap B) > 0$ .

In plain words To find the probability that both events happen, follow a path. First $A$ happens, and then $B$ happens given that $A$ did. Each branch of a probability tree carries the conditional probability of its step, and the probability of a whole path is the product along it.

Example In the visit table, $P(M)\,P(L \mid M) = 0.40 \times 0.25 = 0.10$ , and the reverse route gives $P(L)\,P(M \mid L) = 0.22 \times \tfrac{5}{11} = 0.10$ , the same joint probability. The rule is most useful when the second step depends on the first. Draw two items without replacement from a box of $10$ items of which $3$ are defective, with each draw equally likely to be any item still in the box. Let $D_1$ and $D_2$ be the events that the first and the second item are defective. After a defective first draw, $2$ of the remaining $9$ items are defective, so

\begin{aligned} P(D_1 \cap D_2) &= \frac{3}{10} \times \frac{2}{9} \\[0.4em] &= \frac{1}{15}. \end{aligned}

Independent events

Definition Events $A$ and $B$ are independent if

P(A \cap B) = P(A)\,P(B).

When $P(B) > 0$ this is equivalent to $P(A \mid B) = P(A)$ , and when $P(A) > 0$ it is equivalent to $P(B \mid A) = P(B)$ . If $A$ and $B$ are independent, then so are $A$ and $B^c$ , $A^c$ and $B$ , and $A^c$ and $B^c$ . Disjoint events with $P(A) > 0$ and $P(B) > 0$ are never independent, because $P(A \cap B) = 0$ while $P(A)\,P(B) > 0$ . An event with probability $0$ or $1$ is independent of every event.

In plain words For events with positive probabilities, independence means that learning one of them occurred leaves the probability of the other unchanged. It is a property of a probability model, justified by the mechanism that produces the outcomes, such as a separate coin and die, or checked against data. Two disjoint events with positive probabilities are strongly related, since each one rules out the other. In observed counts the product equation usually fails to hold exactly even when the process behind them is independent, so judging independence from a sample is a task for later inference lessons.

Example In the visit table, $P(M)\,P(L) = 0.40 \times 0.22 = 0.088$ , while $P(M \cap L) = 0.10$ . The two differ, so $M$ and $L$ are not independent in this set of visits, as the comparison $P(L \mid M) = 0.25 \ne 0.22 = P(L)$ also shows. In a model of a fair coin and a fair die tossed independently, the probability of heads and a six is $\tfrac12 \times \tfrac16 = \tfrac{1}{12}$ , and knowing the coin shows heads leaves the probability of a six at $\tfrac16$ . The events $M$ and $M^c$ are disjoint, and $P(M)\,P(M^c) = 0.40 \times 0.60 = 0.24$ is not $0$ , so they are not independent.

Law of total probability

Definition Let $B_1, \dots, B_k$ be disjoint, exhaustive events with $P(B_i) > 0$ for every $i$ . Then for every event $A$ ,

P(A) = \sum_{i=1}^{k} P(B_i)\,P(A \mid B_i).

The events $A \cap B_i$ are disjoint with union $A$ , so the addition rule gives $P(A) = \sum_i P(A \cap B_i)$ , and the multiplication rule turns each term into $P(B_i)\,P(A \mid B_i)$ . With two groups $B$ and $B^c$ , where $0 < P(B) < 1$ , the sum has the two terms $P(B)\,P(A \mid B)$ and $P(B^c)\,P(A \mid B^c)$ .

In plain words An overall rate is a weighted average of the rates within the groups, with each group weighted by its share of the population. So it lies between the smallest and the largest group rate. If the mix of groups changes, the overall rate can change even when every group rate stays the same.

Example For the visits, with the groups morning and not morning,

\begin{aligned} P(L) &= 0.40 \times 0.25 \\[0.4em] &\quad + 0.60 \times 0.20 \\[0.4em] &= 0.22. \end{aligned}

The two terms $0.10$ and $0.12$ are the joint probabilities in the long-wait column. If the clinic shifted to $70$ percent morning visits and the two group rates stayed at $0.25$ and $0.20$ , the overall rate would become $0.70 \times 0.25 + 0.30 \times 0.20 = 0.235$ . Whether the group rates would really stay fixed after such a change is an assumption about the clinic, not a consequence of the formula.

Check yourself

A constructed record of $200$ orders has $50$ express orders, $5$ of which arrived late, and $150$ standard orders, $30$ of which arrived late. One order is chosen at random, with all $200$ equally likely. Find the probability that it is express and late, the probability that it is late, the probability that it is late given that it is express, and the probability that it is express given that it is late.

Show the answer

Let $E$ be the event of an express order and $L$ the event of a late one. The joint probability is $P(E \cap L) = 5/200 = 0.025$ , and the marginal probability is $P(L) = 35/200 = 0.175$ . Within the $50$ express orders, $P(L \mid E) = 5/50 = 0.10$ . Within the $35$ late orders,

P(E \mid L) = \frac{5}{35} = \frac{1}{7},

about $0.143$ . The two conditional probabilities share the numerator $5$ but have different denominators, and swapping them is a common mistake. Ten percent of express orders arrived late, while about $14.3$ percent of late orders were express.

In a probability model, $P(A) = 0.3$ , $P(B) = 0.5$ and $P(A \cap B) = 0.15$ . Are $A$ and $B$ independent? Are they disjoint? Find $P(A \cup B)$ and $P(A \mid B^c)$ .

Show the answer

They are independent, since $P(A)\,P(B) = 0.3 \times 0.5 = 0.15 = P(A \cap B)$ . They are not disjoint, because their intersection has probability $0.15$ , not $0$ . The addition rule gives $P(A \cup B) = 0.3 + 0.5 - 0.15 = 0.65$ . Next, $P(A \cap B^c) = 0.3 - 0.15 = 0.15$ and $P(B^c) = 0.5$ , so

P(A \mid B^c) = \frac{0.15}{0.5} = 0.3,

which equals $P(A)$ , as the independence of $A$ and $B^c$ requires. Independent events with positive probabilities always overlap, so they are never disjoint.

A factory has three machines. Machine 1 makes $50$ percent of the parts with defect rate $0.01$ , machine 2 makes $30$ percent with defect rate $0.02$ , and machine 3 makes $20$ percent with defect rate $0.05$ . Find the probability that a randomly chosen part is defective. Why is the answer not the simple average of the three rates?

Show the answer

Let $D$ be the event that the part is defective. The machines split the parts into three disjoint, exhaustive groups, so the law of total probability gives

\begin{aligned} P(D) &= 0.5 \times 0.01 \\[0.4em] &\quad + 0.3 \times 0.02 \\[0.4em] &\quad + 0.2 \times 0.05 \\[0.4em] &= 0.021. \end{aligned}

The simple average $(0.01 + 0.02 + 0.05)/3 \approx 0.0267$ would treat the machines as if each made a third of the parts. Machine 3 has the highest rate but makes only $20$ percent of the parts, so its rate gets the weight $0.2$ . Even so, machine 3 supplies $0.010/0.021 = 10/21$ , about $0.48$ , of the defect probability, which shows again that a rate within a group and a share of a total are different quantities.

Conditional Probability and Independence

Key terms

Joint and marginal probability

Conditional probability

Multiplication rule

Independent events

Law of total probability

Check yourself

How this lesson connects

Builds on

Used later in