Conditional Independence and Sequential Evidence, introduction to probability lesson

A machine is in a high state or a low state that nobody can see, and two sensors each report positive or negative. If both sensors report positive, how far should the probability of the high state move, and may the two reports simply be treated as independent? The answer depends on how the reports are related to the state and to each other. This lesson separates independence given the state from independence overall, shows how a shared hidden state can make reports dependent, and updates the state probability one report at a time, including the case of a report that only repeats another. The models on this page are constructed for illustration.

Key terms

Conditional independence

Definition Let $C$ be an event with $P(C) > 0$ . Events $A$ and $B$ are conditionally independent given $C$ if

\begin{aligned} &P(A \cap B \mid C) \\[0.4em] &\quad = P(A \mid C)\,P(B \mid C). \end{aligned}

When $P(B \cap C) > 0$ , this is equivalent to $P(A \mid B \cap C) = P(A \mid C)$ , so once $C$ is known, learning $B$ does not change the probability of $A$ . For a state with values $H$ and $L$ , each of positive probability, $A$ and $B$ are conditionally independent given the state if the product rule holds both given $H$ and given $L$ . Conditional independence given $C$ neither implies nor is implied by ordinary independence of $A$ and $B$ .

In plain words Once $C$ is known to have happened, $A$ and $B$ carry no information about each other. For two sensor reports, conditional independence given the state means that inside each state the reports behave like two independent coin flips, each with the rate that belongs to that state. The condition after the bar is part of the claim, and dropping it gives a different statement that may be false.

Example In the running sensor model the state is $H$ or $L = H^{c}$ , with $P(H) = P(L) = 0.5$ . Let $A$ be the event that the first sensor is positive and $B$ the event that the second is positive. Each sensor is positive with probability $0.8$ given $H$ and $0.2$ given $L$ , and the model assumes that the two reports are conditionally independent given the state. Then

\begin{gathered} P(A \cap B \mid H) = 0.8^2 = 0.64, \\[0.4em] P(A \cap B \mid L) = 0.2^2 = 0.04. \end{gathered}

In the same way, a positive first report with a negative second one has probability $0.8 \cdot 0.2 = 0.16$ given $H$ and $0.2 \cdot 0.8 = 0.16$ given $L$ . Each product uses only rates from the same state.

Latent state

Definition A latent state is an unobserved quantity, such as the condition of a machine, whose value affects the probabilities of the observations. In this lesson it takes finitely many values, and the events for those values partition $\Omega$ and each have positive probability. When the conditional probabilities of two or more observations change with the value of the same latent state, the state is called a shared cause of those observations. In a probability model this structure is an assumption about how the data arise. An association between observations does not by itself prove that a particular shared cause exists.

In plain words The latent state is what the evidence is about, but it is not observed directly. Each report is a noisy reading of it, and two readings of the same hidden state are linked through it even when neither reading affects the other.

Example In the sensor model the machine state is latent, and only the two reports are observed. The state is a shared cause of the reports, since each report is positive with probability $0.8$ in state $H$ and $0.2$ in state $L$ . If the state were known to be $H$ , a positive first report would leave the second report’s chance of being positive at $P(B \mid A \cap H) = 0.8$ . When the state is unknown, a positive first report makes $H$ more probable, and that raises the chance that the second report is positive, as the calculations below show.

Marginal dependence

Definition Two events are marginally dependent if they are dependent in the overall model, in which the latent state is not conditioned on, that is, if $P(A \cap B) \ne P(A)\,P(B)$ . The marginal probabilities come from the law of total probability over the states, for example $P(A) = P(A \mid H)\,P(H) + P(A \mid L)\,P(L)$ . Suppose $A$ and $B$ are conditionally independent given a state with values $H$ and $L = H^{c}$ , where $0 < P(H) < 1$ . Writing $\delta_A = P(A \mid H) - P(A \mid L)$ and $\delta_B = P(B \mid H) - P(B \mid L)$ , a short calculation gives

\begin{aligned} &P(A \cap B) - P(A)\,P(B) \\[0.4em] &\quad = P(H)\,P(L)\,\delta_A\,\delta_B. \end{aligned}

So the two events are marginally independent exactly when at least one of them has the same probability in both states. With two states, whenever both respond to the state, conditional independence comes with marginal dependence.

In plain words While the state is unknown, a positive report from a sensor whose rate differs between the states shifts probability toward the state in which that sensor is positive more often. If the other sensor’s rate also differs between the states, that shift changes its forecast. Once the state is known, conditional independence leaves nothing for the first report to reveal about the second.

Example In the sensor model each report has marginal probability $P(A) = P(B) = 0.5 \cdot 0.8 + 0.5 \cdot 0.2 = 0.5$ , but both are positive with probability $P(A \cap B) = 0.5 \cdot 0.64 + 0.5 \cdot 0.04 = 0.34$ , which is larger than $0.5 \cdot 0.5 = 0.25$ . The full joint table is below.

	$B$	$B^{c}$
$A$	$0.34$	$0.16$
$A^{c}$	$0.16$	$0.34$

The difference $0.34 - 0.25 = 0.09$ matches $0.5 \cdot 0.5 \cdot 0.6 \cdot 0.6$ from the formula. Matching reports are more common than they would be for independent reports that are each positive half the time.

Sequential updating

Definition Sequential updating processes evidence one observation at a time. After observing $A$ , the posterior $P(H \mid A)$ serves as the prior for the next observation $B$ , and the likelihoods of $B$ must condition on everything already observed. If $P(A \cap B) > 0$ and both $H \cap A$ and $L \cap A$ have positive probability, then

\begin{aligned} &P(H \mid A \cap B) \\[0.4em] &= \frac{P(B \mid H \cap A) P(H \mid A)}{P(B \mid A)}, \end{aligned}

where $P(B \mid A) = P(B \mid H \cap A)\,P(H \mid A) + P(B \mid L \cap A)\,P(L \mid A)$ . Under conditional independence given the state, $P(B \mid H \cap A) = P(B \mid H)$ and $P(B \mid L \cap A) = P(B \mid L)$ , so the single-report likelihoods can be reused. The result always equals the one-step update $P(H \cap A \cap B)/P(A \cap B)$ , because both calculations use the same model.

In plain words Update on the first report, then treat the result as the new starting point for the second. The second report’s rates must be the ones that apply after the first report, and under conditional independence these are simply its rates in each state.

Example In the sensor model a positive first report gives

P(H \mid A) = \frac{0.5 \cdot 0.8}{0.5} = 0.8.

The forecast for the second report becomes $P(B \mid A) = 0.8 \cdot 0.8 + 0.2 \cdot 0.2 = 0.68$ , up from $0.5$ . If the second report is also positive,

P(H \mid A \cap B) = \frac{0.64}{0.68} = \frac{16}{17},

about $0.941$ . The one-step calculation agrees, since $P(H \cap A \cap B) = 0.5 \cdot 0.64 = 0.32$ and $0.32/0.34 = 16/17$ . A positive report followed by a negative one has likelihood $0.16$ under both states, so it leaves the probability of $H$ at the prior $0.5$ . That return to the prior happens here because the rates $0.8$ and $0.2$ mirror each other, and it is not a general rule.

Duplicated evidence

Definition Evidence is duplicated when a later observation repeats information that has already been counted, for example a second report that copies the first, or two reports that pass on the same underlying reading. Such observations are generally not conditionally independent given the state, so their likelihoods must not be multiplied as if they were. If the second report always equals the first, then $A \cap B = A$ and

P(H \mid A \cap B) = P(H \mid A),

so the copy changes nothing.

In plain words Hearing the same reading twice is still one reading. Multiplying the separate likelihoods of two displays, documents or messages is justified when the model makes them conditionally independent given the state, and repeated displays of one noisy reading do not qualify.

Example Suppose the second display in the sensor model only copies the first sensor. Given $H$ , two positive displays then have probability $0.8$ , not $0.64$ , and given $L$ they have probability $0.2$ , not $0.04$ . The posterior after two positive displays is

\frac{0.5 \cdot 0.8}{0.5 \cdot 0.8 + 0.5 \cdot 0.2} = 0.8,

the same as after one genuine report. Squaring the likelihoods would give $16/17 \approx 0.941$ and count the same evidence twice.

Check yourself

A new model has high-state prior $0.25$ and low-state prior $0.75$ . Given the state, two sensors are independent, and each is positive with probability $0.6$ in the high state and $0.2$ in the low state. Find the probability that both are positive, the probability of the high state after two positives, and the probability that a single sensor is positive. Are the sensors independent when the state is unknown?

Show the answer

The high state contributes $0.25 \cdot 0.6^2 = 0.09$ and the low state $0.75 \cdot 0.2^2 = 0.03$ , so $P(A \cap B) = 0.12$ and

P(H \mid A \cap B) = \frac{0.09}{0.12} = \frac{3}{4}.

Each sensor is positive with probability $0.25 \cdot 0.6 + 0.75 \cdot 0.2 = 0.3$ . The product of the marginals is $0.3^2 = 0.09$ , which is less than the actual $0.12$ , so the sensors are dependent when the state is unknown, although they are independent given the state. The gap formula for two states confirms the difference, since $P(H)\,P(L)\,\delta_A\,\delta_B = 0.25 \cdot 0.75 \cdot 0.4 \cdot 0.4 = 0.03$ . Assuming marginal independence would understate the chance of two positives.

In the same model, the first sensor is positive and the second is negative. Find the posterior probability of the high state. Does the negative report undo the positive one?

Show the answer

Given the high state the pattern has probability $0.6 \cdot 0.4 = 0.24$ , and given the low state $0.2 \cdot 0.8 = 0.16$ . The contributions are $0.25 \cdot 0.24 = 0.06$ and $0.75 \cdot 0.16 = 0.12$ , so

P(H \mid A \cap B^{c}) = \frac{0.06}{0.18} = \frac{1}{3}.

The negative report does not undo the positive one, since the result $1/3$ is above the prior $0.25$ . A positive report is $3$ times as likely in the high state as in the low state, while a negative report is only half as likely there, since $0.4/0.8 = 0.5$ , so together they still favor the high state. Sequentially, the first report gives $P(H \mid A) = 0.15/0.3 = 0.5$ , and the negative report then gives $0.5 \cdot 0.4/(0.5 \cdot 0.4 + 0.5 \cdot 0.8) = 1/3$ again.

Toss two fair coins independently. Let $A$ be the event that the first shows heads, $B$ the event that the second shows heads and $C$ the event that exactly one coin shows heads. Are $A$ and $B$ independent? Are they conditionally independent given $C$ ?

Show the answer

They are independent, since $P(A \cap B) = 1/4 = P(A)\,P(B)$ . The event $C$ consists of the two equally likely outcomes heads then tails and tails then heads, so $P(C) = 1/2 > 0$ and $P(A \mid C) = P(B \mid C) = 1/2$ . Both heads is impossible given exactly one head, so $P(A \cap B \mid C) = 0$ , which differs from $1/2 \cdot 1/2 = 1/4$ . Hence $A$ and $B$ are not conditionally independent given $C$ . Once $C$ is known, learning that the first coin shows heads forces the second to show tails. Independence does not imply conditional independence, just as conditional independence does not imply independence.

Conditional Independence and Sequential Evidence

Key terms

Conditional independence

Latent state

Marginal dependence

Sequential updating

Duplicated evidence

Check yourself

How this lesson connects

Builds on

Used later in