Posterior Prediction and Prior Sensitivity, statistics lesson

Seven of ten participants succeeded at a task, and with a $\operatorname{Beta}(2, 2)$ prior the previous lesson turned that into a $\operatorname{Beta}(9, 5)$ posterior for the success probability $p$ . Now two more participants are about to try. Their outcomes are $0$ or $1$ , and they would remain uncertain even if $p$ were known exactly, so a forecast has to carry both the uncertainty about $p$ and the randomness of each attempt. This lesson builds such forecasts, shows why two future outcomes that share one unknown $p$ are not independent, and then asks how much the conclusions depend on the prior and whether the model can reproduce the data. The data are constructed for teaching.

Key terms

Posterior predictive distribution

Definition Let $y$ be the observed data and $\tilde{y}$ future data from the same model. Assume that, given $p$ , the future data are independent of the observed data. The posterior predictive distribution gives each possible value of $\tilde{y}$ the probability

P(\tilde{y} \mid y) = E\bigl(P(\tilde{y} \mid p) \bigm| y\bigr),

the conditional probability of $\tilde{y}$ given $p$ , averaged over the posterior distribution of $p$ . With a posterior density $\pi(p \mid y)$ this average is the area under $P(\tilde{y} \mid p)\,\pi(p \mid y)$ over $0 < p < 1$ . The same distribution can be approximated by simulation. Draw $p$ from the posterior, then draw $\tilde{y}$ from the model with that $p$ , and repeat many times.

In plain words A forecast has two layers of uncertainty. The parameter $p$ is unknown, and even a known $p$ leaves each outcome random. Plugging a single estimate in for $p$ keeps the second layer and drops the first. The posterior predictive distribution keeps both, and it inherits the assumption that future participants follow the same model as the observed ones.

Example Number the observed attempts $1$ to $10$ and let $Y_{11}$ be the next outcome, with $Y_{11} = 1$ for a success. Given $p$ , the probability of a success is $p$ itself, so its posterior average is the posterior mean,

\begin{aligned} P(Y_{11} = 1 \mid y) &= E(p \mid y) \\[0.4em] &= \frac{9}{14}, \end{aligned}

about $0.643$ , and $P(Y_{11} = 0 \mid y) = 5/14$ . For this single binary outcome, plugging in the posterior mean gives the same number. That agreement comes from averaging $p$ itself, and it need not carry over to events whose probability depends on $p$ in a nonlinear way.

Conditional independence

Definition Outcomes $Y_{11}$ and $Y_{12}$ are conditionally independent given $p$ if, for every value of $p$ and all outcomes $a$ and $b$ , $P(Y_{11} = a, Y_{12} = b \mid p) = P(Y_{11} = a \mid p)\,P(Y_{12} = b \mid p)$ . Outcomes that are conditionally independent given $p$ need not be independent once $p$ is averaged over its posterior. The pair is exchangeable if its joint probabilities do not change when the order of the outcomes is swapped. Exchangeable outcomes need not be independent.

In plain words If $p$ were known, one participant’s result would say nothing about the next. But $p$ is unknown and both outcomes depend on the same $p$ . A success makes larger values of $p$ more plausible, and that raises the forecast for the next attempt.

Example Given $p$ , the chance that both $Y_{11}$ and $Y_{12}$ are successes is $p^2$ . The predictive probability is therefore the posterior average of $p^2$ , which for $\operatorname{Beta}(9, 5)$ is

\begin{aligned} E(p^2 \mid y) &= \frac{9 \cdot 10}{14 \cdot 15} = \frac{3}{7} \\[0.4em] &\approx 0.4286, \end{aligned}

whereas squaring the posterior mean gives

\bigl(E(p \mid y)\bigr)^2 = \frac{81}{196},

about $0.4133$ . The gap $3/7 - 81/196 = 3/196$ is exactly the posterior variance $\operatorname{Var}(p \mid y)$ , so it is not rounding error. The same answer comes in two steps. The first success has predictive probability $9/14$ , and if it happened the posterior would become $\operatorname{Beta}(10, 5)$ , giving the second success probability $10/15$ . The product $\tfrac{9}{14} \cdot \tfrac{10}{15}$ is $3/7$ . So $P(Y_{12} = 1 \mid Y_{11} = 1, y) = 2/3$ , which is larger than $9/14$ , and the two future outcomes are dependent. They are exchangeable, since success then failure and failure then success both have predictive probability $\tfrac{9 \cdot 5}{14 \cdot 15} = \tfrac{3}{14}$ .

Beta-binomial predictive distribution

Definition Suppose the posterior is $\operatorname{Beta}(a, b)$ and $m$ future trials are conditionally independent given $p$ , each with success probability $p$ , and independent of the observed data given $p$ . The number $K$ of future successes has the beta-binomial predictive distribution

\begin{gathered} P(K = k \mid y) \\[0.4em] = \binom{m}{k} \frac{B(a + k, b + m - k)}{B(a, b)} \end{gathered}

for $k = 0, 1, \dots, m$ , where $B$ is the constant from the beta density. Its mean is $m\bar{p}$ with $\bar{p} = a/(a + b)$ , the same as the plug-in $\operatorname{Binomial}(m, \bar{p})$ mean. Its variance is the plug-in variance $m\bar{p}(1 - \bar{p})$ multiplied by $(a + b + m)/(a + b + 1)$ , a factor larger than $1$ when $m \ge 2$ .

In plain words Given $p$ the count is binomial, and the posterior says how uncertain $p$ is. Mixing the two spreads the forecast more widely than the plug-in binomial with $p$ fixed at $\bar{p}$ , as soon as $m \ge 2$ . A simulation has to respect that structure. Draw one $p$ for the whole group of future trials, then the count. Drawing a fresh $p$ for every single trial gives independent trials that each succeed with probability $\bar{p}$ , which is the plug-in binomial again.

Example For the next $m = 2$ attempts under $\operatorname{Beta}(9, 5)$ , no successes has probability $\tfrac{5 \cdot 6}{14 \cdot 15}$ , exactly one success has probability $2 \cdot \tfrac{9 \cdot 5}{14 \cdot 15}$ , and two successes have probability $\tfrac{9 \cdot 10}{14 \cdot 15}$ , so

\begin{aligned} P(K = 0 \mid y) &= 1/7, \\[0.4em] P(K = 1 \mid y) &= 3/7, \\[0.4em] P(K = 2 \mid y) &= 3/7. \end{aligned}

The plug-in $\operatorname{Binomial}(2, 9/14)$ gives $25/196 \approx 0.128$ , $45/98 \approx 0.459$ and $81/196 \approx 0.413$ , with less weight on the two extremes. A 95 percent credible interval for $p$ is a set of values between $0$ and $1$ , while a predictive set for $K$ is a set of counts. Here the smallest count set with predictive probability at least $0.95$ is all of $\{0, 1, 2\}$ , because leaving out even the least probable count keeps only $6/7 \approx 0.857$ .

Prior sensitivity

Definition Prior sensitivity is the extent to which posterior conclusions change when the prior is replaced by other defensible priors while the data and the likelihood stay the same. A prior sensitivity analysis reports the conclusions under several stated priors, each with its reason. Sensitivity to the likelihood, such as the assumptions of a common $p$ and of independence, is a separate question that changing the prior does not address.

In plain words If several reasonable priors lead to the same practical conclusion, that conclusion does not hinge on the choice among them. If they lead to different conclusions, the data alone have not settled the question, and the report should say so. A prior is chosen for a reason given in advance, not after seeing which one gives a preferred answer. A highly concentrated prior is a strong assumption and not extra data.

Example Hold the $7$ successes in $10$ attempts fixed and vary the prior.

Prior	Posterior	Mean	$P(p > 0.5 \mid y)$
$\operatorname{Beta}(1, 1)$	$\operatorname{Beta}(8, 4)$	$0.667$	$0.887$
$\operatorname{Beta}(2, 2)$	$\operatorname{Beta}(9, 5)$	$0.643$	$0.867$
$\operatorname{Beta}(20, 20)$	$\operatorname{Beta}(27, 23)$	$0.540$	$0.716$

The posterior means are exactly $2/3$ , $9/14$ and $27/50$ . The two weak priors give similar answers. The $\operatorname{Beta}(20, 20)$ prior is concentrated near $\tfrac12$ , and in the posterior mean formula its two shape parameters add to $40$ , which leaves the ten attempts a weight of only $10/50 = 1/5$ . Whether that concentration is justified depends on what was known before the study, and the arithmetic cannot decide it.

Prior predictive check

Definition The prior predictive distribution is the distribution of the data implied by the prior and the likelihood together, before any data are observed. It gives each possible data set $y$ the probability $E\bigl(P(y \mid p)\bigr)$ , the average over the prior. A prior predictive check draws $p$ from the prior, draws a data set with the planned design from the model with that $p$ , repeats this many times, and compares summaries of the simulated data with what is known to be plausible. With a $\operatorname{Beta}(\alpha, \beta)$ prior and $n$ binomial trials, the prior predictive distribution of the count is the beta-binomial distribution of the earlier card with $a = \alpha$ , $b = \beta$ and $m = n$ .

In plain words Before trusting a prior, look at the data it expects. Parameters are hard to judge directly, but predicted data are often easy to compare with background knowledge. A prior that regularly predicts implausible data needs another look, and so do the likelihood and the measurement scale behind it.

Example For $10$ attempts the flat $\operatorname{Beta}(1, 1)$ prior gives every count from $0$ to $10$ probability $1/11$ . The $\operatorname{Beta}(2, 2)$ prior gives $9$ or $10$ successes probability about $0.108$ . The $\operatorname{Beta}(20, 20)$ prior gives $9$ or $10$ successes probability only about $0.019$ and puts about $0.848$ on counts from $3$ to $7$ . If earlier experience suggests that a good instruction card can make nearly every participant succeed, the $\operatorname{Beta}(20, 20)$ prior conflicts with that experience, and the check shows the conflict before any data are collected.

Posterior predictive check

Definition A posterior predictive check draws $p$ from the posterior, draws a replicated data set $y^{\text{rep}}$ with the same design as the observed data from the model with that $p$ , and repeats this many times. It compares a chosen test quantity $T$ , computed on each replication, with its observed value $T(y)$ , for example through the proportion of replications with $T(y^{\text{rep}}) \ge T(y)$ . The test quantity should target a feature the model might fail to capture. A check can reveal a mismatch, but it cannot prove the model correct. The posterior was fitted to the same data, so the check is not an independent test, and its tail proportion does not behave like a classical p-value in general.

In plain words Ask whether data simulated from the fitted model look like the data actually seen, in the respects that matter for the question. A total that the model was fitted to will usually look fine. A feature such as a difference between groups is a sharper test of a model that assumes one common $p$ .

Example Suppose the ten attempts came from two sites of five participants each, with $5$ successes at one site and $2$ at the other, so $T(y)$ , the absolute difference between the site counts, is $3$ . Draw $p$ from $\operatorname{Beta}(9, 5)$ , then two site counts from $\operatorname{Binomial}(5, p)$ with that same $p$ . Replicated totals are $7$ or more in about 51 percent of replications, so the total gives no warning, as expected. A site difference of $3$ or more occurs in about 8.3 percent of replications. That is somewhat unusual under one common $p$ , but ten attempts are too few to settle whether the sites differ. The check points to a question for more data or for the study design, and it does not prove the common- $p$ model either right or wrong.

Check yourself

Under the $\operatorname{Beta}(9, 5)$ posterior, find the predictive probability that the next two attempts contain exactly one success. Compare it with the plug-in binomial calculation that fixes $p = 9/14$ , and say which model each calculation uses.

Show the answer

Success then failure has predictive probability $\tfrac{9}{14} \cdot \tfrac{5}{15}$ , because after a hypothetical success the posterior would be $\operatorname{Beta}(10, 5)$ . Failure then success has the same probability $\tfrac{5}{14} \cdot \tfrac{9}{15}$ . The total is

\frac{2 \cdot 9 \cdot 5}{14 \cdot 15} = \frac{90}{210} = \frac{3}{7},

about $0.429$ . The plug-in calculation gives $2 \cdot \tfrac{9}{14} \cdot \tfrac{5}{14} = 45/98 \approx 0.459$ . The plug-in model treats $p$ as known to equal $9/14$ and the two attempts as independent. The predictive calculation averages over the posterior uncertainty about the shared $p$ , which moves probability away from the mixed result, one success and one failure, toward the two extremes, both successes or both failures.

A colleague says that the 95 percent equal-tailed credible interval $[0.386, 0.861]$ for $p$ is also a 95 percent prediction for the next participant’s outcome. What is wrong, and what is the smallest set of outcomes with predictive probability at least $0.95$ ?

Show the answer

The interval describes uncertainty about the parameter $p$ . The next outcome is $0$ or $1$ , so it never lies in $[0.386, 0.861]$ . Its predictive distribution gives probability $9/14 \approx 0.643$ to a success and $5/14 \approx 0.357$ to a failure. A set containing only one outcome has predictive probability at most $0.643$ , so the smallest set with probability at least $0.95$ is $\{0, 1\}$ . For a single binary outcome the 95 percent predictive set is trivial, and the predictive probabilities themselves are the informative summary.

Keep the data fixed at $7$ successes in $10$ attempts. Find the predictive probability that the next attempt succeeds and that the next two both succeed, under the $\operatorname{Beta}(1, 1)$ prior and under the $\operatorname{Beta}(20, 20)$ prior. If the two weaker priors $\operatorname{Beta}(1, 1)$ and $\operatorname{Beta}(2, 2)$ give similar answers, does that show the model is correct?

Show the answer

Under $\operatorname{Beta}(1, 1)$ the posterior is $\operatorname{Beta}(8, 4)$ . The next attempt succeeds with probability $8/12 = 2/3$ , and the next two both succeed with probability $\tfrac{8 \cdot 9}{12 \cdot 13} = 6/13 \approx 0.462$ . Under $\operatorname{Beta}(20, 20)$ the posterior is $\operatorname{Beta}(27, 23)$ , with probabilities $27/50 = 0.54$ and $\tfrac{27 \cdot 28}{50 \cdot 51} = 126/425 \approx 0.296$ . The $\operatorname{Beta}(2, 2)$ answers, $9/14 \approx 0.643$ and $3/7 \approx 0.429$ , sit close to the $\operatorname{Beta}(1, 1)$ answers, so the forecasts change only a little between those two priors. The $\operatorname{Beta}(20, 20)$ forecasts are much lower, so a strongly concentrated prior would change the conclusion and needs its own justification. Agreement between the weaker priors does not show the model is correct. All three analyses share the same likelihood, with one common $p$ and attempts that are independent given $p$ , and if participants at different sites have different success probabilities, the weaker priors can agree while the model they share is wrong. Agreement across reasonable priors shows stability with respect to the prior, while checks of the likelihood and the design address the rest.

Posterior Prediction and Prior Sensitivity

Key terms

Posterior predictive distribution

Conditional independence

Beta-binomial predictive distribution

Prior sensitivity

Prior predictive check

Posterior predictive check

Check yourself

How this lesson connects

Builds on

Used later in