Prior, Likelihood and Posterior

Definition Suppose a model allows a finite list of hypotheses $H_1, \dots, H_k$, exactly one of which is true within the model, and each hypothesis fixes a probability model for the data, for example by fixing the value of an unknown parameter. A prior distribution assigns each hypothesis a probability $P(H_i) \ge 0$ before the data at hand are used, with

The prior is a stated input of the analysis. It may come from earlier evidence, from knowledge of the process or from a deliberate convention such as equal weights, and its basis should be reported with the result.

Definition Let $D$ be the observed data. The likelihood of a hypothesis $H$ in the list is $P(D \mid H)$, the probability that $H$ gives to the data actually observed, read as a function of the hypothesis while the data stay fixed. Provided the normalizing constant $P(D)$ of the next card is positive, the posterior distribution assigns each hypothesis $H$ in the list the probability

This is Bayes’ theorem applied to each hypothesis. For continuous data the probabilities of the data are replaced by densities. The posterior probabilities are nonnegative and add to $1$, and they are conditional on the model, the prior and the data together.

Definition The normalizing constant of the update is the total probability of the observed data under the model,

with the sum running over the $k$ hypotheses in the list, by the law of total probability. It is the probability the model, prior included, gave to the data before they were observed, and it is also called the marginal likelihood or the evidence. It is the same number for every hypothesis, so the posterior is often written $P(H \mid D) \propto P(D \mid H)\,P(H)$, where $\propto$ means equal up to a positive factor that is the same for every hypothesis. The update needs $P(D) > 0$. If every hypothesis with positive prior probability gives the observed data probability $0$, the update is not defined, and the model itself needs another look.

Definition Bayesian updating is the passage from a prior distribution to a posterior distribution by conditioning on observed data within a stated model. With finitely many hypotheses it has three steps. Multiply each prior probability by the likelihood of the data under that hypothesis, add the products, and divide each product by the sum. When data arrive in two parts $D_1$ and then $D_2$, the posterior after $D_1$ serves as the prior for $D_2$, and the likelihood for the second step is $P(D_2 \mid H_i, D_1)$. If the two parts are conditionally independent given each hypothesis, that likelihood is simply $P(D_2 \mid H_i)$. Updating in two steps then gives the same posterior as one update with the likelihood $P(D_1 \mid H_i)\,P(D_2 \mid H_i)$ of both parts together, provided each observation is used exactly once.

Definition For two hypotheses $H_1$ and $H_0$ with $P(H_0) > 0$, the prior odds of $H_1$ against $H_0$ are $P(H_1)/P(H_0)$. After data $D$ with $P(H_0 \mid D) > 0$, the posterior odds of $H_1$ against $H_0$ are $P(H_1 \mid D)/P(H_0 \mid D)$. Odds can be any number from $0$ upward. The odds of $H_0$ against $H_1$ are the reciprocal, when $P(H_1) > 0$. If $H_1$ and $H_0$ are the only hypotheses in the model, odds $O$ correspond to the probability

and the same conversion turns posterior odds into a posterior probability. With more hypotheses in the model, the number $O/(1 + O)$ is the probability of $H_1$ given that one of the two is true.

Definition For hypotheses $H_1$ and $H_0$ and data $D$ with $P(D \mid H_0) > 0$, the Bayes factor in favor of $H_1$ against $H_0$ is

If $P(H_0) > 0$, Bayes’ theorem gives

because the normalizing constant cancels from the ratio. In words, posterior odds equal prior odds times the Bayes factor. The Bayes factor of $H_0$ against $H_1$ is $1/B_{10}$ when $B_{10} > 0$. For a simple hypothesis, one that fixes every parameter, $P(D \mid H)$ is an ordinary likelihood value. For a composite hypothesis that allows a range of parameter values, $P(D \mid H)$ is the likelihood averaged over a prior for the parameter within that hypothesis, so the Bayes factor then depends on that prior too.