Bayesian Estimation and Credible Intervals

Definition Let $\alpha > 0$ and $\beta > 0$ be two shape parameters. A quantity $p$ with values in the interval $(0, 1)$ has the beta distribution $\operatorname{Beta}(\alpha, \beta)$ if it has the density

for $0 < p < 1$, where the constant $B(\alpha, \beta)$ is the area under $p^{\alpha - 1}(1 - p)^{\beta - 1}$ over $(0, 1)$, so that the total area is $1$. For whole numbers it equals $(\alpha - 1)!\,(\beta - 1)!/(\alpha + \beta - 1)!$. The mean and variance are

As for every density, probabilities are areas under the curve, and the height of the curve is not a probability.

Definition Suppose that, given $p$, the number $Y$ of successes in $n$ trials is $\operatorname{Binomial}(n, p)$, as it is when the trials are independent given $p$ and each succeeds with probability $p$. Write $\pi(p)$ for the prior density and $\pi(p \mid y)$ for the posterior density. If the prior is $\operatorname{Beta}(\alpha, \beta)$ and $y$ successes are observed, with $0 \le y \le n$, then the posterior density is proportional to the likelihood times the prior density,

where

The binomial coefficient $\binom{n}{y}$ does not depend on $p$ and is absorbed into the normalizing constant. The only density with this shape is a beta density, so the posterior is $\operatorname{Beta}(\alpha', \beta')$, that is, $\operatorname{Beta}(\alpha + y, \beta + n - y)$. A family of priors is called conjugate to a likelihood when the posterior always stays in the same family, and the beta family is conjugate to the binomial likelihood.

Definition The posterior mean $E(p \mid y)$ is the mean of the posterior distribution. For the $\operatorname{Beta}(\alpha + y, \beta + n - y)$ posterior it is

For $n \ge 1$ it is a weighted average of the prior mean and the sample proportion $\hat{p} = y/n$,

with weight $w = n/(\alpha + \beta + n)$ on the data. So it lies between the prior mean and $\hat{p}$, and for a fixed prior the weight on $\hat{p}$ approaches $1$ as $n$ grows. Among all single-number estimates of $p$, the posterior mean gives the smallest posterior expected squared error.

Definition A posterior median of $p$ is a value $m$ with $P(p \le m \mid y) \ge \tfrac12$ and $P(p \ge m \mid y) \ge \tfrac12$. The cumulative distribution function $F$ of a beta posterior is continuous and strictly increasing on $[0, 1]$, so the posterior median is the unique $m$ with $F(m) = \tfrac12$. Except in special cases it has no simple formula and is found numerically. Among all single-number estimates of $p$, the posterior median gives the smallest posterior expected absolute error.

Definition Let $0 < c < 1$. A credible interval for $p$ at level $c$ is an interval $[a, b]$ whose posterior probability is $c$,

The probability is computed from the posterior distribution, so it is conditional on the likelihood model, the prior and the observed data. Many intervals have the same posterior probability, so a report should name its construction. Two common ones are the equal-tailed interval of the next card and the highest posterior density interval, which keeps every value whose posterior density is above a cutoff. For a posterior density with a single peak, the highest posterior density interval is the shortest interval with probability $c$. A credible interval is defined differently from a confidence interval. A 95 percent confidence procedure is built to cover each fixed value of $p$ in 95 percent of repeated samples, exactly or approximately, while a credible interval is a probability statement about $p$ given the data, and the procedure that produces it need not have 95 percent coverage at every $p$.

Definition Let $F$ be the cumulative distribution function of a continuous posterior, and for $0 < u < 1$ let the posterior quantile $q_u$ be a value with $F(q_u) = u$. The equal-tailed credible interval at level $c$ is $[q_{(1 - c)/2}, q_{(1 + c)/2}]$. At level $0.95$ its endpoints solve

so posterior probability $0.025$ lies below the interval and $0.025$ above it. For a beta posterior the quantiles are unique, and they are usually found with a numerical quantile routine. When the posterior is skewed, the equal-tailed interval generally differs from the highest posterior density interval, and it can contain values of lower density than some values it leaves out.