Confidence Intervals and Coverage

Definition Let $\theta$ be a parameter of a statistical model. An interval procedure is a rule that turns each possible data set into an interval $[L, U]$, whose endpoints $L$ and $U$ are statistics computed from the data. The procedure gives confidence intervals for $\theta$ at confidence level $1 - \alpha$ if, under the model,

for every allowed value of $\theta$ and of any other parameters in the model, with the probability computed over random samples while the parameters stay fixed. Many practical procedures reach this only approximately, and some guarantee at least $1 - \alpha$. The interval computed from the observed data is also called a confidence interval. It is one outcome of the procedure. The simplest example assumes independent observations from $N(\mu, \sigma^2)$ with $\sigma$ known, and uses

where $z_{0.975} \approx 1.959964$ is the $0.975$ quantile of the standard normal distribution. This is the 95 percent z interval for $\mu$.

Definition The confidence level $1 - \alpha$, often written as a percentage, is the coverage probability that an interval procedure is designed to have. It is chosen before the data are seen. For the z interval, level $1 - \alpha$ uses the critical value $z_{1 - \alpha/2}$, the number with $P(-z_{1 - \alpha/2} \le Z \le z_{1 - \alpha/2}) = 1 - \alpha$ when $Z \sim N(0, 1)$, and the interval is

Definition The coverage probability of an interval procedure at a parameter value $\theta$ is the probability, computed under the model with $\theta$ and any other parameters fixed, that the random interval $[L, U]$ contains $\theta$. For the z interval, with $z = z_{1 - \alpha/2}$, the interval contains $\mu$ exactly when the standardized sample mean lies between $-z$ and $z$,

Under independent normal sampling with known $\sigma$ the middle quantity has the $N(0, 1)$ distribution, so the coverage is exactly $1 - \alpha$ for every value of $\mu$. The level a procedure is designed for is its nominal coverage. The actual coverage under the process that really produced the data can differ when the assumptions fail.

Definition For an interval of the form estimate $\pm\, m$, the margin of error $m$ is the half-width, and the full width is $2m$. For the z interval

It grows with the confidence level and with $\sigma$, and it is proportional to $1/\sqrt{n}$. It reflects only random sampling variation under the model. It does not include bias from how the data were collected, such as selection of the sample, nonresponse or measurement error.

Definition Imagine repeating the whole study many times independently with the parameter fixed, each time drawing a new sample and computing an interval by the same rule. The repeated-sampling interpretation says that the long-run proportion of these intervals that contain $\theta$ equals the coverage probability, which is $1 - \alpha$ for an exact procedure whose assumptions hold. Once the data are observed, a computed interval such as $[48.08, 55.92]$ either contains the fixed $\mu$ or does not, and the 95 percent describes the method that produced it. A probability that $\mu$ lies in a particular interval treats $\mu$ as uncertain, and it is the kind of statement a Bayesian credible interval makes from a prior and the data.