Confidence and Prediction Intervals, statistics lesson

A delivery service has fitted a line relating trip time to distance, and a dispatcher asks two questions about 12 km trips. What is the average time of all such trips, and how long will the next one take? Both answers start from the same fitted value, but the second is far less certain, because a single trip varies around the average however well the average is known. This lesson builds the confidence interval for a mean response and the wider prediction interval for one new observation, and explains why neither formula makes a prediction outside the observed range trustworthy.

Key terms

Mean response

Definition In the simple linear regression model

Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i ,

where the errors have mean $0$ at every predictor value, the mean response at a predictor value $x_0$ is the conditional mean

\begin{aligned} \mu(x_0) &= E(Y \mid x = x_0) \\[0.4em] &= \beta_0 + \beta_1 x_0 . \end{aligned}

It is a fixed, unknown population quantity. It is estimated by the fitted value $\hat{y}_0 = b_0 + b_1 x_0$ , where $b_0$ and $b_1$ are the least-squares coefficients. One new observation at $x_0$ is a different target, $Y_{\text{new}} = \mu(x_0) + \varepsilon_{\text{new}}$ , because it carries its own error.

In plain words The mean response is the average outcome over all units in the population with that predictor value, such as the average time of all 12 km trips. The same fitted value serves as the estimate of that average and as the point prediction for one new trip, but the two targets come with different amounts of uncertainty.

Example Take the constructed sample with predictor values $0, 1, 2, 3, 4, 5$ and responses $2, 2, 5, 7, 8, 12$ . Least squares gives $\hat{y} = 1 + 2x$ with residuals $1, -1, 0, 0, -1, 1$ , so $\text{SSE} = 4$ . With $n - 2 = 4$ residual degrees of freedom the residual variance estimate is $s^2 = 4/4 = 1$ . The predictor mean is $\bar{x} = 2.5$ and the sum of squared predictor deviations is $S_{xx} = \sum (x_i - \bar{x})^2 = 17.5$ . At $x_0 = 2.5$ the estimated mean response is $\hat{y}_0 = 1 + 2(2.5) = 6$ , and $6$ is also the point prediction for one new observation there.

Confidence interval for a mean response

Definition Assume the model above with a correct linear mean, with predictor values treated as fixed, and with errors that are independent, have the same variance $\sigma^2$ at every $x$ and are normally distributed. Suppose $n \ge 3$ and the data contain at least two distinct predictor values. Let $s^2 = \text{SSE}/(n - 2)$ and define

h_0 = \frac{1}{n} + \frac{(x_0 - \bar{x})^2}{S_{xx}} .

Then $\operatorname{Var}(\hat{Y}_0) = \sigma^2 h_0$ , the standard error is $\operatorname{SE}(\hat{y}_0) = s\sqrt{h_0}$ , and $(\hat{Y}_0 - \mu(x_0))/(s\sqrt{h_0})$ has the $t_{n-2}$ distribution. The $1 - \alpha$ confidence interval for the mean response at $x_0$ is

\hat{y}_0 \pm t_{1 - \alpha/2,\, n-2} \, s\sqrt{h_0} .

The coverage is pointwise. For one value $x_0$ fixed in advance, the procedure covers $\mu(x_0)$ in a proportion $1 - \alpha$ of repeated samples. Intervals at several predictor values all cover at once with a smaller probability in general, and a band meant to cover the whole line at once, such as the Working-Hotelling band, uses a larger multiplier.

In plain words The term $1/n$ comes from estimating the height of the line at $\bar{x}$ , which is as hard as estimating an average from $n$ observations. The second term grows with the squared distance from $x_0$ to the center of the predictor values, because an error in the estimated slope moves the line more the farther you are from $\bar{x}$ . So the interval is narrowest at $\bar{x}$ and widens toward the edges of the data. When $x_0$ is one of the observed predictor values, $h_0$ is the hat value of an observation there. The interval describes the population average, not the spread of individual responses.

Example In the six-point sample, $t_{0.975, 4} \approx 2.776$ . At the center $x_0 = 2.5$ the distance term vanishes, so $h_0 = 1/6$ and $\operatorname{SE} = \sqrt{1/6} \approx 0.408$ . The margin is about $2.776 \cdot 0.408 \approx 1.133$ , and the 95 percent interval is about $[4.867, 7.133]$ . At the edge $x_0 = 5$ ,

h_0 = \frac{1}{6} + \frac{6.25}{17.5} = \frac{11}{21} ,

which is about $0.524$ , so $\operatorname{SE} \approx 0.724$ and the interval around $\hat{y}_0 = 11$ is about $[8.991, 13.009]$ . In a simulation of this design, the six pointwise 95 percent intervals at $x = 0, 1, \dots, 5$ all covered their means in only about 89 percent of samples.

Prediction error

Definition Let $Y_{\text{new}} = \mu(x_0) + \varepsilon_{\text{new}}$ be a new observation at $x_0$ that follows the same model independently of the fitting sample. The prediction error of the fitted value is

\begin{aligned} Y_{\text{new}} - \hat{Y}_0 &= \bigl(\mu(x_0) - \hat{Y}_0\bigr) \\[0.4em] &\quad + \varepsilon_{\text{new}} . \end{aligned}

Under the model it has mean $0$ , its two parts are independent, the first has variance $\sigma^2 h_0$ and the second has variance $\sigma^2$ , so

\begin{aligned} &\operatorname{Var}(Y_{\text{new}} - \hat{Y}_0) \\[0.4em] &\quad = \sigma^2 (1 + h_0) . \end{aligned}

The estimated standard error of prediction is $s\sqrt{1 + h_0}$ . If the target is instead the average of $m$ new observations at $x_0$ with independent errors, the new-error part has variance $\sigma^2/m$ and the variance of the prediction error is $\sigma^2 (h_0 + 1/m)$ .

In plain words Under the model, a prediction misses for two reasons. The fitted line is somewhat off from the true mean line, and the new unit varies around the true mean. More fitting data shrinks the first part, since $h_0$ falls toward $0$ when $n$ grows and the predictor values keep their spread. The second part is the population’s own variation around its mean, and no amount of fitting data removes it.

Example At $x_0 = 2.5$ in the six-point sample, the estimated variance is $s^2 h_0 = 1/6$ for the mean part and $s^2 = 1$ for the new error. The standard error of prediction is $\sqrt{7/6} \approx 1.080$ , against $0.408$ for the mean response. For the average of four new independent observations at $2.5$ it is $\sqrt{1/6 + 1/4} = \sqrt{5/12} \approx 0.645$ , between the two. If the four observations share a source of variation, such as the same driver on the same day, the term $\sigma^2/4$ understates the new-error variance.

Prediction interval

Definition Under the same model with normal errors, and for a new observation at $x_0$ that follows the model independently of the fitting sample, the ratio $(Y_{\text{new}} - \hat{Y}_0)/(s\sqrt{1 + h_0})$ has the $t_{n-2}$ distribution. The $1 - \alpha$ prediction interval for the new observation is

\hat{y}_0 \pm t_{1 - \alpha/2,\, n-2} \, s\sqrt{1 + h_0} .

The probability $1 - \alpha$ that the interval contains the new observation is computed over both the fitting sample and the new observation. At the same $x_0$ and level, the prediction interval is wider than the confidence interval for the mean response whenever $s > 0$ .

In plain words A prediction interval targets one future outcome, a random quantity, rather than a fixed parameter. It leans on the normal shape of the errors much more than the interval for the mean does. With a large sample the interval for the mean is approximately valid under milder conditions, but the new error is a single draw, so the coverage of the prediction interval depends on the shape of the error distribution at every sample size. The 95 percent is also an average over fitting samples. It does not promise that 95 percent of future observations will fall inside one particular computed interval, and it is pointwise, like the interval for the mean.

Example At $x_0 = 2.5$ the 95 percent prediction interval is about $6 \pm 2.776 \cdot 1.080$ , or $[3.001, 8.999]$ , while the confidence interval for the mean response is $[4.867, 7.133]$ . The data, the fitted line, the center and the level are the same, yet the prediction interval is $\sqrt{7} \approx 2.65$ times as wide, because it addresses a more variable target. At the edge $x_0 = 5$ the prediction interval is about $[7.573, 14.427]$ , against $[8.991, 13.009]$ for the mean.

Extrapolation

Definition Extrapolation is using a fitted model to estimate a mean response or predict an outcome at predictor values outside the range of the data used for fitting, or more broadly for a population or conditions the data did not represent. The formulas still return intervals there, and the growing $h_0$ makes them wider. That extra width accounts only for sampling variability within the assumed model. It does not account for a true relation that bends beyond the observed range or for a population that is different there.

In plain words Inside the data, a straight line that fits badly can show up as a pattern in the residuals. Outside the data there are no residuals to check, so the coverage claim rests entirely on the assumption that the line continues. A wider interval is not a validated one.

Example The six-point sample covers predictor values from $0$ to $5$ . At $x_0 = 10$ the fitted value is $21$ and

h_0 = \frac{1}{6} + \frac{56.25}{17.5} = \frac{71}{21} ,

which is about $3.381$ . The 95 percent confidence interval for the mean response is about $[15.89, 26.11]$ and the prediction interval is about $[15.19, 26.81]$ . If the true mean response levels off after $x = 5$ and stays at $11$ , the mean at $x = 10$ lies well below both intervals, and nothing in the six observations could have revealed that.

Check yourself

A fitted line relates trip time in minutes to distance in kilometers for a city’s delivery trips. Which interval answers each request, and what multiplies $\sigma^2$ in the variance it uses? The requests are the average time of all 12 km trips, the time of the next 12 km trip, and the average time of the next four 12 km trips, assuming those four trips are independent.

Show the answer

The average of all 12 km trips is a mean response, so it calls for the confidence interval with variance factor $h_0$ . The next trip is one new observation, so it calls for the prediction interval with factor $1 + h_0$ . The average of four new independent trips calls for a prediction interval for a future average, with factor $h_0 + 1/4$ . All three are centered at the same fitted value, and their widths are ordered from the mean response, narrowest, to the single trip, widest.

A least-squares fit to $20$ independent trips gives $\hat{y} = 4 + 3x$ , with time in minutes and distance in kilometers, residual SD $s = 3$ minutes, $\bar{x} = 8$ km and $S_{xx} = 180 \ \text{km}^2$ . Assuming the regression conditions hold, find the 95 percent confidence interval for the mean time of 11 km trips and the 95 percent prediction interval for one new 11 km trip, using $t_{0.975, 18} \approx 2.101$ . Explain why the first can be narrow while the second is wide.

Show the answer

The fitted value is $4 + 3 \cdot 11 = 37$ minutes, and

h_0 = \frac{1}{20} + \frac{(11 - 8)^2}{180} = 0.10 .

For the mean response, $\operatorname{SE} = 3\sqrt{0.10} \approx 0.949$ , the margin is about $2.101 \cdot 0.949 \approx 1.99$ , and the interval is about $[35.01, 38.99]$ minutes. For one new trip, the standard error of prediction is $3\sqrt{1.10} \approx 3.146$ , the margin is about $6.61$ , and the interval is about $[30.39, 43.61]$ minutes. Twenty trips pin down the average time at 11 km fairly well, but individual trips scatter around that average with an SD of about $3$ minutes, and the fitting data do nothing to reduce the next trip’s own error.

The data in the previous question cover distances from 2 to 15 km. A manager proposes two things. First, collect $2000$ trips over the same range of distances so that the prediction interval for one 11 km trip becomes about a minute wide. Second, use the current fit to predict a 40 km trip, arguing that the interval there is wide and therefore safe. Evaluate both proposals.

Show the answer

With $2000$ trips spread like the original ones, $S_{xx}$ grows to about $180 \cdot 1999/19 \approx 18{,}900 \ \text{km}^2$ and $h_0$ drops to about $0.001$ . If the new estimate $s$ is still about $3$ minutes, the margin for the mean response shrinks to about $0.18$ minutes, but the prediction margin only falls to about $1.96 \cdot 3 \approx 5.9$ minutes, because the factor $1 + h_0$ never drops below $1$ . The prediction interval stays about $11.8$ minutes wide, far from a minute, as long as individual trips scatter with an SD near $3$ minutes. For the 40 km trip the formula gives $h_0 = 1/20 + 32^2/180 \approx 5.74$ and a prediction margin of about $16.4$ minutes around $124$ minutes. That width assumes the straight line continues far past 15 km, which the data cannot check. Longer trips may use different roads or include stops, so the interval has no validated coverage there.

Confidence and Prediction Intervals

Key terms

Mean response

Confidence interval for a mean response

Prediction error

Prediction interval

Extrapolation

Check yourself

How this lesson connects

Builds on