Bootstrap Standard Errors and Intervals, statistics lesson

Six waiting times of 2, 3, 4, 5, 7 and 9 minutes have median $4.5$ minutes. For a sample mean the estimated standard error is simply $s/\sqrt{n}$ , but there is no comparably simple formula for the standard error of a median, and large-sample approximations say little about a sample of six. The bootstrap estimates the sampling variability of such a statistic by recomputing it on many samples drawn from the data themselves. This lesson builds the bootstrap distribution of the median for these six values, reads off a standard error, a bias estimate and a percentile interval, and shows what to resample and what the method cannot deliver.

Key terms

Bootstrap

Definition Let $x_1, \dots, x_n$ be independent observations from an unknown distribution $F$ , and let $T$ be a statistic computed from them. The empirical distribution $\hat{F}$ puts probability $1/n$ on each observed record. The sampling distribution of $T$ describes its values over repeated samples of size $n$ from $F$ . The nonparametric bootstrap approximates it by the bootstrap distribution, the distribution of the same statistic $T^*$ computed on samples of size $n$ drawn from $\hat{F}$ . Quantities that would require the unknown $F$ , such as the standard error or the bias of $T$ , are estimated by the same quantities under $\hat{F}$ . For tiny $n$ the bootstrap distribution can be found exactly by listing every resample, and otherwise it is approximated by simulation.

In plain words New samples from the population are not available, so the sample stands in for the population and new samples are drawn from it instead. This works when the empirical distribution resembles the population in the features that drive the statistic’s variability. The bootstrap reuses the information already in the data and adds none. It approximates a sampling distribution, and it does not guarantee that an interval built from it has its nominal coverage.

Example For the waiting times, assume the six records are independent observations from one population. Then $\hat{F}$ gives probability $\tfrac16$ to each of 2, 3, 4, 5, 7 and 9 minutes, and the observed median is $(4 + 5)/2 = 4.5$ minutes. There are $6^6 = 46656$ equally likely ordered resamples, and the medians of all of them form the exact bootstrap distribution of the median. Its values run from 2 to 9 minutes and never leave that range, because the median of resampled values always lies between the smallest and largest observation.

Bootstrap sample

Definition A bootstrap sample, or resample, $x_1^*, \dots, x_n^*$ has the same size $n$ as the data and is drawn with replacement from the observed records. Each draw picks every record with probability $1/n$ , independently of the other draws. Every resample is drawn from the original data, never from a previous resample.

In plain words Replacement is what makes a resample differ from the data. A record can appear several times or not at all, and drawing $n$ records without replacement would only return the original data in a new order. The chance that a given record is left out of a resample is $(1 - 1/n)^n$ , which is $(5/6)^6 \approx 0.3349$ for $n = 6$ and rises toward $e^{-1} \approx 0.368$ as $n$ grows. Resamples are possible results under the stand-in population, not new observations.

Example One possible resample of the waiting times is $(2, 2, 4, 7, 9, 9)$ . Its third and fourth sorted values are 4 and 7, so its median is $5.5$ minutes. Another is $(3, 3, 3, 5, 7, 9)$ , with median $(3 + 5)/2 = 4$ minutes.

Bootstrap standard error

Definition The bootstrap standard error of $T$ is the standard deviation of the bootstrap distribution of $T^*$ . In practice one draws $B$ resamples, computes $T_1^*, \dots, T_B^*$ with mean $\bar{T}^*$ and sample variance $V$ , and uses

\begin{gathered} \widehat{\operatorname{SE}}_{\text{boot}} = \sqrt{V}, \\[0.4em] V = \frac{\sum_{b=1}^{B} (T_b^* - \bar{T}^*)^2}{B - 1}. \end{gathered}

As $B$ grows, this estimate converges to the standard deviation of the exact bootstrap distribution, sometimes called the ideal bootstrap standard error.

In plain words Two different errors are involved. The sampling error from having only $n$ observations is what the bootstrap standard error estimates. The Monte Carlo error from using finitely many resamples is a numerical error, and it shrinks as $B$ grows. A larger $B$ makes the answer more stable for the given data, but it adds no information about the population. For the sample mean, the ideal bootstrap standard error equals $\hat{\sigma}/\sqrt{n}$ , where $\hat{\sigma}^2$ uses the denominator $n$ , so for $n \ge 2$ it equals $s/\sqrt{n}$ multiplied by $\sqrt{(n - 1)/n}$ .

Example Listing all 46656 resamples gives bootstrap medians with mean $1025/216 \approx 4.7454$ and standard deviation $\approx 1.3523$ minutes, so the ideal bootstrap standard error of the median is about $1.35$ minutes. A simulation with $B = 20000$ resamples, using NumPy’s default generator with seed 20261012, gives $1.3435$ , and seed 20261013 gives $1.3447$ . The difference between them is Monte Carlo variation. Recording the seed, $B$ and the resampling unit lets anyone repeat the calculation. For comparison, the mean of the same six values has $s/\sqrt{6} \approx 1.0646$ and ideal bootstrap standard error $\approx 0.9718$ .

Bootstrap bias

Definition The bootstrap estimate of the bias of $T$ is

\widehat{\text{bias}}_{\text{boot}} = \bar{T}^* - T,

the mean of the bootstrap statistics minus the observed statistic, where $\bar{T}^*$ is the exact bootstrap mean $E_*(T^*)$ or its simulated estimate. It estimates the true bias $E(T) - \theta$ , where $\theta$ is the parameter that $T$ estimates, by treating $\hat{F}$ as the population and the observed $T$ as its parameter value. That fits when $T$ is the plug-in estimate, computed from $\hat{F}$ in the same way that $\theta$ is computed from $F$ , as the sample median is for the median. Subtracting the bias estimate from $T$ gives the bias-corrected estimate $2T - \bar{T}^*$ , but the correction adds variability of its own.

In plain words The bootstrap bias shows how the statistic behaves systematically when the sample plays the role of the population. It is itself an estimate with sampling error, and it does not reveal the bias under the real population with certainty. Compare it with the bootstrap standard error before acting on it.

Example For the waiting times the estimate is

\begin{aligned} \frac{1025}{216} - 4.5 &= \frac{53}{216} \\[0.4em] &\approx 0.2454 \end{aligned}

minutes. The values above the observed median, 5, 7 and 9, are more widely spaced than those below it, 2, 3 and 4, so a resampled median that moves up tends to move farther than one that moves down. The bias estimate is small next to the bootstrap standard error of about $1.35$ minutes. The bias-corrected median would be $9 - 4.7454 \approx 4.2546$ minutes.

Percentile interval

Definition The percentile interval with nominal level $1 - \alpha$ is

\bigl[q^*_{\alpha/2}, \, q^*_{1 - \alpha/2}\bigr],

where $q^*_u$ is a $u$ quantile of the bootstrap distribution of $T^*$ . Because a bootstrap distribution is discrete, the quantile rule must be stated. The inverse cumulative distribution rule takes $q^*_u$ to be the smallest value $v$ with $F^*(v) \ge u$ , where $F^*$ is the cumulative distribution function of $T^*$ .

In plain words At 95 percent the interval runs from the 2.5th to the 97.5th percentile of the bootstrap statistics. Its coverage is approximate and rests on the bootstrap distribution being a good stand-in for the sampling distribution. It can do badly for biased estimators, parameters near a boundary, extreme statistics such as the sample maximum, and very small samples. Other bootstrap intervals, such as studentized or bias-corrected ones, adjust for some of these problems but have conditions of their own.

Example For the exact bootstrap distribution of the median, the cumulative probabilities include $F^*(2) \approx 0.0087$ , $F^*(2.5) \approx 0.0349$ , $F^*(7) \approx 0.9651$ and $F^*(8) \approx 0.9913$ . The inverse cumulative distribution rule therefore gives $q^*_{0.025} = 2.5$ and $q^*_{0.975} = 8$ , and the nominal 95 percent percentile interval is $[2.5, 8]$ minutes. Both seeded simulations give the same endpoints. The bootstrap distribution actually places probability $F^*(8) - F^*(2) \approx 0.9826$ on this interval, not $0.95$ , and with only six observations the stated level is a rough guide.

Resampling unit

Definition The resampling unit is the object drawn with replacement in each bootstrap sample. For the bootstrap to imitate the real design, it is the unit that was independently sampled, kept whole with all of its measurements. For paired measurements it is the pair, and for a cluster sample it is the whole cluster, such as a household with all its members. Time series and other dependent sequences need schemes that keep the dependence, such as resampling blocks of consecutive values.

In plain words Resample whatever was independently selected. Splitting a unit apart, by resampling the before and after columns separately or the rows inside clusters one by one, destroys dependence that was really there and changes the sampling behavior of the statistic. The choice follows the design and the target, not the layout of the spreadsheet. A permutation test is a different tool. It rearranges labels in the ways that a stated null hypothesis or the random assignment allows, to judge a test statistic, while the bootstrap resamples units to estimate the variability of an estimate.

Example Three devices are measured before and after a repair, with readings $(10, 12)$ , $(20, 21)$ and $(30, 33)$ . The differences are 2, 1 and 3, with mean difference 2. Resampling devices keeps each pair together, and the ideal bootstrap standard error of the mean difference is $\sqrt{(2/3)/3} \approx 0.47$ . Resampling the before and after columns independently treats the two readings of a device as unrelated, and the same calculation gives

\sqrt{\frac{200/3 + 74}{3}} \approx 6.85.

The wrong unit inflates the standard error more than fourteenfold here, because the large differences between devices are counted as noise in the difference.

Check yourself

Which of the following could be a bootstrap sample of the waiting times $(2, 3, 4, 5, 7, 9)$ ? First $(2, 2, 2, 2, 2, 2)$ , second $(3, 4, 5, 7, 9)$ , third $(2, 3, 4, 5, 7, 8)$ , and fourth $(9, 7, 5, 4, 3, 2)$ .

Show the answer

The first is possible. Sampling with replacement can pick the same record every time, and as an ordered draw this happens with probability $(1/6)^6$ . The second is not, because a resample has the same size as the data, six. The third is not, because 8 is not an observed value. The fourth is possible. It contains each record once, which happens for $6!/6^6 \approx 0.0154$ of the ordered resamples.

For the three observations 1, 3 and 8, find the exact bootstrap distribution of the median, the bootstrap bias estimate and the ideal bootstrap standard error. What is the 95 percent percentile interval with the inverse cumulative distribution rule?

Show the answer

There are $3^3 = 27$ equally likely ordered resamples, and the median of three values is the middle one. The median is 1 when at least two draws are 1, which happens in $3 \cdot 2 + 1 = 7$ resamples, and by symmetry it is 8 in 7 resamples, leaving 13 with median 3. The bootstrap mean is

\begin{aligned} &\frac{7 \cdot 1 + 13 \cdot 3 + 7 \cdot 8}{27} \\[0.4em] &\quad = \frac{34}{9} \approx 3.778, \end{aligned}

so the bias estimate is $34/9 - 3 = 7/9 \approx 0.778$ . The second moment is $572/27$ , so the variance is $572/27 - (34/9)^2 = 560/81$ and the bootstrap standard error is about $2.63$ .

The cumulative probability already reaches $7/27 \approx 0.26$ at 1, and it first reaches $0.975$ at 8, so the percentile interval is $[1, 8]$ , the whole range of the data. Three observations carry little information, and the bootstrap cannot add any.

Ten households are selected independently and every member of each household is measured, 34 people in all. The target is the average over households of the household mean. An analyst bootstraps by resampling the 34 people. What should be resampled instead, what does raising $B$ from 1000 to 100000 change, and how would a permutation test differ from this bootstrap?

Show the answer

The households were sampled independently, so the household is the resampling unit. Each bootstrap sample draws ten households with replacement, keeps all members of each, and computes the household-level statistic exactly as in the original analysis. Resampling people treats 34 records as independent even though members of one household tend to resemble each other, which usually makes the standard error too small. If the statistic is then computed as a mean over the resampled people, large households also get more weight, which changes the target from an average over households to an average over people.

Raising $B$ reduces Monte Carlo error, so the estimate of the bootstrap standard error becomes more stable. It does not add households or make the bootstrap distribution a better model of the population. A permutation test answers a different question. It shuffles group labels as a null hypothesis or random assignment allows, to see how a test statistic would behave if that null were true, while the bootstrap estimates how much an estimate varies from sample to sample.

Bootstrap Standard Errors and Intervals

Key terms

Bootstrap

Bootstrap sample

Bootstrap standard error

Bootstrap bias

Percentile interval

Resampling unit

Check yourself

How this lesson connects

Builds on