Sampling, Estimation, and Statistical Uncertainty, machine learning lesson

Suppose the average of a file of past deliveries is used to estimate the average delay of next month’s parcels. That average is one number, but another file of the same size would have given a different number, and the file may not even represent the deliveries that matter. Before trusting the average, we need to know how much it would vary from sample to sample and whether it aims at the right target. This lesson describes that variation through the sampling distribution and the standard error, and it settles what the law of large numbers and a confidence interval can and cannot promise.

Key terms

Estimator

Definition The population is the collection of units, or the data-generating process, that a question is about, and a parameter $\theta$ is a number that describes it, such as its mean $\mu$ . A sample is the set of observations $Y_1, \dots, Y_n$ obtained from the population by a stated sampling process. An estimator of $\theta$ is a rule $\hat{\theta} = g(Y_1, \dots, Y_n)$ , fixed before the data are seen, that turns a sample into a number. Because the sample is random, the estimator is a random variable, and the number it returns for the observed sample is an estimate. The sampling distribution of $\hat{\theta}$ is its probability distribution over all the samples the process could produce, with the population, the sample size $n$ and the sampling design held fixed. The estimator is unbiased for $\theta$ under a model if $E[\hat{\theta}] = \theta$ for every distribution the model allows. The sample mean

\bar{Y} = \frac{1}{n}\sum_{i=1}^{n} Y_i

is unbiased for $\mu$ whenever every $Y_i$ has mean $\mu$ , by linearity of expectation.

In plain words The estimator is the recipe, and the estimate is the number the recipe produced from one data set. The sampling distribution lists every number the recipe could have produced under the same process, with its probability. It is a different distribution from the distribution of individual observations. Unbiased means right on average over repeated samples, not right in each sample.

Example In a constructed population, each delivery is delayed $0$ minutes or $10$ minutes, each with probability $1/2$ . Its mean is $\mu = 5$ minutes, and since both outcomes lie $5$ minutes from the mean, its variance is $\sigma^2 = 25$ minutes squared and its standard deviation is $\sigma = 5$ minutes. These are constructed numbers, not measured delays. The observed sample $0, 10, 10, 10$ gives the estimate $\bar{y} = 7.5$ minutes. With four independent deliveries there are $16$ equally likely ordered samples. When $k$ of the four delays are ten minutes the mean is $10k/4$ , and this happens in $\binom{4}{k}$ of the samples.

Sample mean in minutes	$0$	$2.5$	$5$	$7.5$	$10$
Number of samples	$1$	$4$	$6$	$4$	$1$

The expectation of this sampling distribution is $(0 + 10 + 30 + 30 + 10)/16 = 5$ minutes, so the sample mean is unbiased here, yet $10$ of the $16$ samples give an estimate other than $5$ .

Independent and identically distributed

Definition Random variables $Y_1, \dots, Y_n$ are identically distributed if they all have the same probability distribution, so that $P(Y_i \in A) = P(Y_1 \in A)$ for every $i$ and every set $A$ of values. They are independent if, for all sets $A_1, \dots, A_n$ ,

\begin{aligned} &P(Y_1 \in A_1, \dots, Y_n \in A_n) \\[0.4em] &\quad = \prod_{i=1}^{n} P(Y_i \in A_i) . \end{aligned}

For discrete variables it is enough to check single values, $A_i = \{y_i\}$ . The two conditions together are abbreviated iid, and neither one implies the other. The variables are exchangeable if, for every permutation $\pi$ of the indices $1, \dots, n$ , the rearranged vector $(Y_{\pi(1)}, \dots, Y_{\pi(n)})$ has the same joint distribution as $(Y_1, \dots, Y_n)$ . Independent and identically distributed variables are exchangeable, and exchangeable variables are identically distributed, but exchangeable variables need not be independent.

In plain words Identically distributed is a statement about each observation on its own. Independence is a statement about how the observations relate, and it says that learning the values of some of them does not change the probabilities for the others. Random splits of rows into training, validation and test data rely on this picture of independent rows from one distribution, so rows that share a customer, a device or a time period need splits that keep those groups together or respect time order.

Example Drawing a fresh zero-or-ten delay for each of four parcels gives iid observations. Now draw one delay $Z$ for a van and give it to all four parcels on the van, so $Y_1 = Y_2 = Y_3 = Y_4 = Z$ . Each parcel still has delay $0$ or $10$ with probability $1/2$ , so the four are identically distributed and exchangeable. They are not independent, since $P(Y_1 = 0, Y_2 = 0) = 1/2$ instead of $1/2 \cdot 1/2 = 1/4$ . Drawing two records without replacement from a file of four deliveries with delays $0$ , $0$ , $10$ and $10$ gives another exchangeable pair that is not independent. The second record is $10$ with probability $1/2$ overall, but with probability $1/3$ when the first is $10$ . In the other direction, delays from two services drawn independently, one with outcomes $0$ or $10$ and one with outcomes $0$ or $20$ , each with probability $1/2$ , are independent but not identically distributed.

Standard error

Definition The standard error of an estimator is the standard deviation of its sampling distribution, for a stated population, sample size and sampling design. Let $Y_1, \dots, Y_n$ be independent with common mean $\mu$ and common finite variance $\sigma^2$ , as in iid sampling from a population with finite variance. Independence makes the variance of the sum equal to $n\sigma^2$ , and dividing the sum by $n$ divides its variance by $n^2$ , so

\begin{gathered} E[\bar{Y}] = \mu, \\[0.4em] \operatorname{Var}(\bar{Y}) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}, \\[0.4em] \operatorname{SE}(\bar{Y}) = \frac{\sigma}{\sqrt{n}} . \end{gathered}

Without independence the covariances stay in the variance of the sum, as in Expectation, Variance, and Covariance, and

\begin{aligned} &\operatorname{Var}(\bar{Y}) = \frac{1}{n^2}\Bigl(\sum_{i} \operatorname{Var}(Y_i) \\[0.4em] &\quad + 2\sum_{i<j} \operatorname{Cov}(Y_i, Y_j)\Bigr) . \end{aligned}

When $\sigma$ is unknown, it is usually replaced by the sample standard deviation

s = \sqrt{\frac{1}{n - 1}\sum_{i=1}^{n} (Y_i - \bar{Y})^2} ,

defined for $n \ge 2$ , which gives the estimated standard error $\widehat{\operatorname{SE}}(\bar{Y}) = s/\sqrt{n}$ . Under iid sampling with finite variance the sample variance is unbiased, $E[s^2] = \sigma^2$ , but $s$ is in general not unbiased for $\sigma$ , and the estimated standard error is itself a statistic with its own sampling distribution.

In plain words The standard deviation $\sigma$ says how much individual delays vary. The standard error says how much the average of $n$ delays would vary if the whole sampling were repeated. More independent observations shrink the standard error like $1/\sqrt{n}$ , so quadrupling $n$ halves it, while $\sigma$ stays the same. Rows that copy one another add no independent information.

Example For the delay population and $n = 4$ , the standard error is $5/\sqrt{4} = 2.5$ minutes, which matches the sampling distribution in the table above, whose variance is $6.25$ . Sixteen independent deliveries give $5/4 = 1.25$ minutes. For the observed sample $0, 10, 10, 10$ the squared deviations from $7.5$ are $56.25$ , $6.25$ , $6.25$ and $6.25$ , which sum to $75$ , so $s^2 = 75/3 = 25$ , $s = 5$ and the estimated standard error is $2.5$ minutes. The match with the true value is luck. The samples $0, 0, 0, 0$ and $10, 10, 10, 10$ , with total probability $1/8$ , give $s = 0$ and an estimated standard error of $0$ . Over all $16$ samples $s^2$ averages exactly $25$ , while $s$ averages about $4.67$ . In the shared-van file the four-row mean equals the van delay $Z$ , so its standard error is $5$ minutes, not $2.5$ .

Law of large numbers

Definition Let $Y_1, Y_2, \dots$ be iid with mean $\mu$ and finite variance $\sigma^2$ , and let $\bar{Y}_n$ be the mean of the first $n$ of them. The weak law of large numbers says that $\bar{Y}_n$ converges in probability to $\mu$ , which means that for every fixed $\varepsilon > 0$

\lim_{n \to \infty} P\bigl(\lvert \bar{Y}_n - \mu \rvert \ge \varepsilon\bigr) = 0 .

A proof notes that $(\bar{Y}_n - \mu)^2 \ge \varepsilon^2$ on the event $\lvert \bar{Y}_n - \mu \rvert \ge \varepsilon$ and $(\bar{Y}_n - \mu)^2 \ge 0$ everywhere, so the expectation of $(\bar{Y}_n - \mu)^2$ is at least $\varepsilon^2$ times the probability of that event. Since $E[\bar{Y}_n] = \mu$ , that expectation is $\operatorname{Var}(\bar{Y}_n) = \sigma^2/n$ , and dividing by $\varepsilon^2$ gives Chebyshev’s bound

\begin{aligned} &P\bigl(\lvert \bar{Y}_n - \mu \rvert \ge \varepsilon\bigr) \\[0.4em] &\quad \le \frac{E[(\bar{Y}_n - \mu)^2]}{\varepsilon^2} = \frac{\sigma^2}{n\varepsilon^2} , \end{aligned}

which tends to $0$ as $n$ grows. These conditions are sufficient, not necessary. For iid observations a finite mean is already enough, and other versions allow some dependence or observations that are not identically distributed. The law is a statement about a limit and gives no exact probability at any finite $n$ .

In plain words Pick any tolerance around $\mu$ . With enough iid observations, an average that misses by at least that tolerance becomes as unlikely as you like. The law does not say that each new average is closer than the last one, the law alone does not say how large $n$ must be in a given problem, and it does not make the average settle near the mean of a population the data did not come from.

Example For the delay population $\sigma^2 = 25$ , so with tolerance $\varepsilon = 1$ minute the bound is $25/n$ . At $n = 2500$ it is $0.01$ , so the mean of $2500$ independent delays misses $\mu = 5$ by at least one minute with probability at most $0.01$ . The exact probability is about $1.3 \times 10^{-23}$ , so the bound is valid but very loose. At $n = 100$ the bound is $0.25$ and the exact probability is about $0.0569$ . At $n = 4$ the bound is $6.25$ , which says nothing, and the exact probability is $10/16 = 0.625$ , because only the samples with mean $5$ miss by less than a minute.

Confidence interval

Definition A confidence interval procedure at level $1 - \alpha$ for a parameter $\theta$ is a rule that turns each possible sample into an interval $[L, U]$ whose endpoints are statistics, such that under the stated model

P(L \le \theta \le U) = 1 - \alpha

for every allowed value of the parameters. The probability is over repeated samples, with $\theta$ fixed and the endpoints random. A conservative procedure guarantees at least $1 - \alpha$ , and an approximate one reaches $1 - \alpha$ only approximately. The probability that the interval contains $\theta$ under the process that actually produced the data is its coverage probability. For iid observations with known $\sigma > 0$ , the normal-based 95 percent interval for $\mu$ is

\bar{Y} \pm 1.96 \, \frac{\sigma}{\sqrt{n}} .

It is exact, apart from rounding $1.96$ , when the observations are normal. Otherwise its justification is the central limit theorem, which says that for iid observations with finite variance $\sigma^2 > 0$

\frac{\bar{Y}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0, 1) ,

meaning that the distribution function of the standardized mean converges to the standard normal one at every point. This is a limit statement about the standardized mean, not about individual observations. How accurate it is at a given $n$ depends on the population, so no single sample size works for every case. When $\sigma$ is estimated by $s$ , the Student t interval $\bar{Y} \pm t\, s/\sqrt{n}$ , with $t$ the $0.975$ quantile of the t distribution with $n - 1$ degrees of freedom, is exact for iid normal observations, but not for arbitrary nonnormal or dependent data.

In plain words The 95 percent belongs to the recipe, not to one interval. If the conditions hold, the long-run proportion of intervals from repeated samples that contain the fixed $\mu$ is 95 percent. Once the data are in, a computed interval either contains $\mu$ or does not. The label is a target that the assumptions have to justify, and the interval is about the mean, not about where 95 percent of individual delays fall.

Example Take the delay population with known $\sigma = 5$ and $n = 4$ . The standard error is $2.5$ , so the half-width is $1.96 \cdot 2.5 = 4.9$ minutes. The intervals centered at $2.5$ , $5$ and $7.5$ contain $\mu = 5$ , and those centered at $0$ and $10$ miss it. The covering samples number $4 + 6 + 4 = 14$ of $16$ , so the probability that the random interval $I$ contains $5$ is

P(5 \in I) = \frac{14}{16} = 0.875 ,

not $0.95$ . With $n = 100$ the half-width is $0.98$ , the interval contains $5$ exactly when the number $k$ of ten-minute delays is from $41$ to $59$ , and the binomial probabilities of those $k$ add up to a coverage of about $0.9431$ . Coverage need not move steadily toward $0.95$ , since it is about $0.9561$ at $n = 99$ and about $0.9540$ at $n = 101$ . A simulation estimates the same coverage by repeating the sampling.

import numpy as np

rng = np.random.default_rng(18019)
n, reps = 4, 20_000
samples = rng.choice([0, 10], size=(reps, n))
means = samples.mean(axis=1)
half = 1.96 * 5 / np.sqrt(n)
print(np.mean(np.abs(means - 5) <= half))

One run printed $0.87205$ . The Monte Carlo standard error of such a proportion is about $\sqrt{0.875 \cdot 0.125 / 20{,}000} \approx 0.0023$ , so the simulation agrees with the exact $0.875$ to within about $1.3$ Monte Carlo standard errors. More repetitions make the estimate of the coverage more precise, while a larger $n$ changes the coverage itself.

Selection bias

Definition Selection bias is a systematic error that arises when the process deciding which units or outcomes enter the data is related to the outcomes being measured, so that the recorded observations follow a different distribution from the target population. Let $S = 1$ mean that a unit is recorded, and suppose the records are iid draws from the distribution of $Y$ given $S = 1$ , with finite variance. Then the mean of $n$ records has expectation $E[Y \mid S = 1]$ and, by the law of large numbers, converges in probability to it rather than to $\mu = E[Y]$ . As an estimator of $\mu$ its bias at every sample size is

E[Y \mid S = 1] - \mu .

If recording is independent of $Y$ , this difference is $0$ . A larger sample, a smaller standard error or a narrower interval does not remove the bias, since all of them describe variation around $E[Y \mid S = 1]$ .

In plain words More data from the wrong process give a more precise answer to a different question. Before trusting a small standard error, ask how the data were collected and which cases could not appear in them. The same problem arises when a model is trained or evaluated on records that were filtered differently from the cases it will meet in use.

Example Suppose a logging system in the delay population records only the deliveries that had no delay. Then $E[Y \mid S = 1] = 0$ , and the mean of the records is $0$ at every sample size, $5$ minutes below $\mu$ . Every record is $0$ , so $s = 0$ , the estimated standard error is $0$ , and the interval $\bar{y} \pm 1.96\, s/\sqrt{n}$ is the single point $0$ , which never contains $\mu = 5$ . The interval looks perfectly certain and is always wrong.

Check yourself

A constructed delay population has standard deviation $\sigma = 8$ minutes, and deliveries are sampled independently. Find the standard error of the mean for $n = 16$ and for $n = 64$ , and the smallest $n$ that makes it at most $0.5$ minutes. A report on $64$ deliveries then says that individual delays vary by only about $1$ minute. What is wrong with that sentence?

Show the answer

The standard errors are $8/\sqrt{16} = 2$ minutes and $8/\sqrt{64} = 1$ minute, so quadrupling $n$ halves the standard error. The condition $8/\sqrt{n} \le 0.5$ means $\sqrt{n} \ge 16$ , so $n \ge 256$ .

The figure of $1$ minute is the standard error of the mean of $64$ deliveries. It describes how much that average would vary over repeated samples. Individual delays still have standard deviation $8$ minutes at every sample size, so a correct report says that individual delays have standard deviation $8$ minutes and that their mean has standard error $1$ minute.

A delay population has standard deviation $\sigma = 10$ minutes. A file holds $80$ parcel rows from $16$ independently sampled vans, five parcels per van, and every parcel on a van has exactly that van’s delay. The rows average $42$ minutes. An analyst reports $42 \pm 1.96 \cdot 10/\sqrt{80}$ as a 95 percent confidence interval for the mean delay. Find the standard error under this model and the corrected normal-based interval. What does the corrected calculation still not establish?

Show the answer

Write $Z_g$ for the delay of van $g$ . Each van contributes five copies of $Z_g$ , so the row average is the average of $16$ independent van delays,

\bar{Y} = \frac{1}{80}\sum_{r=1}^{80} Y_r = \frac{1}{16}\sum_{g=1}^{16} Z_g ,

and its standard error is $10/\sqrt{16} = 2.5$ minutes. The analyst’s $10/\sqrt{80} \approx 1.118$ treats the $80$ rows as independent and is too small by a factor of $\sqrt{5} \approx 2.24$ . The analyst’s interval, here $[39.81, 44.19]$ , comes from a procedure that adds about $2.19$ minutes on either side of the row average. Even if the van delays were normal, that procedure would cover the true mean in only about 62 percent of repeated samples, since that half-width is $1.96/\sqrt{5} \approx 0.877$ true standard errors and a standard normal variable lies within $0.877$ of $0$ with probability about $0.62$ .

The corrected half-width is $1.96 \cdot 2.5 = 4.9$ minutes, so the interval is $[37.1, 46.9]$ . Its 95 percent label still needs a justification, either normal van delays or a normal approximation that is adequate for a mean of $16$ independent vans, and the vans must be a random sample of the vans the question is about. The interval also concerns the mean delay, not the range that holds 95 percent of individual parcel delays.

In the constructed delay population, with delays of $0$ or $10$ minutes each with probability $1/2$ , a logging system records every zero-minute delivery but records each ten-minute delivery only with probability $1/2$ , independently of everything else. An analyst treats the records as iid draws and reports $\bar{y} \pm 1.96\, s/\sqrt{n}$ for the mean delay. What does the mean of the records converge to, and what happens to the probability that the interval contains $\mu = 5$ as the number of records grows?

Show the answer

Recorded deliveries make up $1/2 + 1/4 = 3/4$ of all deliveries, and the ten-minute ones are $(1/4)/(3/4) = 1/3$ of the records. The records are therefore iid with mean

\begin{aligned} E[Y \mid S = 1] &= 10 \cdot \tfrac{1}{3} \\[0.4em] &= \tfrac{10}{3} \approx 3.33 \end{aligned}

minutes, and by the law of large numbers their average converges in probability to $10/3$ , not to $5$ . The selection bias is $10/3 - 5 = -5/3 \approx -1.67$ minutes at every sample size.

The recorded delays have standard deviation $10\sqrt{2}/3 \approx 4.71$ minutes, so with $400$ records the standard error is about $0.236$ minutes and a typical interval is about $3.33 \pm 0.46$ . Exact binomial calculations put the probability that the interval contains $5$ at about $0.63$ for $25$ records, about $0.066$ for $100$ records and below one in a million for $400$ records. The coverage of the true mean goes to zero as the data grow, because the interval narrows around the wrong value. The remedy is knowledge of the logging process, not more records. In this model, weighting each ten-minute record by $2$ , because only half of those deliveries are logged, makes the weighted mean converge to $5$ .

Sampling, Estimation, and Statistical Uncertainty

Key terms

Estimator

Independent and identically distributed

Standard error

Law of large numbers

Confidence interval

Selection bias

Check yourself

How this lesson connects

Builds on

Used later in