A Repeated-Sampling Investigation, introduction to probability lesson

In a constructed model, a delay is either 0 or 4 minutes, each with probability one half, independently from one observation to the next. What does the average of four such delays look like across many samples? The binomial distribution gives the exact answer, and a simulation can produce ten thousand averages to set beside it. This lesson carries out both, compares them honestly, and keeps the number of observations in one sample separate from the number of times the whole sample is repeated.

Key terms

Repeated sampling

Definition Fix a sampling procedure, here drawing a random sample $X_1, \dots, X_n$ from a model and computing a statistic $T = g(X_1, \dots, X_n)$ . Repeated sampling carries out the whole procedure $R$ times with fresh random inputs, giving values $T_1, \dots, T_R$ , one per repetition. When the repetitions are independent, $T_1, \dots, T_R$ are independent and identically distributed, each with the sampling distribution of $T$ .

In plain words Each repetition draws a whole new sample and records one value of the statistic. Taking one sample and copying its average thousands of times is not repeated sampling, and neither is reusing the same draws in several repetitions.

Example In the delay model each $X_i$ equals $0$ or $4$ minutes with probability $\tfrac12$ each, the observations are independent, the sample size is $n = 4$ and the statistic is

\bar{X}_4 = \frac{X_1 + X_2 + X_3 + X_4}{4}.

For this lesson the procedure was actually run $R = 10000$ times in Python 3.12 with the generator random.Random(20261014). Each uniform draw from random() below $0.5$ gave a delay of $4$ and every other draw gave $0$ . Consecutive groups of four among the 40,000 draws formed the 10,000 samples, and each sample gave one recorded mean.

Exact benchmark

Definition An exact benchmark is a quantity derived exactly from the model before the simulation output is examined, such as the distribution of the statistic, its mean and variance, or the probability of an event, and then used as the reference for that output. Agreement with a benchmark, up to simulation error, supports the claim that the program carries out the stated model but does not prove it. It does not check that the model describes a real process.

In plain words Work out what the answer should be, wherever that is possible, before looking at simulated numbers. A mismatch much larger than the standard error then points to a mistake in the program or in the benchmark calculation, and a match is evidence about the program, not about the real-world assumptions behind the model.

Example Let $K$ be the number of observations equal to $4$ in one sample. Then $K \sim \operatorname{Bin}(4, \tfrac12)$ , the total delay is $4K$ , and so $\bar{X}_4 = K$ .

$\bar{x}$	$0$	$1$	$2$	$3$	$4$
$P(\bar{X}_4 = \bar{x})$	$\tfrac{1}{16}$	$\tfrac{4}{16}$	$\tfrac{6}{16}$	$\tfrac{4}{16}$	$\tfrac{1}{16}$

One delay has mean $2$ and variance $\sigma^2 = 4$ , so $E[\bar{X}_4] = 2$ and $\operatorname{Var}(\bar{X}_4) = 4/4 = 1$ . The table agrees, since the squared deviations $4$ , $1$ , $0$ , $1$ and $4$ from $2$ , weighted by the probabilities, give $\tfrac{4 + 4 + 0 + 4 + 4}{16} = 1$ . Two benchmark event probabilities are $P(\bar{X}_4 \ge 3) = 5/16 = 0.3125$ and $P(1 \le \bar{X}_4 \le 3) = 14/16 = 7/8$ .

Empirical distribution

Definition For recorded values $t_1, \dots, t_R$ , the empirical distribution assigns to each value $v$ the relative frequency $N_v/R$ , where $N_v$ is the number of repetitions $r$ with $t_r = v$ . It is a probability distribution, with nonnegative values that add to $1$ , but it is computed from one finite run. It approximates the sampling distribution and generally differs from it.

In plain words The empirical distribution is the histogram of the output, scaled so that the bars add to one. It varies from one independent run to another, while the exact distribution stays fixed.

Example The recorded counts of the sample means $0$ to $4$ were 647, 2447, 3782, 2491 and 633, which add to 10,000.

$\bar{x}$	$0$	$1$	$2$	$3$	$4$
Count	647	2447	3782	2491	633
Empirical	0.0647	0.2447	0.3782	0.2491	0.0633
Exact	0.0625	0.25	0.375	0.25	0.0625

The largest difference is $0.0053$ , at $\bar{x} = 1$ . That is ordinary for 10,000 repetitions, since a relative frequency estimating $0.25$ has standard error $\sqrt{0.25 \cdot 0.75/10000} \approx 0.0043$ .

Empirical variance

Definition The empirical variance of recorded values $t_1, \dots, t_R$ with average $\bar{t}$ is the variance of their empirical distribution,

\begin{aligned} \widehat{\operatorname{Var}} &= \frac{1}{R} \sum_{r=1}^{R} (t_r - \bar{t})^2 \\[0.4em] &= \frac{1}{R} \sum_{r=1}^{R} t_r^2 - \bar{t}^{\,2}, \end{aligned}

with denominator $R$ . The sample variance with denominator $R - 1$ is a different statistic, so a report should say which one it uses.

In plain words The empirical variance describes the spread of the recorded output itself. It is computed like any variance, with the relative frequencies in the role of probabilities. For $R \ge 2$ the sample variance equals the empirical variance times $R/(R - 1)$ , so for large $R$ the two are close, and they are equal only when every recorded value is the same.

Example The recorded means have sum $2447 + 2 \cdot 3782 + 3 \cdot 2491 + 4 \cdot 633 = 20016$ , and their squares have sum $2447 + 4 \cdot 3782 + 9 \cdot 2491 + 16 \cdot 633 = 50122$ . Dividing by $10000$ ,

\begin{aligned} \bar{t} &= 2.0016, \\[0.4em] \widehat{\operatorname{Var}} &= 5.0122 - 2.0016^2 \\[0.4em] &= 1.00579744 . \end{aligned}

The exact values are $2$ and $1$ . The average misses $2$ by $0.0016$ , small next to the standard error $1/\sqrt{10000} = 0.01$ of an average of 10,000 independent sample means. With denominator $9999$ the variance would be about $1.00590$ .

Replication count

Definition The replication count $R$ is the number of independent repetitions of the whole experiment. It differs from the sample size $n$ , the number of observations inside one repetition. The sample size decides which distribution is being studied, for example through $\operatorname{Var}(\bar{X}_n) = \sigma^2/n$ , while $R$ decides how precisely the simulation estimates that distribution. An estimate of the probability $q$ of an event about the statistic averages $R$ independent indicators, one per repetition, so its standard error is $\sqrt{q(1 - q)/R}$ .

In plain words More observations per sample make each average less variable. More repetitions estimate the distribution of the averages more precisely. The total number of random draws, $nR$ , counts inputs, and when $n > 1$ putting it in place of $R$ in a standard error makes the simulation look more precise than it is.

Example The event $\bar{X}_4 \ge 3$ occurred in $2491 + 633 = 3124$ of the 10,000 repetitions, an estimate of $0.3124$ against the exact $0.3125$ . Its standard error is

\sqrt{\frac{(5/16)(11/16)}{10000}} \approx 0.0046 .

Treating the 40,000 draws as repetitions would give about $0.0023$ , half the correct value. Changing the sample size changes the target instead. With $n = 2$ the sample mean takes the values $0$ , $2$ and $4$ with probabilities $\tfrac14$ , $\tfrac12$ and $\tfrac14$ , and its variance is $2$ rather than $1$ .

Check yourself

Use the recorded table to estimate $P(\bar{X}_4 = 2)$ and compare it with the exact value. Does the agreement show that delays in a real service follow the 0 or 4 model?

Show the answer

The estimate is $3782/10000 = 0.3782$ , and the exact probability is $6/16 = 0.375$ . The difference $0.0032$ is smaller than the standard error

\sqrt{\frac{0.375 \cdot 0.625}{10000}} \approx 0.0048,

so the run is consistent with a program that carries out the model. It does not show that real delays follow the model. The simulation explores consequences of the stated assumptions, and whether real delays take the values 0 and 4 with equal probability, independently, has to be checked with real data. Running the same program with the same seed again would reproduce 3782 exactly, which confirms the computation but is not a new replication.

Change the probability of a 4-minute delay to $\tfrac14$ , keeping independence and four observations per sample. Find the exact distribution of $\bar{X}_4$ , its mean and variance, and $P(\bar{X}_4 \ge 3)$ . Can the recorded counts serve as a simulation of this new model?

Show the answer

The sample mean still equals the count $K$ of 4-minute delays, now with $K \sim \operatorname{Bin}(4, \tfrac14)$ . So

P(\bar{X}_4 = k) = \binom{4}{k} \frac{3^{4 - k}}{256}

for $k = 0, \dots, 4$ , which gives $81$ , $108$ , $54$ , $12$ and $1$ over $256$ . The mean is $4 \cdot \tfrac14 = 1$ and the variance is $4 \cdot \tfrac14 \cdot \tfrac34 = \tfrac34$ . The event $\bar{X}_4 \ge 3$ has probability $(12 + 1)/256 = 13/256 \approx 0.0508$ . The recorded counts cannot serve, because they come from the model with probability $\tfrac12$ . These answers are exact theoretical values, and a new simulation of the changed model would have to be run and compared with this new benchmark.

A report estimates $P(\bar{X}_4 \ge 3)$ as $0.3124$ with standard error $\sqrt{0.3124 \cdot 0.6876/40000} \approx 0.0023$ , arguing that 40,000 random draws were made. What is wrong, and what is the correct plug-in standard error?

Show the answer

The event concerns one sample mean, so there is one indicator per repetition, and the estimate averages $R = 10000$ independent indicators. The 40,000 draws are inputs, four for each indicator, and they are not separate trials of the event. The correct plug-in standard error is

\sqrt{\frac{0.3124 \cdot 0.6876}{10000}} \approx 0.0046,

twice the reported value, because $\sqrt{40000/10000} = 2$ . Using the draw count understates the Monte Carlo error.

A Repeated-Sampling Investigation

Key terms

Repeated sampling

Exact benchmark

Empirical distribution

Empirical variance

Replication count

Check yourself

How this lesson connects

Builds on