Choosing Exact, Approximate or Simulated Answers, introduction to probability lesson

Twelve independent trials each succeed with probability $0.25$ . What is the probability of at least four successes? A binomial sum answers the question exactly, a normal curve answers it approximately and a simulation estimates it, and the three numbers differ. This lesson compares the three methods on one fixed model and event, explains what each kind of error means, and shows which answer to report and how to describe it.

Key terms

Exact model probability

Definition The exact model probability of an event is the value that the stated model assigns to it, written as an expression that follows from the model’s assumptions without approximation. A decimal shown for it is often rounded, with an error of at most half a unit in the last digit shown. Exactness refers to the model only. It does not show that the model describes the real process.

In plain words Within a model there is one correct number for each event, and an exact expression is that number. Whether the model’s assumptions, such as independent trials with a common success probability, hold for real trials is a separate question that no calculation inside the model can settle.

Example Let $X \sim \operatorname{Bin}(12, 0.25)$ and $p_* = P(X \ge 4)$ , where $p_*$ is the target probability and $0.25$ is the success probability of each trial. The complement needs only the four masses

p_X(k) = \binom{12}{k} \frac{3^{12 - k}}{4^{12}}

for $k = 0, 1, 2, 3$ , and

\begin{aligned} p_* &= 1 - \sum_{k=0}^{3} p_X(k) \\[0.4em] &= \frac{5892517}{16777216} \\[0.4em] &\approx 0.351221. \end{aligned}

Approximation error

Definition An approximation replaces the model’s distribution, or a step of the calculation, by something simpler, for example a normal law with the same mean and variance. If it gives the value $a$ for an event whose exact model probability is $p_*$ , its approximation error is $a - p_*$ . For a given model, event and method this error is one fixed number, not a random quantity. Its size depends on the parameters and on the event, and the limit theorem behind a method does not give its size at a fixed $n$ .

In plain words An approximation answers a nearby question about a simpler distribution. Repeating it gives the same number with the same error. Its size becomes known through a comparison with an exact value, and a theorem with an explicit error bound can limit it without giving it exactly.

Example For $\operatorname{Bin}(12, 0.25)$ the mean is $3$ , the variance is $12 \cdot 0.25 \cdot 0.75 = 2.25$ and the standard deviation is $1.5$ . With the continuity correction, $X \ge 4$ becomes $Y > 3.5$ for $Y \sim N(3, 2.25)$ , and $(3.5 - 3)/1.5 = \tfrac13$ , so

\begin{aligned} a &= P(Y > 3.5) \\[0.4em] &= 1 - \Phi\bigl(\tfrac13\bigr) \\[0.4em] &\approx 0.369441. \end{aligned}

The approximation error is $a - p_* \approx 0.018220$ , about $1.82$ percentage points or $5.2$ percent of $p_*$ . The binomial law here is skewed to the right, with its mean $3$ only two standard deviations above the smallest count $0$ , and a symmetric normal curve does not follow it closely.

Simulation error

Definition Let $\hat{p}$ be the fraction of $n$ independent simulated replications in which the event occurs. Its simulation error is $\hat{p} - p_*$ . This error is random. Across repeated runs it has mean $0$ and standard deviation equal to the Monte Carlo standard error

\operatorname{SE}(\hat{p}) = \sqrt{\frac{p_*(1 - p_*)}{n}}.

The standard error describes the spread over possible runs. It does not give the error of the run actually performed.

In plain words A new seed generally gives a different estimate, and the standard error says how far a typical estimate falls from the model probability. More replications shrink this spread in proportion to $1/\sqrt{n}$ , but when $0 < p_* < 1$ no finite run removes it, and more replications do nothing about errors in the model itself.

Example One batch is twelve pseudorandom uniform numbers, each number below $0.25$ counts as a success, and the batch is scored $1$ when it has at least four successes. A recorded run of $20{,}000$ batches with Python’s random.Random generator and seed $88012$ scored $7071$ batches, so $\hat{p} = 7071/20000 = 0.35355$ . The standard error is

\sqrt{\frac{p_*(1 - p_*)}{20000}} \approx 0.003375,

and the error of this run, $\hat{p} - p_* \approx 0.002329$ , is about $0.69$ standard errors.

Uncertain model inputs

Definition A model input such as a success probability, a rate or a number of trials is often known only approximately. Changing an input can change the exact model probability itself, so input uncertainty is a different kind of error from approximation error and simulation error. One way to show its effect is to compute the target over the range of plausible input values.

In plain words A calculation is exact only for the inputs it was given. If the success probability is uncertain, no amount of simulation at one value of it can tell which value is right, since every run answers a question about the input that was used.

Example Suppose the success probability of each trial is known only to lie between $0.2$ and $0.3$ . The probability of at least four successes in twelve independent trials increases with the success probability, from about $0.205431$ at $0.2$ to about $0.507484$ at $0.3$ , with $0.351221$ at $0.25$ . That spread of about $0.30$ is far larger than the approximation error $0.018$ and the simulation standard error $0.0034$ .

Error interpretation

Definition Error interpretation means stating, for each reported number, which kind of error it carries and what the error statement means. Rounding error is at most half a unit in the last digit shown. Approximation error is a fixed amount for the given model and event, known when it has been checked against an exact value and limited when an error bound applies. Simulation error is random, and its standard error describes its spread over repeated runs. Model error, including uncertain inputs, lies outside all three and is addressed by examining the assumptions and inputs, not by a more precise calculation with the same inputs.

In plain words A useful report names the model, its parameters, the event, the method and the meaning of its error. For a simulation it also gives the number of independent replications, the number of occurrences, and the generator and seed. Answers from different methods are not averaged together. Each carries its own kind of error, and an average of them would carry a mixture that none of the error statements describes.

Example For $P(X \ge 4)$ with $X \sim \operatorname{Bin}(12, 0.25)$ , a clear report reads as follows. The exact model probability is $5892517/16777216 \approx 0.351221$ , rounded to six decimals. The corrected normal approximation $0.369441$ is $0.018220$ too high, a fixed error for this event. A recorded simulation of $20{,}000$ independent batches with Python’s random.Random generator and seed $88012$ had $7071$ occurrences, an estimate of $0.35355$ with standard error about $0.003375$ . All three numbers assume independent trials with success probability $0.25$ . Since the exact value is cheap here, it is the answer to report, and the other two serve as checks.

Check yourself

Let $X \sim \operatorname{Bin}(5, 0.5)$ . Choose a method for $P(X \ge 1)$ and find the probability.

Show the answer

The complement gives an exact answer at once. No successes has probability $0.5^5 = \tfrac{1}{32}$ , so

\begin{aligned} P(X \ge 1) &= 1 - \frac{1}{32} \\[0.4em] &= \frac{31}{32} = 0.96875. \end{aligned}

A corrected normal approximation with $Y \sim N(2.5, 1.25)$ would give $P(Y > 0.5) \approx 0.963181$ , adding an avoidable error of about $0.005569$ , and a simulation would give only a random estimate of the same number.

Someone simulates $100$ independent batches of five trials, each trial a success with probability $0.5$ , sees at least one success in every batch and concludes that $P(X \ge 1) = 1$ . How likely is such a run, and what can a simulation add here?

Show the answer

Each batch has at least one success with probability $\tfrac{31}{32}$ , and the batches are independent, so all $100$ batches do with probability

\Bigl(\frac{31}{32}\Bigr)^{100} \approx 0.0418,

about one run in $24$ . The observed fraction $100/100$ is a relative frequency in a finite simulation, and the model probability is still $\tfrac{31}{32}$ . With $100$ batches the standard error is about $\sqrt{(31/32)(1/32)/100} \approx 0.0174$ , and the observed fraction exceeds $\tfrac{31}{32}$ by $\tfrac{1}{32} \approx 0.031$ , about $1.8$ standard errors. Such a run is uncommon, but it is no reason to change the exact model answer. A simulation can check a program against the exact value $\tfrac{31}{32}$ or show how frequencies vary between runs, but it cannot improve an exact answer.

For $P(X \ge 4)$ with $X \sim \operatorname{Bin}(12, 0.25)$ , would a second simulation of $20{,}000$ batches with a different seed have the same error as the recorded run? Would the error of the normal approximation change? What standard error would $80{,}000$ batches have?

Show the answer

A run with a different seed generally gives a different estimate, and so an error other than $0.002329$ . Its error is random, with mean $0$ and standard deviation about $0.003375$ . The normal approximation involves no randomness, so its error stays $0.018220$ every time it is computed. Four times as many batches halve the standard error, so $80{,}000$ batches give about $0.003375/2 \approx 0.001688$ . Neither more batches nor a better approximation would change the target itself, which depends on the inputs $12$ and $0.25$ .

Choosing Exact, Approximate or Simulated Answers

Key terms

Exact model probability

Approximation error

Simulation error

Uncertain model inputs

Error interpretation

Check yourself

How this lesson connects

Builds on