A proposed process model says each of the 20 items in a batch is defective with probability , independently of the others. How likely is a batch with at least 5 defects? The binomial model answers this exactly, which makes it a good place to learn simulation, a tool that still works when no formula is available. This lesson defines one replication, makes a run reproducible with a seed, measures the numerical error of the estimate, and separates that error from the question of whether the model describes the real process.
Key terms
Simulation and the Monte Carlo method
Definition A simulation is an algorithm that generates outcomes according to a fully specified probability model. In its basic form, the Monte Carlo method uses many independently simulated outcomes to approximate a quantity defined by the model, such as a probability, an expected value or a whole distribution. A probability is estimated by the proportion of simulated outcomes in which the event occurs, and an expected value by the average of the simulated values.
In plain words Simulation lets a computer play out the model many times and keep score. Every simulated outcome is a consequence of the assumptions written into the code. It is not an observation of the real process, however large the simulated data set looks.
Example Let be the defect count of a modeled batch, and let the target be . The binomial formula gives
A Monte Carlo estimate of generates many such counts and records the fraction that are at least 5. Here the exact value serves as a benchmark for checking the simulation.
Simulation replication
Definition A simulation replication is one complete run of the modeled experiment, generated independently of the other runs, that ends in the recorded value of the statistic of interest. For a probability, replication records the indicator , equal to if the event occurs and otherwise. With replications the estimate is
The number counts replications of the whole experiment, not the random draws inside one replication.
In plain words Say in words what one replication is before writing any code, and make the summary match the question. A probability calls for the fraction of replications in which the event occurred, while an expected count would call for the average count. Replications are draws from the model, not new observations of the process, so a large adds no real data.
Example One replication of the batch model generates 20 independent draws, adds them to get a count and records exactly when . The next replication starts again with 20 new draws. Recording a hit only when would estimate , a different target, even though the code would run without any error.
Random seed
Definition A pseudorandom generator is a deterministic algorithm whose output is designed to behave like independent random draws. A random seed is a value that sets the generator’s starting state. The same generator, seed, code and software environment reproduce a run exactly, and a different seed gives a different stream of draws, so its results usually differ. A full record of a simulation also states the model and its parameters, the event or statistic, and the number of replications.
In plain words The seed lets anyone replay the identical run and check every count. Reproducing a run shows that the reported numbers come from the recorded code. It does not show that the code or the model is right, and the seed says nothing about whether the run is typical. Fix the seed and record it before running. Trying seeds until one gives a preferred result is a misuse that makes the reported estimate misleading.
Example With NumPy 2.3.5, the generator default_rng (PCG64), seed 20261012 and the call binomial(20, 0.1, size=10000), 420 of the 10000 simulated counts are at least 5, so . This call draws each count directly from the binomial distribution, which gives counts with the same distribution as adding 20 Bernoulli draws. Running the same code again in the same version reproduces all 10000 counts. Seed 20261013 gives a different run, with 462 hits. A run of 40000 replications with seed 20261012 begins with the same 10000 counts in this version, so it is not an independent second run.
Monte Carlo error
Definition Monte Carlo error is the difference between a simulation estimate and the exact model quantity it estimates, caused by using finitely many random replications. A fresh run with new random numbers usually gives a different estimate and a different error. With independent replications, an error larger than any fixed positive amount becomes unlikely as grows, a fact made precise later by the law of large numbers. Simulation error in a wider sense also includes coding mistakes, a wrong event boundary or a wrongly parameterized model, and more replications do not remove those.
In plain words Monte Carlo error is numerical noise in the answer to a question about the model. Its exact size is known only when the exact answer is known, which is rarely the case when simulation is needed. Bugs are a separate risk, best caught by testing the code on simple cases with known answers.
Example The seeded run gave , and the exact value is , so the Monte Carlo error of that run is about . The run with seed 20261013 gave , an error of about . Both runs are faithful outputs of the same model.
Monte Carlo standard error
Definition When a probability is estimated from independent replications, the number of hits is , and the standard deviation of over repeated runs is the Monte Carlo standard error
Since is unknown in practice, the usual estimate replaces with . For fixed the standard error is proportional to , so four times as many replications halve it.
In plain words The Monte Carlo standard error measures how much the estimate would typically vary from run to run under the model. It describes numerical precision only. It is not the spread of one batch’s defect count, and it is not the uncertainty that comes from estimating the real defect rate from real data. When no replication hits a rare event, and the plug-in formula also gives , which wrongly suggests certainty.
Example For the seeded run with and , the estimated Monte Carlo standard error is
The error of that run, about , is about estimated standard errors, an ordinary size. The standard deviation of a single modeled batch count, defects, describes something else entirely.
Model error
Definition Model error is the mismatch between an assumed probability model and the real process the model is meant to describe. A simulation estimates a quantity defined by its model, so the result inherits every model assumption, such as independent trials and a common success probability. More replications shrink the typical size of the Monte Carlo error but leave model error unchanged.
In plain words Precision and validity answer different questions. A billion replications pin down the model’s probability very precisely, which helps only if the model fits the process. Checking the assumptions takes knowledge of the process and real data, not more replications of the same model.
Example Suppose that in the real process half the batches have defect probability and half have , with items independent within a batch, so the average rate is still . The binomial tail probabilities of at least 5 defects are about and for these two kinds of batch, so a batch has at least 5 defects with probability about
about twice the binomial model’s . Any number of replications of the model would still estimate .
Check yourself
- Each of the 50 items in a batch is defective with probability , independently. Specify a Monte Carlo simulation to estimate the probability of at least 3 defective items, and say roughly what Monte Carlo standard error to expect from 10000 replications.
Show the answer
One replication generates 50 independent draws, or equivalently one count, and records when the count is at least 3. The estimate is the number of hits divided by . Record the generator, the seed, the software version and the code so the run can be reproduced. The exact value is
and the Monte Carlo standard error is about
The exact value also gives a benchmark for checking the code.
- A second simulation of the batch model from this page uses 40000 replications instead of 10000. How does its Monte Carlo standard error compare, and must its estimate be closer to ?
Show the answer
The standard error is proportional to , and , so it is half as large, about instead of at the exact . That describes the typical spread of estimates over many runs, not the error of one run. The code from this page with seed 20261012 and 40000 replications gives 1672 hits, so , with error about . That is farther from than the run with 10000 replications, whose error was about . A larger run is more precise on average, but no particular larger run is guaranteed to be closer.
- Suppose a run of 2000 replications of the model records no batch with at least 10 defects. A report concludes that such a batch has probability , with Monte Carlo standard error . What is wrong?
Show the answer
Under the model every count from 0 to 20 has positive probability, so . Its exact value is about , so 2000 replications are expected to produce only about hits, and zero hits is the most likely outcome, with probability about . The estimate only reports that this run had no hits, and the plug-in standard error is an artifact of the boundary, not a statement of certainty. Report the hit count, 0 out of 2000. Any probability above would make zero hits in 2000 independent replications less likely than 5 percent, since , so the run supports only a rough upper bound near . A probability this small calls for the exact formula or methods designed for rare events, and none of this says anything about real batches unless the model fits the process.