Statistics reference sheet

This sheet follows the twelve units of the course in order. Each unit starts with the big idea in a few sentences, then gives the definitions, formulas, rules and procedures you will look up most often, with the conditions they need. When you want the full explanation, a derivation or more examples, follow the lesson links under each topic.

Population quantities, called parameters, are the population mean $\mu$, the population proportion $p$ and the population standard deviation $\sigma$. The matching sample statistics are the sample mean $\bar{x}$, the sample proportion $\hat{p}$ and the sample standard deviation $s$, which divides by $n - 1$. A capital letter such as $X$ names a random variable and a lowercase letter an observed value, so $\bar{X}$ is the sample mean before the data are collected and $\bar{x}$ is its observed value. Probabilities are written $P(A)$ and $P(A \mid B)$, and a random variable has expected value $E(X)$, variance $\operatorname{Var}(X)$ and standard deviation $\operatorname{SD}(X)$. The standard error $\operatorname{SE}$ of a statistic is the standard deviation of its sampling distribution, and $\widehat{\operatorname{SE}}$ is an estimate of it computed from the data.

A normal distribution is written $N(\mu, \sigma^2)$ with its variance, $Z$ stands for a standard normal variable and $\Phi$ for its CDF. Quantiles are lower-tail quantiles. The $q$ quantile of $N(0, 1)$ is $z_q$, so $\Phi(z_q) = q$, and $t_{q,\nu}$, $\chi^2_{q,\nu}$ and $F_{q;\,d_1,d_2}$ are the $q$ quantiles of the $t_\nu$, $\chi^2_\nu$ and $F_{d_1,d_2}$ distributions, whose subscripts $\nu$, $d_1$ and $d_2$ are degrees of freedom. A two-sided 95 percent procedure therefore uses $z_{0.975} \approx 1.960$. Hypotheses are $H_0$ and $H_1$, and $\alpha$ is the significance level unless a topic gives it another meaning. Logarithms are natural logarithms, and $\approx$ marks a rounded or approximate value.

Statistical Thinking and Data

Statistics starts with a question about a group and a clear idea of what each recorded value means. A number computed from data describes the units that were observed, and a statement about any other units needs an argument about how the data were produced.

Populations, samples, parameters and statistics

A statistical question asks about a defined collection of units and expects the answers to vary from unit to unit. The observational unit is the person, thing or event that one row of data describes. The population is the complete set of units the question is about, of size $N$, and a sample is the part of it on which data are collected, of size $n$. A census collects data on every unit.

For a fixed population a parameter is one number, usually unknown. A statistic is known once the data are in, but its value changes from sample to sample, which is sampling variability, and for one sample the difference $\bar{x} - \mu$ is its sampling error. Descriptive statistics summarizes the data at hand. Inferential statistics draws conclusions about a larger population, a process or a cause, with a statement of uncertainty, and that step needs random sampling, random assignment or a model for how the data arose.

For example, six waits of 2, 3, 4, 5, 7 and 9 minutes have $\bar{x} = 30/6 = 5$ minutes, and two of the six exceed 6 minutes, so $\hat{p} = 1/3$. Six other visits from the same population would usually give other values.

Lessons Introduction to Statistics

Variables and levels of measurement

A variable is a characteristic recorded for each unit. A categorical variable places a unit in one of several categories, and a numerical variable records an amount in stated units, so that differences and averages of its values are amounts of the same thing. Identifiers and numeric codes for categories stay categorical. A numerical variable is discrete when its possible values form a finite or countable list, such as counts, and continuous when, in the model used, it can take any value in an interval.

An operational definition says exactly how a value is obtained, including the instrument, the units, the recording resolution and how a missing value is marked. Accuracy is closeness to the true value, and precision is agreement among repeated measurements. A binary indicator is $1$ for a unit that meets a condition and $0$ otherwise, and the mean of the indicators is the proportion of units that meet it.

Ratios fail on an interval scale. The temperatures $10$ and $20$ degrees Celsius are $50$ and $68$ degrees Fahrenheit, and $68/50 = 1.36$, not $2$. Differences convert consistently, since a change of $10$ Celsius degrees is always a change of $18$ Fahrenheit degrees.

Lessons Types of Data and Measurement

Data tables and data quality

A data table gives each observational unit a row and each variable a column. The data dictionary records each variable’s meaning, units, allowed values and missing-value code, and the provenance records who produced the data, how the units were selected and how the file reached the analyst. The raw data stay unchanged, and every edit goes into a dated data cleaning log with its reason.

• Missing values. A missing value is not zero. Four observed values totaling 105 have mean $26.25$, while recording a fifth, missing value as zero gives $105/5 = 21$, an average that includes a value nobody measured.
• Duplicates. A duplicate record is the same unit on the same occasion entered more than once. Equal values alone do not show that two records are duplicates.
• Outliers. An outlier lies unusually far from the bulk of the data. It is a reason to check the value, not evidence of an error, since it can be a mistake, a unit from outside the population or a genuine extreme.

A complete-case analysis drops every record with a missing value on the variables it needs, so its results describe the complete cases. They answer a question about all the intended units only under an assumption about the mechanism.

Lessons Data Tables, Missing Values and Data Quality

Describing and Exploring Data

Tables, graphs and a few numbers describe what a set of data shows. Center, spread, position and shape summarize one variable, while scatter plots, correlation and two-way tables describe how two variables vary together. Each summary answers a particular question, so the question and the shape of the data decide which one to use.

Frequency tables and categorical charts

The frequency of a category is its count. When the categories are mutually exclusive and exhaustive, the frequencies add to $n$, and the relative frequency of a category with frequency $f$ is $f/n$. A bar chart draws one bar per category, separated by gaps, with lengths measured from a baseline at zero. A contingency table cross-classifies two categorical variables, its row and column totals are the margins, and a conditional percentage uses one row or one column as its denominator.

For example, if 12 of 25 people use the bus and 8 of the 10 people in group A do, then 80 percent of group A use the bus, while group A makes up $8/12 \approx 66.7$ percent of bus users. The two percentages have different denominators and answer different questions.

Cumulative frequencies are meaningful only for ordered categories or numerical values. In a multiple-response question a respondent may choose several categories, so the percentages of respondents can add to more than 100 percent.

Lessons Frequency Tables and Categorical Charts

Dot plots, histograms and distribution shape

A dot plot shows one dot per observation. A histogram divides an interval containing the data into adjacent bins and draws touching bars. When the bins have different widths, the height of each bar must be its frequency density, so that areas show counts. A bin of width $w_j$ holding $n_j$ observations gets

and dividing by $n$ as well gives bars with total area $1$. A bin from 0 to 2 and a bin from 2 to 8 that each hold 6 observations have heights $3$ and $1$ and equal areas.

Describe a distribution by its shape, center, spread and unusual values. It is skewed to the right when its right tail stretches much farther from the bulk than its left tail, and its modality is the number of clear peaks. The apparent shape can depend on the choice of bins, especially in a small sample.

Lessons Dot Plots, Histograms and Distribution Shape

Center

The values 2, 3, 4, 5 and 16 have mean 6 and median 4. Changing 16 to 26 moves the mean to 8 and leaves the median at 4. Combining groups of 2 and 8 people with means 10 and 20 gives the overall mean $(2 \cdot 10 + 8 \cdot 20)/10 = 18$, not the unweighted average 15.

Lessons Mean, Median, Mode and Weighted Averages

Variance and standard deviation

For all $N$ values of a finite population with mean $\mu$, and for a sample of $n \ge 2$ values with mean $\bar{x}$,

The standard deviations $\sigma$ and $s$ are the nonnegative square roots, in the units of the data, while variances are in squared units. A standard deviation is $0$ exactly when all the values are equal. The sample variance divides by the $n - 1$ degrees of freedom because the deviations from $\bar{x}$ add to zero, and for independent observations from a population with variance $\sigma^2$ this gives $E(S^2) = \sigma^2$.

The values 2, 4, 6 and 8 have mean 5 and squared deviations adding to 20. As a whole population they have $\sigma^2 = 5$ and $\sigma \approx 2.236$, and as a sample $s^2 = 20/3$ and $s \approx 2.582$.

Lessons Variance and Standard Deviation

Percentiles, quartiles and box plots

A $p$th percentile has about $p$ percent of the data below it. Sample percentiles need a stated rule, and the common rules can disagree in small samples. Under the median-of-halves rule, $Q_1$ and $Q_3$ are the medians of the lower and upper halves of the ordered data, with the middle value left out of both halves when $n$ is odd. Then

The five-number summary lists the minimum, $Q_1$, the median, $Q_3$ and the maximum. A box plot draws a box from $Q_1$ to $Q_3$ with a line at the median. In the modified box plot the whiskers stop at the most extreme values that are not beyond the fences $L$ and $U$, and values below $L$ or above $U$ are drawn as points and flagged as potential outliers.

Two more resistant summaries are the median absolute deviation, $\operatorname{MAD} = \operatorname{median}_i \lvert x_i - m \rvert$ for the median $m$, and the trimmed mean, which drops the $k$ smallest and the $k$ largest values, with $2k < n$, and averages the rest. Some software multiplies the MAD by about $1.4826$, so a report says which version it gives.

For delays of 1, 2, 3, 4, 5, 6, 7 and 20 minutes, $Q_1 = 2.5$, the median is $4.5$ and $Q_3 = 6.5$, so the IQR is 4 and the fences are $-3.5$ and $12.5$. Only the delay of 20 minutes is flagged. The mean is 6, while the MAD is 2 and the mean trimmed by one value at each end is $4.5$.

Lessons Percentiles, Quartiles and Box Plots

Z-scores and changes of scale

The z-score of a value $x$ relative to a distribution with mean $\mu$ and standard deviation $\sigma > 0$ is

and for a sample $\bar{x}$ and $s$ take the places of $\mu$ and $\sigma$. It counts standard deviations above or below the mean, and standardized data have mean $0$ and standard deviation $1$. For a linear transformation $y_i = a + bx_i$ of every value,

A shift alone changes no measure of spread. Temperatures with mean 20 and SD 5 degrees Celsius have mean $32 + 1.8 \cdot 20 = 68$ and SD $1.8 \cdot 5 = 9$ in degrees Fahrenheit, and 30 degrees Celsius, which is 86 degrees Fahrenheit, has $z = 2$ on both scales.

Lessons Z-Scores and Changes of Scale

Comparing distributions and reading graphs

Compare groups on a common scale, looking at center, spread, shape, unusual values and the number of observations behind each group. The length of a bar must start from a meaningful baseline, normally zero. For a change from $a$ to $b$, with $a > 0$ for the relative change,

A change between two percentages is measured in percentage points, so a rise from 10 percent to 15 percent is 5 percentage points and a relative increase of 50 percent. Counts compare groups only when the denominators match, and 20 events among 1000 people and 80 among 2000 are rates of 2 and 4 percent. Graphs mislead through truncated baselines, side-by-side panels on different scales, unequal histogram bins drawn with frequency heights, uneven time axes and icons enlarged in both directions.

Lessons Comparing Distributions and Reading Graphs

Scatter plots, covariance and correlation

For $n \ge 2$ pairs $(x_i, y_i)$ measured on the same individuals, the sample covariance and the Pearson correlation are

The covariance is in the product of the two units. The correlation needs $s_x > 0$ and $s_y > 0$, has no units and satisfies $-1 \le r \le 1$, with $r = \pm 1$ exactly when all the points lie on one straight line, rising for $r = 1$ and falling for $r = -1$. It does not change when either variable is shifted or multiplied by a positive constant, and it changes sign when one variable is multiplied by a negative constant.

Correlation measures only linear association. The points $(-1, 1)$, $(0, 0)$ and $(1, 1)$ lie exactly on the curve $y = x^2$ and have $r = 0$. A single influential point, or data that cover only part of the range of a variable, can change $r$ a lot. The Spearman rank correlation $r_s$ is the Pearson correlation of the ranks, with tied values given the average of their ranks, and it measures monotonic association. Neither coefficient shows that one variable causes the other.

Lessons Scatter Plots, Covariance and Correlation

Two-way tables and Simpson’s paradox

The aggregated proportion of a group is a weighted average of its subgroup proportions, each weighted by that subgroup’s share of the group’s cases,

Simpson’s paradox occurs when an association has the same direction in every subgroup but reverses or disappears once the subgroups are combined. It can happen only when the groups being compared are spread differently over the subgroups.

Service A has the higher success rate for both kinds of case and the lower rate overall, because most of its cases are hard. A standardized rate applies one fixed mixture of subgroups to every group. With half easy and half hard cases, A scores $0.5(0.90) + 0.5(0.30) = 0.60$ and B scores $0.50$. Which comparison answers the question, and whether the subgroup variable is a confounder, depends on the causal setting, not on the arithmetic.

Lessons Two-Way Tables and Simpson’s Paradox

Collecting Trustworthy Data

How data are collected decides which conclusions they can support. A biased selection stays biased however large the sample, random sampling supports generalizing to the population that was sampled, and random assignment supports conclusions about cause and effect.

Sampling bias and representativeness

The target population is the set of units a study wants to describe, and the sampling frame is the list or procedure from which units are actually selected. Selection bias is a systematic difference between the analyzed units and the target population, and a larger sample from the same process does not remove it.

If a share $c$ of the target population is covered, with $0 < c < 1$, proportion $p_C$ among the covered units and $p_U$ among the others, then

The error is the uncovered share times the gap, whatever the sample size. With $c = 0.2$, $p_C = 0.90$ and $p_U = 0.50$, the target proportion is $0.58$ and the covered units overstate it by $0.32$. The same calculation applies to nonresponse, with respondents in place of covered units. A sample is representative for a question when its selection and measurement give no systematic tendency to misstate the answer, which is a property of the process, not of how one sample looks.

Lessons Sampling Bias and Representativeness

Random sampling methods

A probability sample gives every unit of the frame a known, positive inclusion probability $\pi_i$, and the design weight of a sampled unit is $w_i = 1/\pi_i$.

With unequal inclusion probabilities, estimate a proportion by the weighted proportion $\hat{p}_w = \sum w_i y_i / \sum w_i$, where both sums run over the sample and $y_i$ is a 0/1 indicator. Sampling 10 of 20 units in one stratum and 10 of 80 in another gives weights 2 and 8. If 8 and 2 of the sampled units have the property, the weighted proportion is $(8 \cdot 2 + 2 \cdot 8)/(10 \cdot 2 + 10 \cdot 8) = 0.32$, while the unweighted $10/20 = 0.50$ overrepresents the small stratum.

The design effect of an estimator $\hat{\theta}$ compares its variance under the design with its variance under simple random sampling of the same size from the same frame,

and $n/\text{DEFF}$ is the effective sample size. Stratifying on a variable related to the response often brings it below 1, and sampling clusters of similar units often pushes it above 1, but the value depends on the population, the estimator and the variable measured.

Lessons Random Sampling Methods

Observational studies, confounding and causation

In an observational study the researcher records the explanatory variable and the response but does not assign the explanatory variable. For a unit $i$ with potential outcomes $Y_i(1)$ under treatment and $Y_i(0)$ under control, the causal effect is $\tau_i = Y_i(1) - Y_i(0)$, and at most one of the two outcomes is ever observed, so $\tau_i$ is never observed directly. An association between the explanatory variable and the response can come from

• a causal effect of the explanatory variable,
• confounding, a factor associated with the explanatory variable that also affects the response,
• reverse causation, where the response or an early stage of it affects the explanatory variable,
• selection, when entry into the analysis depends on both variables,
• chance variation.

Adjusting for a measured confounder, for example by comparing units with similar values of it, removes confounding by that variable only as far as the compared units really are similar, and unmeasured confounders remain.

Lessons Observational Studies, Confounding and Causation

Randomized experiments

In an experiment the investigator assigns treatments to experimental units, the entities that can receive different treatments. Random assignment by a chance mechanism balances measured and unmeasured characteristics on average over the possible assignments, which supports a causal comparison of the treatments for the units studied.

• Control and placebo. A control group receives a comparison condition, and a placebo imitates the treatment without its active component.
• Blinding. Some or all of the participants, the staff delivering treatments and the people assessing outcomes are kept from knowing the assignments, and a report says exactly who was blinded.
• Replication. Each treatment goes to several experimental units. Repeated measurements of one unit are not replicates.
• Blocking. Treatments are randomized separately within blocks of similar units and compared within blocks, which keeps block differences out of the comparison. A matched pairs design uses blocks of two and analyzes the within-pair differences.

Lessons Randomized Experiments, Blocking and Matched Pairs

Probability Models for Statistics

Probability describes chance through a model, and statistical inference rests on such models. The rules for events lead to conditional probability and Bayes’ theorem, random variables turn outcomes into numbers, and a few standard distributions model counts, waiting times and measurements.

Probability rules, conditioning and independence

A sample space $\Omega$ contains all possible outcomes, an event is a set of outcomes, and a probability model gives each event a number with $0 \le P(A) \le 1$, $P(\Omega) = 1$ and the probabilities of disjoint events adding. When a finite sample space has equally likely outcomes, $P(A)$ is the number of outcomes in $A$ divided by the number in $\Omega$.

When $P(B) > 0$, independence means $P(A \mid B) = P(A)$. Disjoint events with positive probabilities are never independent, since one rules out the other. For a fair die, the events $A = \{2, 4, 6\}$ and $B = \{4, 5, 6\}$ give $P(A \cup B) = 3/6 + 3/6 - 2/6 = 2/3$, and since $P(A \cap B) = 1/3$ differs from $P(A)\,P(B) = 1/4$, they are not independent. In repeated independent trials the relative frequency of an event tends to its probability, which is the long-run reading of probability.

Lessons Probability Rules and Events, Conditional Probability and Independence

Bayes’ theorem and base rates

For a condition $D$ with $0 < P(D) < 1$ and a positive test result, written $+$, with $P(+) > 0$,

The predictive values depend on the base rate as well as on the test. With base rate $0.10$, sensitivity $0.90$ and specificity $0.95$, natural frequencies for 1000 people give 90 true positives and 45 false positives, so $P(D \mid +) = 90/135 = 2/3$. With a base rate of $0.02$ the same test gives $P(D \mid +) = 18/67 \approx 0.269$. Reading the sensitivity $P(+ \mid D)$ as if it were $P(D \mid +)$ ignores the base rate.

Lessons Bayes’ Theorem and Base Rates

Counting and sampling without replacement

If a procedure has $k$ stages with $n_1, n_2, \dots, n_k$ possible results at the successive stages, whatever happened before, it has $n_1 n_2 \cdots n_k$ possible results in all. For whole numbers $0 \le k \le n$, with $0! = 1$,

Draws without replacement are dependent. From 8 items of which 3 are marked, a sample of 3 drawn without replacement, with every set of 3 equally likely, contains exactly 2 marked items with probability $\binom{3}{2}\binom{5}{1}/\binom{8}{3} = 15/56$, and no marked item with probability $10/56 \approx 0.179$, not the $(5/8)^3 \approx 0.244$ that independent draws with replacement would give.

Lessons Counting and Sampling Without Replacement

Random variables, expected value and variance

A random variable assigns a number to each outcome. A discrete random variable has a probability mass function $p_X(x) = P(X = x)$ whose values are nonnegative and add to 1 over its support, the set of values with positive probability. Every random variable has a cumulative distribution function $F_X(x) = P(X \le x)$, which never decreases and runs from 0 to 1, and $P(a < X \le b) = F_X(b) - F_X(a)$ for $a < b$.

For a discrete $X$ with mean $\mu = E(X)$,

provided $\sum_x \lvert x \rvert\,p_X(x)$ is finite for the mean and $E(X^2)$ is finite for the variance, and $\operatorname{SD}(X) = \sqrt{\operatorname{Var}(X)}$.

For uncorrelated variables the variances add for differences too, while standard deviations do not add. If $X$ is $-1$ or $1$ with probabilities $1/4$ and $3/4$, then $E(X) = 1/2$, $E(X^2) = 1$ and $\operatorname{Var}(X) = 3/4$. For an independent copy $Y$ of $X$, $\operatorname{Var}(X + Y) = 3/2$, while $\operatorname{Var}(X + X) = \operatorname{Var}(2X) = 3$.

Lessons Discrete Random Variables and Probability Distributions, Expected Value, Variance and Sums of Random Variables

Models for counts

Each model needs its setting.

• Binomial. The number of successes in $n$ independent trials, with $n$ fixed in advance and the same success probability $p$ in every trial.
• Hypergeometric. The number of marked items in a uniformly random sample of $n$ drawn without replacement from $N$ items, $K$ of them marked. Its possible values are the whole numbers from $\max(0, n - N + K)$ to $\min(n, K)$, and the variance formula needs $N \ge 2$. When $n$ is a small fraction of $N$ it is close to $\operatorname{Binomial}(n, K/N)$.
• Poisson. The number of events that occur one at a time, independently in nonoverlapping intervals, at a constant rate $r$, counted over an exposure $t$, so that $\lambda = rt$ is an expected count, not a rate. Counts whose variance clearly exceeds their mean are overdispersed relative to this model.

For $X \sim \operatorname{Binomial}(4, 0.2)$, $P(X = 2) = 6(0.2)^2(0.8)^2 = 0.1536$. With $r = 0.3$ events per minute over 10 minutes, $\lambda = 3$, $P(X = 0) = e^{-3} \approx 0.0498$ and $P(X = 2) = 4.5e^{-3} \approx 0.224$.

Lessons Bernoulli, Binomial and Hypergeometric Models, Poisson Models for Event Counts

Continuous distributions

A continuous random variable has a density $f$ with $f(x) \ge 0$ and total area $1$ under its graph, and probabilities are areas, so for $a \le b$

Each single value has probability $0$, so $<$ and $\le$ give the same probabilities. A density value is not a probability and can exceed $1$, and its units are the reciprocal of the units of $X$.

An exponential waiting time $T$ has survival probability $P(T > t) = e^{-rt}$ for $t \ge 0$. It is the wait for the first event of a Poisson process with rate $r$, and it is memoryless, $P(T > s + t \mid T > s) = P(T > t)$ for all $s, t \ge 0$. With $r = 0.5$ per minute, $P(2 < T \le 4) = e^{-1} - e^{-2} \approx 0.2325$ and $P(T > 6 \mid T > 2) = e^{-2} \approx 0.1353$.

Lessons Continuous Distributions, Density and Cumulative Probability

Normal distributions and quantiles

The normal distribution $N(\mu, \sigma^2)$ with $\sigma > 0$ has the density

which is symmetric about $\mu$ with inflection points at $\mu \pm \sigma$. If $X \sim N(\mu, \sigma^2)$, then $Z = (X - \mu)/\sigma \sim N(0, 1)$, so

If $X$ is normal and $b \ne 0$, then $a + bX$ is normal, and a sum of independent normal variables is normal.

For every normal distribution the probabilities within 1, 2 and 3 standard deviations of the mean are about $0.6827$, $0.9545$ and $0.9973$. This empirical rule describes the normal model, and for other shapes it can be far off. For $X \sim N(100, 10^2)$, $P(X > 120) = 1 - \Phi(2) \approx 0.0228$ and the $0.95$ quantile is $100 + 10 \cdot 1.6449 \approx 116.45$.

Lessons Normal Distributions and Quantiles

Normal approximations and their limits

For $X \sim \operatorname{Binomial}(n, p)$ with $0 < p < 1$, the normal approximation replaces $X$ by $Y \sim N\big(np,\ np(1 - p)\big)$, and the continuity correction treats each whole number $k$ as the interval from $k - 0.5$ to $k + 0.5$.

The success-failure condition asks for $np \ge 10$ and $n(1 - p) \ge 10$, and some texts use 5 or 15 instead. It is a rule of thumb, not a bound on the error. For $n = 100$ and $p = 0.5$ the approximation gives $P(X \le 50) \approx \Phi(0.1) \approx 0.5398$, which matches the exact value to four decimals. For $n = 100$ and $p = 0.01$ it gives $P(X \le 0) \approx 0.3077$, while the exact value is $0.99^{100} \approx 0.3660$. Near $p = 0$ or $p = 1$, and far in the tails, use the exact binomial probability.

Lessons Normal Approximations and Their Limits

Simulation and Monte Carlo error

A simulation generates outcomes from a fully specified probability model. To estimate a probability $q$ from $B$ independent replications, record the indicator $I_b$ of the event in replication $b$. The estimate and its Monte Carlo standard error are

with $\hat{q}$ in place of $q$ to estimate the MCSE. It shrinks like $1/\sqrt{B}$, so four times as many replications halve it. When no replication shows the event, or every one does, the estimated MCSE is $0$, which does not mean the estimate is exact. A run that finds the event $X \ge 5$ for $X \sim \operatorname{Binomial}(20, 0.1)$ in 420 of 10000 replications gives $\hat{q} = 0.042$ with an estimated MCSE of about $0.0020$, and the exact value is about $0.0432$. Record the model, the generator, the seed and $B$ so that the run can be repeated exactly. More replications reduce Monte Carlo error but leave model error, the mismatch between the model and the real process, unchanged.

Lessons Simulation and Monte Carlo Error

Sampling Distributions and Estimation

A statistic varies from sample to sample, and its sampling distribution describes how. The standard error measures that variation, the law of large numbers and the central limit theorem describe sample means in large samples, and together these ideas lead to estimators, confidence intervals and sample-size plans.

Sampling distributions and standard errors

The sampling distribution of a statistic is its distribution over repeated samples of the same size, drawn by the same design from the same population. It is a different object from the population distribution and from the distribution of the values in one sample. Its standard deviation is the standard error, which describes how much the statistic varies, while the SD of the data describes how much individual values vary.

The parameters in these formulas are usually unknown, and the estimated standard errors use $s$ and $\hat{p}$ in their place, as in $s/\sqrt{n}$. The factor $(N - n)/(N - 1)$ is the finite population correction for the variance, close to $1$ when the sample is a small fraction of the population, and it multiplies the variance of $\hat{p}$ in the same way. Quadrupling $n$ halves a standard error, so with $\sigma = 12$ the SE of the mean is $2.4$ for $n = 25$ and $1.2$ for $n = 100$.

Dependence changes the standard error. For $k$ independent clusters of $m$ observations, each with variance $\sigma^2$ and with correlation $\rho$ between observations in the same cluster, $\operatorname{Var}(\bar{X}) = D\sigma^2/n$ with $n = km$, the design effect $D = 1 + (m - 1)\rho$ and the effective sample size $n/D$. With 100 readings in clusters of 10, $\rho = 0.2$ and $\sigma = 12$, the design effect is $2.8$, the SE is about $2.01$ instead of $1.2$, and the effective sample size is about $35.7$.

Lessons Sampling Distributions of Means and Proportions, Standard Deviation and Standard Error

The law of large numbers and the central limit theorem

Let $X_1, X_2, \dots$ be independent and identically distributed with mean $\mu$ and variance $\sigma^2$, and let $\bar{X}_n$ be the mean of the first $n$ of them.

Law of large numbers. When $\sigma^2$ is finite, for every $\varepsilon > 0$ the probability $P(\lvert \bar{X}_n - \mu \rvert \ge \varepsilon)$ tends to $0$ as $n \to \infty$, and Chebyshev’s inequality bounds it by $\sigma^2/(n\varepsilon^2)$. For independent and identically distributed observations a finite mean is already enough. Averages settle by dilution, not by compensation, so a run of one outcome does not make the other outcome more likely on the next trial, and expecting it to is the gambler’s fallacy.

Central limit theorem. When $0 < \sigma^2 < \infty$, for every real number $z$

as $n \to \infty$, so for large $n$ the sample mean is approximately $N(\mu, \sigma^2/n)$. For a normal population this holds exactly for every $n \ge 1$. No single sample size, such as 30, is enough for every population, because skewness and heavy tails slow the approach, especially for tail probabilities. Without a finite variance the theorem can fail, and the mean of independent standard Cauchy variables has the same standard Cauchy distribution for every $n$.

The law of large numbers says where the mean settles, and the central limit theorem describes the shape and size of its fluctuations. Neither one describes the shape of the data in a single sample. Independent exponential waits with mean 2 minutes have SD 2 minutes, so the mean of 100 waits has $\operatorname{SE} = 0.2$, and $P(1.6 \le \bar{X} \le 2.4) \approx \Phi(2) - \Phi(-2) \approx 0.9545$, while the exact value is about $0.9550$.

Lessons The Law of Large Numbers, The Central Limit Theorem

Student’s t, chi-square and F distributions

If $T \sim t_\nu$, then $T^2 \sim F_{1,\nu}$. For $n \ge 2$ independent observations from $N(\mu, \sigma^2)$, the sample mean and the sample variance are independent, and

Degrees of freedom belong to the construction, so they are not always $n - 1$. A critical value is a quantile used as a cutoff, such as $\chi^2_{0.95,1} \approx 3.841$, $\chi^2_{0.95,4} \approx 9.488$ and $F_{0.95;\,4,8} \approx 3.838$. The t critical values for two-sided 95 percent procedures are

Lessons Student’s t, Chi-Square and F Distributions

Estimators, bias and mean squared error

An estimator $\hat{\theta}$ is a rule that computes a value for a parameter $\theta$ from the data, and an estimate is the value it gives for the data actually observed. Its properties describe its behavior over repeated samples,

An estimator is unbiased when its bias is 0 for every allowed $\theta$, and consistent when it converges in probability to $\theta$ as $n$ grows. For independent observations from one population with finite variance, $\bar{X}$ is unbiased for $\mu$ and $S^2$ is unbiased for $\sigma^2$, while $E(S) \le \sigma$, so $S$ tends to underestimate $\sigma$. A small bias can be worth a large drop in variance. An unbiased estimator with variance 4 has MSE 4, while one with bias 1 and variance 1 has MSE 2.

Lessons Estimation, Bias, Precision and Mean Squared Error

Likelihood and maximum likelihood

For a model with parameter $\theta$ and fixed observed data, the likelihood $L(\theta)$ is the probability of the data under $\theta$, or their density for continuous data, read as a function of $\theta$. It is not a probability distribution for $\theta$. The ratio $L(\theta_1)/L(\theta_2)$ compares how well two values account for the data. The maximum likelihood estimate maximizes $L(\theta)$, or equivalently the log-likelihood $\ell(\theta) = \log L(\theta)$, and for independent observations the likelihood is a product and the log-likelihood a sum.

Six successes in eight independent Bernoulli trials give $L(p) = p^6(1 - p)^2$, which is largest at $\hat{p} = 6/8 = 0.75$, and $L(0.75)/L(0.5) \approx 2.85$. With no successes the maximum is at the boundary value $\hat{p} = 0$, although small positive values of $p$ remain compatible with the data.

Lessons Likelihood and Maximum Likelihood Estimation

Confidence intervals

An interval procedure $[L, U]$ has confidence level $1 - \alpha$ when $P(L \le \theta \le U) = 1 - \alpha$ for every allowed value of $\theta$, with the probability taken over repeated samples while $\theta$ stays fixed. Some procedures reach this only approximately, and some guarantee at least $1 - \alpha$. Once the data are in, a computed interval either contains $\theta$ or does not, so a 95 percent confidence level describes how often the procedure captures $\theta$, not the probability that this particular interval does. Many intervals have the form estimate $\pm$ critical value $\times$ standard error. The margin of error, the half-width, reflects only random sampling error, not bias from selection, nonresponse or measurement.