Random Sampling Methods, statistics lesson

A school district with 1,000 students in 20 schools wants to know what share of its students use a library every week, and it can afford 100 interviews. It could draw 100 names from one district list, draw separately from the lower and the upper grades, or pick two schools and interview everyone in them. All three designs use chance, but they differ in which groups of students can be selected together, and the resulting samples carry different amounts of information. This lesson defines the main random sampling designs, turns selection probabilities into survey weights and compares the precision of designs.

Key terms

Probability sample and inclusion probability

Definition A probability sample is a sample chosen by a random procedure from a sampling frame, in which every unit of the frame has a known, positive probability of being selected. The inclusion probability $\pi_i$ of unit $i$ is the probability that unit $i$ appears in the sample, computed over all the samples the procedure could produce. If the frame has $N$ units, the inclusion probabilities add up to the expected sample size,

\sum_{i=1}^{N} \pi_i = E(n),

which is simply $n$ when the design always selects exactly $n$ units.

In plain words A probability sample comes with a record of how likely each unit was to be chosen. The inclusion probability belongs to the selection plan, not to the one sample that was actually drawn. Units missing from the frame have no chance of selection, so a probability sample from a frame with undercoverage still misses the uncovered part of the target population.

Example Suppose a computer gives each of the district’s 1,000 students, independently, a probability of 0.1 of being selected. This is a probability sample with $\pi_i = 0.1$ for every student. Its sample size is itself random, and the inclusion probabilities add up to $1000 \times 0.1 = 100$ , the expected sample size. By contrast, interviewing the first 100 students who walk into a library is not a probability sample. Nobody knows anyone’s chance of inclusion, and students who never visit a library have no chance at all.

Simple random sample

Definition A simple random sample of size $n$ from a frame of $N$ units is drawn without replacement, so no unit can be selected twice, in such a way that every subset of $n$ distinct units has the same probability $1/\binom{N}{n}$ of being the sample. Every unit then has inclusion probability $n/N$ . Equal inclusion probabilities do not by themselves make a design a simple random sample, because the definition concerns whole subsets, not single units.

In plain words Put all $N$ labels in a well-mixed hat and draw $n$ of them without putting any back. No unit and no combination of units is favored. Because the draws are without replacement, the selections are dependent, since once one unit is in, there is one fewer place left for the others.

Example From 10 labeled units choose 4. There are $\binom{10}{4} = 210$ possible samples, each with probability $1/210$ . Unit 1 belongs to $\binom{9}{3} = 84$ of them, so its inclusion probability is $84/210 = 0.4$ , which equals $4/10$ . The probability that units 1 and 2 are both selected is $\binom{8}{2}/210 = 28/210 = 2/15 \approx 0.133$ , a little less than $0.4^2 = 0.16$ . Now compare a design that flips a fair coin and takes either all five odd-numbered units or all five even-numbered units. Every unit has inclusion probability $0.5$ , the same as in a simple random sample of 5, but only 2 of the $\binom{10}{5} = 252$ subsets of size 5 can occur, so it is not a simple random sample.

Stratified sampling

Definition Stratified sampling divides the frame into nonoverlapping groups called strata, which together contain every unit, using information available before selection. It then draws a separate probability sample within every stratum, independently from stratum to stratum. If stratum $h$ has $N_h$ units and a simple random sample of $n_h$ units is taken there, each of its units has inclusion probability $n_h/N_h$ . The allocation is proportional when $n_h/N_h$ is the same in every stratum, and then every unit has the same inclusion probability.

In plain words Stratifying guarantees that every group appears in the sample with a chosen size, instead of leaving the group counts to chance. A small group can be sampled more heavily than its size alone would suggest, but then its units have higher inclusion probabilities, and an estimate for the whole population has to account for that.

Example The district’s 1,000 students include 200 in the lower grades and 800 in the upper grades. With 100 interviews, proportional allocation takes 20 and 80 students, so every student has inclusion probability $0.1$ . Equal allocation takes 50 from each stratum, giving the inclusion probabilities

\begin{gathered} \pi_{\text{lower}} = \frac{50}{200} = 0.25, \\[0.4em] \pi_{\text{upper}} = \frac{50}{800} = 0.0625. \end{gathered}

Suppose 40 of the 50 sampled lower-grade students and 10 of the 50 sampled upper-grade students use a library weekly. The unweighted sample proportion $50/100 = 0.5$ is the plain average of the stratum sample proportions $0.8$ and $0.2$ . Under equal allocation it estimates the average of the two stratum proportions, which differs from the district proportion unless the strata have equal sizes or equal proportions. The next card estimates the district proportion itself.

Survey weight

Definition The basic survey weight, or design weight, of a sampled unit is the reciprocal of its inclusion probability,

w_i = \frac{1}{\pi_i}.

It counts how many units of the frame the sampled unit stands for. A population proportion is estimated by the weighted proportion

\hat{p}_w = \frac{\sum w_i y_i}{\sum w_i},

where both sums run over the sample and $y_i$ is $1$ for a yes and $0$ for a no. In practice weights are often adjusted further for nonresponse or to match known population totals, and those adjustments rest on additional assumptions. Design weights correct for known unequal selection probabilities, but they cannot stand in for units that had no chance of selection.

In plain words Each sampled answer counts once for every frame unit it represents. Weighting puts each stratum back at its true size. The uncertainty of a weighted estimate also has to be computed for the actual design, because treating the rows as equal, independent draws can misstate it.

Example With equal allocation the weights are $1/0.25 = 4$ in the lower grades and $1/0.0625 = 16$ in the upper grades. The weights add up to $50 \cdot 4 + 50 \cdot 16 = 1000$ , the size of the district, and

\begin{aligned} \hat{p}_w &= \frac{40 \cdot 4 + 10 \cdot 16}{1000} \\[0.4em] &= \frac{320}{1000} = 0.32. \end{aligned}

The same value comes from weighting the stratum proportions by the population shares, $0.2 \cdot 0.8 + 0.8 \cdot 0.2 = 0.32$ . Under this design, and provided every selected student answers accurately, the weighted proportion is an unbiased estimator of the district proportion, while the unweighted $0.5$ estimates a different quantity. With proportional allocation every weight equals $10$ , and the weighted and unweighted proportions coincide.

Cluster and systematic sampling

Definition Cluster sampling divides the frame into groups called clusters, such as schools, classrooms or city blocks, and selects a probability sample of clusters. It then observes every unit in the selected clusters, or in a two-stage design a probability sample of units within each selected cluster. Systematic sampling takes the frame as an ordered list, chooses a random start $r$ with equal probability from $1, 2, \dots, k$ , and selects the units in positions $r$ , $r + k$ , $r + 2k$ and so on to the end of the list. Each unit then has inclusion probability $1/k$ , and only $k$ different samples are possible, so a systematic sample amounts to choosing one cluster out of $k$ clusters of list positions.

In plain words A stratified design samples within every group, while a cluster design selects some of the groups and leaves the other groups out entirely. Clusters save travel and listing costs, since only the selected clusters need a list of their units. The price is that units of one cluster enter or leave the sample together and often resemble each other, so the observations are dependent, and the sample usually carries less information than the same number of units chosen individually. A systematic sample spreads the selections across the list, which often works well, but it can go badly wrong when a pattern in the list lines up with the interval $k$ .

Example The district has 20 schools of 50 students each. Selecting 2 schools by simple random sampling and interviewing all their students gives 100 interviews, and each student has inclusion probability $2/20 = 0.1$ . Two students of the same school are both selected with probability $0.1$ , while two students of different schools are both selected only with probability $1/\binom{20}{2} = 1/190$ . A systematic sample from an alphabetical list of the 1,000 students with $k = 10$ draws the start from 1 to 10 and takes every tenth student after it, again 100 students with inclusion probability $0.1$ each. A list of 140 school days in calendar order, Monday through Friday in every week, shows the danger. With $k = 5$ , each of the 5 possible samples consists of 28 days that all fall on the same weekday.

Design effect

Definition The design effect of an estimator under a sampling design is the ratio of its variance under that design to the variance of the corresponding estimator under a simple random sample of the same size from the same frame,

\text{DEFF} = \frac{\operatorname{Var}_{\text{design}}(\hat{\theta})}{\operatorname{Var}_{\text{SRS}}(\hat{\theta})}.

A value above 1 means the design is less precise than the simple random sample, and a value below 1 means it is more precise. The ratio $n/\text{DEFF}$ is called the effective sample size. When the sample is a small fraction of the frame, it is approximately the size of a simple random sample with the same variance.

In plain words The design effect says how many times larger the variance is than under simple random sampling with the same number of units. Clustering often pushes it above 1, because similar units in one cluster repeat much of the same information. Stratifying on a variable related to the outcome often brings it below 1, because the sample cannot be lopsided across groups that differ. Neither direction is guaranteed by the name of the design, and the value depends on the population, the estimator and the variable measured.

Example A small constructed school has 16 students in 4 classrooms of 4, and the classrooms contain 4, 3, 1 and 0 weekly library users, so the population proportion is $8/16 = 0.5$ . Over all $\binom{16}{8} = 12870$ simple random samples of 8 students, the sample proportion has variance $1/60$ . Choosing 2 of the 4 classrooms at random and taking all 8 of their students gives one of the six proportions $7/8$ , $5/8$ , $1/2$ , $1/2$ , $3/8$ and $1/8$ , each with probability $1/6$ , and their variance around $0.5$ is $5/96$ . Hence

\begin{aligned} \text{DEFF} &= \frac{5/96}{1/60} \\[0.4em] &= \frac{25}{8} = 3.125, \end{aligned}

so the cluster sample of 8 students has an effective sample size of only $8/3.125 = 2.56$ . Using the same classrooms as strata and drawing 2 students from each gives variance $1/128$ and a design effect of $15/32 \approx 0.47$ .

Check yourself

A school has 300 students in grade 9 and 150 in grade 12. A simple random sample of 30 students is drawn from each grade, and 12 of the sampled ninth graders and 21 of the sampled twelfth graders answer yes. Find the inclusion probabilities, the weights, the unweighted proportion and the weighted proportion. Which one estimates the proportion of yes answers among all 450 students?

Show the answer

The inclusion probabilities are $30/300 = 0.1$ in grade 9 and $30/150 = 0.2$ in grade 12, so the weights are $10$ and $5$ . The unweighted proportion is $(12 + 21)/60 = 0.55$ . The weighted proportion is

\begin{aligned} \hat{p}_w &= \frac{12 \cdot 10 + 21 \cdot 5}{30 \cdot 10 + 30 \cdot 5} \\[0.4em] &= \frac{225}{450} = 0.5. \end{aligned}

As a check, the stratum proportions $0.4$ and $0.7$ weighted by the population shares $2/3$ and $1/3$ also give $0.5$ . The weighted proportion estimates the school proportion. The unweighted one gives the two grades equal influence although grade 9 has twice as many students.

A clerk takes every fourth record from a list of 12 records numbered 1 to 12, with a random start among the first four. What is each record’s inclusion probability, which samples are possible, and is this a simple random sample of size 3?

Show the answer

Each record has inclusion probability $1/4$ . The only possible samples are records 1, 5 and 9, records 2, 6 and 10, records 3, 7 and 11, and records 4, 8 and 12, each with probability $1/4$ . A simple random sample of size 3 would also give each record probability $3/12 = 1/4$ , but it makes all $\binom{12}{3} = 220$ subsets equally likely. The subset of records 1, 2 and 3, for example, has probability $1/220$ under simple random sampling and probability $0$ here, so the systematic sample is not a simple random sample.

A city with 10 neighborhoods, ranging from 50 to 200 households, wants the share of households that commute mainly by bus. Design A selects 3 neighborhoods by simple random sampling and surveys every household in them. Design B selects 5 households at random from every neighborhood. Name each design, give the inclusion probability of a household in a neighborhood of 200 households and in one of 50 households under each design, and describe the main limitation of each for estimating uncertainty.

Show the answer

Design A is one-stage cluster sampling with neighborhoods as clusters. Every household, in a large or a small neighborhood, has inclusion probability $3/10 = 0.3$ , so all weights are equal, and household lists are needed only for the 3 selected neighborhoods. Bus use is likely to depend on where bus routes run, so households in one neighborhood tend to answer alike. The estimate then depends heavily on which 3 neighborhoods were drawn, and the design effect is likely to be well above 1. Its uncertainty has to be computed from the variation between neighborhoods, and 3 neighborhood results give only a rough estimate of that variation.

Design B is stratified sampling with neighborhoods as strata and 5 households per stratum. A household in a neighborhood of 200 has inclusion probability $5/200 = 0.025$ and weight $40$ , and one in a neighborhood of 50 has inclusion probability $5/50 = 0.1$ and weight $10$ . The estimate must use these weights, or the small neighborhoods count too much. Every neighborhood is represented, which tends to reduce the variance when neighborhoods differ, but the design needs a complete household list for the whole city, and the uncertainty calculation must follow the strata and the weights rather than treat the 50 households as one simple random sample.

Random Sampling Methods

Key terms

Probability sample and inclusion probability

Simple random sample

Stratified sampling

Survey weight

Cluster and systematic sampling

Design effect

Check yourself

How this lesson connects

Builds on

Used later in