Sample Size and Margin of Error, statistics lesson

A survey team wants to estimate the proportion of residents who support a proposal to within 5 percentage points, with 95 percent confidence. How many completed responses does that take, and how many people should be invited to get them? This lesson solves the margin of error formula for $n$ , explains which planning values are safe to use, and adds the adjustments for the sampling design and for people who do not respond. It also shows what such a calculation leaves out.

Key terms

Sample-size planning

Definition Sample-size planning chooses, before any data are collected, a number of observations that should give a stated precision. It needs a confidence level with critical value $z = z_{1 - \alpha/2}$ , a target margin of error $E$ , which is the half-width of the planned interval, an anticipated value for the variability, and a sampling design. For a proportion estimated from $n$ independent binary responses with common probability $p$ , and for a mean estimated from $n$ independent observations with standard deviation $\sigma$ , setting $E$ equal to $z$ times the standard error and solving gives the normal planning formulas

\begin{gathered} n \approx \frac{z^2 p(1 - p)}{E^2}, \\[0.4em] n \approx \Bigl(\frac{z\sigma}{E}\Bigr)^2. \end{gathered}

The result is rounded up to a whole number, because rounding a fractional result down would give a planned margin larger than $E$ .

In plain words Work backward from how precise the answer must be to how much data that takes. The margin shrinks in proportion to $1/\sqrt{n}$ , so halving the margin takes about four times as many observations. The answer is a plan under assumptions. The realized interval uses the observed data and the method chosen for the analysis, such as Wilson for a proportion or $t$ for a mean, and a precision target says nothing about bias in who ends up in the sample. Precision planning also differs from power planning. Precision asks how narrow an interval should be, while power asks how often a test would detect a stated effect, and a study planned for one goal need not meet the other.

Example For the survey, take $E = 0.05$ , $z = 1.96$ and the planning value $p = 0.5$ . Then

\begin{aligned} n &\approx \frac{1.96^2 \cdot 0.25}{0.05^2} \\[0.4em] &= \frac{0.9604}{0.0025} = 384.16, \end{aligned}

so the plan needs at least 385 completed independent responses. With 385 responses the planned margin is $1.96\sqrt{0.25/385} \approx 0.0499$ , while 384 would give about $0.05001$ , just above the target. Halving the target to $2.5$ percentage points gives $4 \cdot 384.16 = 1536.64$ , so 1537 responses.

Conservative proportion

Definition In the proportion formula, the conservative planning value is $p = 0.5$ . Since

p(1 - p) = \frac14 - \Bigl(p - \frac12\Bigr)^2,

the product is at most $\tfrac14$ , with equality only at $p = \tfrac12$ . Planning at $0.5$ therefore gives the largest $n$ for a given $z$ and $E$ , and with $n \ge z^2/(4E^2)$ the planned margin $z\sqrt{p(1 - p)/n}$ is at most $E$ whatever the true $p$ is.

In plain words Without a trustworthy guess for $p$ , plan for the largest value the variance term can take. The price is extra observations when the true $p$ is far from $0.5$ . The protection covers only the term $p(1 - p)$ in this formula. It does not cover a design effect, nonresponse, bias in the sample or the failure of the normal approximation. For a rare outcome an absolute margin can also be the wrong target, since 5 percentage points is five times a rate of 1 percent, so the precision goal should fit the decision the study informs.

Example With $E = 0.05$ and $z = 1.96$ , the requirement depends on the planning value as follows.

Planning value $p$	$p(1 - p)$	Unrounded $n$	Required $n$
$0.1$ or $0.9$	$0.09$	$138.30$	$139$
$0.2$ or $0.8$	$0.16$	$245.86$	$246$
$0.3$ or $0.7$	$0.21$	$322.69$	$323$
$0.4$ or $0.6$	$0.24$	$368.79$	$369$
$0.5$	$0.25$	$384.16$	$385$

The formula treats $p$ and $1 - p$ alike, and values near $0.5$ change the requirement little.

Pilot estimate

Definition A pilot estimate is a preliminary value for the quantity that sets the variability, a proportion $p$ or a standard deviation $\sigma$ , taken from a small preliminary study or from earlier relevant work and used as the planning value. With $n$ planned this way, the planned margin, $z\sqrt{p(1 - p)/n}$ or $z\sigma/\sqrt{n}$ evaluated at the true value, is at most $E$ when the true $p(1 - p)$ or $\sigma$ is no larger than the planning value of $p(1 - p)$ or $\sigma$ . When the true value is larger, the margin can exceed $E$ .

In plain words A good pilot can save observations, but a small pilot is itself noisy and can understate the variability by chance. Use evidence from the same population, measurement and time period, look at several plausible values rather than one optimistic number, and consider planning near the less favorable end of what the pilot allows.

Example Suppose earlier work suggests a support proportion near $0.1$ . Then $n \approx 1.96^2 \cdot 0.09/0.05^2 = 138.2976$ , so 139 responses. If the proportion is really $0.5$ , a sample of 139 gives a planned margin of

1.96\sqrt{\frac{0.25}{139}} \approx 0.0831,

well above the target $0.05$ . The same risk applies to a mean. A plan built on a pilot standard deviation that is too small can lead to a larger margin than intended.

Design effect

Definition For a given statistic and sampling design, the design effect is

\text{DEFF} = \frac{\operatorname{Var}_{\text{design}}(\hat{\theta})}{\operatorname{Var}_{\text{SRS}}(\hat{\theta})},

the variance of the estimate $\hat{\theta}$ under the planned design divided by its variance under simple random sampling of the same size. The number of completed responses is planned as $\text{DEFF}$ times the requirement for simple random sampling, rounded up at the end. Equivalently, the effective sample size $n/\text{DEFF}$ should reach the basic requirement.

In plain words Cluster samples, where people in the same household, school or area resemble each other, usually have a design effect above 1 and need more responses for the same precision. Good stratification can bring it below 1. Separately, sampling a large fraction of a finite population without replacement lowers the variance through the finite population correction, which the basic formulas leave out. The value used for planning should come from similar surveys, not be chosen to fit a budget.

Example With a plausible design effect of $1.5$ , the requirement becomes $384.16 \cdot 1.5 = 576.24$ , so 577 completed responses. Their effective sample size is $577/1.5 \approx 384.7$ , which meets the basic $384.16$ . Rounding too early, as $385 \cdot 1.5 = 577.5$ , would give 578, so the unrounded value is kept until the last step.

Nonresponse allowance

Definition A nonresponse allowance converts the required number of completed responses into the number of people to invite, by dividing by the anticipated response rate $r$ and rounding up,

n_{\text{invite}} = \Bigl\lceil \frac{n_{\text{complete}}}{r} \Bigr\rceil .

In plain words Not everyone who is invited responds, so the invitation count exceeds the completion target. The expected number of completions then meets the target, but the actual number is random and can fall short, so a plan may add a further allowance or monitor recruitment as it goes. Inviting more people does not remove nonresponse bias. If people with different answers respond at different rates, the completed sample can misrepresent the population however large it is.

Example With 577 completed responses needed and an expected response rate of 80 percent, $577/0.8 = 721.25$ , so the plan sends 722 invitations, with $722 \cdot 0.8 = 577.6$ expected completions. If each invitee responds independently with probability $0.8$ , the number of completions follows $\operatorname{Binomial}(722, 0.8)$ , and the chance of reaching at least 577 is only about $0.54$ . The full chain runs from the basic requirement of 384.16 to 577 completions to 722 invitations, and each step rests on its own assumption.

Check yourself

A poll wants a margin of 3 percentage points at 95 percent confidence, with $z = 1.96$ and no prior information about the proportion. How many completed independent responses does the basic formula require?

Show the answer

Without prior information, use the conservative value $p = 0.5$ . Then

\begin{aligned} n &\approx \frac{1.96^2 \cdot 0.25}{0.03^2} \\[0.4em] &= \frac{0.9604}{0.0009} \approx 1067.11, \end{aligned}

which rounds up to 1068 completed responses, before any design or nonresponse adjustment.

Plan a mean estimate with anticipated standard deviation $\sigma = 12$ units, $z = 1.96$ and a target margin of $2$ units. Responses are independent and the expected response rate is 70 percent. Find the number of completed responses and the number of invitations, and explain why the two differ.

Show the answer

The basic requirement is

\begin{aligned} n &\approx \Bigl(\frac{1.96 \cdot 12}{2}\Bigr)^2 \\[0.4em] &= 11.76^2 = 138.2976, \end{aligned}

so the plan needs 139 completed responses. Dividing by the response rate gives $139/0.7 \approx 198.57$ , so the plan sends at least 199 invitations, with $199 \cdot 0.7 = 139.3$ expected completions.

The invitation count is larger because only about 70 percent of those invited are expected to respond. Even so, 139 completions are not guaranteed, and extra invitations do nothing about bias if those who respond differ from those who do not. If the analysis will use a $t$ interval and $s$ turns out equal to 12, the critical value with about 140 degrees of freedom is near $1.977$ rather than $1.96$ , and repeating the calculation with it raises the plan to 141 completions.

A pilot of 20 residents finds 2 supporters, so a planner uses $p = 0.1$ and plans 139 responses for a margin of $0.05$ . What margin would 139 responses give if the true proportion were $0.3$ , and what would be a more cautious plan?

Show the answer

With $p = 0.3$ the planned margin is $1.96\sqrt{0.21/139} \approx 0.0762$ , about 7.6 percentage points instead of 5. A true value of $0.3$ is compatible with the pilot, since the 95 percent Wilson interval for 2 successes in 20 trials is about $[0.028, 0.301]$ .

A pilot this small cannot pin down the variability. Planning at $p = 0.3$ , near the upper end of that interval, requires $1.96^2 \cdot 0.21/0.05^2 \approx 322.69$ , so 323 responses. The conservative value $p = 0.5$ requires 385 and keeps the planned margin at or below $0.05$ whatever the true proportion is.

Sample Size and Margin of Error

Key terms

Sample-size planning

Conservative proportion

Pilot estimate

Design effect

Nonresponse allowance

Check yourself

How this lesson connects

Builds on

Used later in