A service emails a satisfaction poll to every active account and reports the result as the opinion of everyone who used the service this year. People who canceled during the year never receive the email, and they may well be less satisfied than those who stayed. Collecting more answers from active accounts cannot reveal what the former customers think. This lesson names the ways a sample can systematically miss the group it is meant to describe, and shows why a larger sample does not repair them.
Key terms
Target population and sampling frame
Definition The target population is the complete set of units, such as people, households or purchases, that a study wants to describe, specified by who or what is eligible, where and during which period. The sampling frame is the list or procedure from which units can actually be selected, such as a membership list, a set of addresses or a method of contacting people. Units of the target population that are missing from the frame cannot be selected, and a frame can also contain units outside the target, such as duplicates or ineligible entries.
In plain words The target population is the group the conclusion is about. The frame is the group the study can reach. Comparing the two before any data are collected shows which part of the target the study cannot see.
Example Suppose the target is the 1,000 people who used the service at some point in the past 12 months. Of these, 600 still have active accounts and 400 have canceled, and the frame is the list of active accounts. If 480 active users and 120 former users are satisfied, the proportion satisfied in the target population is
A poll that reached every active account and received an answer from each would report . That figure describes the frame exactly, yet it misses the target value by 0.20.
Selection bias and undercoverage
Definition Selection bias is a systematic error that arises when the process by which units enter the data makes the analyzed units differ from the target population in the quantity being studied. Because it comes from the procedure, it does not average out when the procedure is repeated. Undercoverage is one source of selection bias. It occurs when some units of the target population are missing from the sampling frame or have no realistic chance of being reached. Suppose a share of the target is covered, with , and the covered and uncovered units have proportions and . The target proportion is then , and the covered proportion misses it by
In plain words The error from undercoverage is the uncovered share times the gap between the covered and the uncovered groups. It grows with the size of the missing group and with how different that group is, and it is zero only when the two groups have the same proportion, which a study usually cannot check because the missing group was never observed. Selection bias is also different from an unusual sample. A random sample from a complete frame, with every unit equally likely to be chosen, can by chance contain too many units of one kind, but that error varies from sample to sample and does not lean in one direction on average.
Example In the service example the covered share is , with among active users and among former users. The formula gives , the gap found above. Repeating the poll of active accounts, or sending them more reminders, would keep producing 0.80 as long as opinions stay the same, because every repetition leaves out the same 400 former users.
Nonresponse bias
Definition Nonresponse occurs when a unit selected for the sample does not provide the requested information, either at all or for a particular question. Nonresponse bias is the systematic error that results when the units that respond differ from those that do not in the quantity being studied. For a fixed set of selected units, the respondent proportion misses the proportion among all selected units by the nonresponse share times the gap between respondents and nonrespondents, the same calculation as for undercoverage with respondents in the role of the covered units. A low response rate signals that nonresponse bias is possible, but the response rate alone does not measure it.
In plain words Undercoverage means some units could never be chosen. Nonresponse means units were chosen but their answers are missing. The missing answers are unknown, so the data alone cannot say how different they would have been. A survey with a high response rate can still be biased if the few nonrespondents differ sharply, and a survey with a low response rate has no nonresponse bias in a proportion when the chance of responding is unrelated to the answer, although fewer answers still mean more chance variation.
Example A company selects 40 customers at random from its complete customer list, and 30 of them answer. Of those, 24 are satisfied, so the respondent proportion is . The 10 missing answers could all be negative or all be positive, so the proportion satisfied among the 40 selected customers lies between and . This range describes only these 40 customers, given every possible pattern of missing answers. It is not a confidence interval for the whole customer base. Following up a subsample of the nonrespondents through another channel, such as a phone call, is one way to learn about the missing group.
Voluntary response bias
Definition A voluntary response sample consists of people who decide for themselves to take part, for example by answering an open online poll or calling in to a radio program. Voluntary response bias is the systematic error that arises when the decision to take part is related to the quantity being studied, as when people with strong opinions are more likely to respond. The makeup of such a sample is set by the participants, not by a design with known selection probabilities.
In plain words An open invitation lets the respondents select themselves. A larger number of volunteers gives a more precise description of the volunteers, not of the target population. Sample size reduces chance variation, and it does nothing about who chose to answer.
Example A service with 10,000 customers posts an open poll. Suppose 2,000 customers are dissatisfied, a proportion of 0.20, and that 30 percent of the dissatisfied and 5 percent of the satisfied customers choose to answer. Then 600 dissatisfied and 400 satisfied customers respond, and the poll reports a dissatisfied proportion of
three times the target value. With 100,000 customers, the same share dissatisfied and the same response tendencies, there would be 10,000 responses, and the poll would still report 0.60.
Measurement bias
Definition Measurement bias, also called response bias in surveys, is a systematic difference between the recorded values and the true values of the quantity being measured. Leading or loaded question wording, an interviewer’s presence or manner, the wish to give a socially acceptable answer and a miscalibrated instrument can all produce it. It can occur even when the sample is selected perfectly and everyone responds. Random measurement errors with mean zero do not bias a sample mean, although they add variability, and when they are unrelated to the true values and to any errors in the other variable, they tend to inflate a standard deviation and weaken a correlation.
In plain words Selection concerns who is measured. Measurement bias concerns how the answers are obtained. A perfect sample asked a slanted question still gives slanted answers, and measuring more people the same way repeats the same distortion.
Example A clinic scale reads every weight 0.5 kg too high. For five patients with true weights 60, 72, 55, 81 and 67 kg, the true mean is 67 kg and the recorded mean is 67.5 kg. Weighing 5,000 patients on the same scale would still give a recorded mean exactly 0.5 kg above their true mean. Question wording can act the same way. Asking whether someone supports “a badly needed upgrade” of a bus line invites different answers from a neutral description of the upgrade and its cost.
Representative sample
Definition A sample is representative of a target population, for a stated question, when the way it was selected and measured gives no systematic tendency to misstate the population quantities that answer that question. Its summaries, weighted by the design when units had different selection probabilities, then differ from the population values only through chance variation, which tends to shrink as the sample grows. Representativeness is a property of the selection and measurement process, judged relative to the target and the question. Checking that one realized sample matches the population on a few visible features cannot confirm it.
In plain words Representativeness comes from a process you can trust for the question at hand, not from a sample that merely looks like the population. A sample that differs from the population by chance is not evidence of bias, and a sample that matches it on one feature can still be biased on others.
Example A random sample of 50 drawn from a complete list of the 1,000 service users contains former users on average, but one draw might contain 23, a share of 0.46 against 0.40 in the population. That is chance variation under a sound design, not bias. Now suppose another sample is built from visitors to the help page, chosen as 30 active and 20 former users to match the 60 and 40 percent split exactly. It matches the population on account status, yet people who visit a help page may be less satisfied than other users with the same status, so it can be biased on the very question being asked.
Check yourself
- Name the most likely source of bias in each design. (a) A news website invites its readers to vote on a policy question. (b) A questionnaire is mailed to a random sample of 500 registered voters, and 150 return it. (c) A survey asks “Do you agree that the overcrowded library needs longer opening hours?”
Show the answer
In design (a) the readers decide whether to vote, so this is voluntary response bias, and people who never visit the site are not covered at all. In design (b) the selection is random, but 350 of the 500 selected voters did not answer, a response rate of , so nonresponse bias is the concern if those who returned the form differ from those who did not. In design (c) the words “Do you agree” and “overcrowded” push toward a yes, which is measurement bias. It would remain even with a perfect sample and full response.
- A library selects 50 residents at random from a complete list of the town’s residents, and 40 respond. Of these, 30 support longer opening hours. Find the respondent proportion and the range of possible values for the proportion among all 50 selected residents. Does a response rate of 80 percent rule out nonresponse bias?
Show the answer
The respondent proportion is . The 10 nonrespondents could all oppose or all support the change, so the proportion among the 50 selected residents lies between and . The response rate of does not rule out bias. It limits how far off the respondent proportion can be for these 50 people, at most 0.15 too high or 0.05 too low, but whether there is any error depends on whether the nonrespondents differ from the respondents, which the data cannot show.
- A student stands outside the campus cafeteria at lunchtime on three weekdays, asks passersby how many hours a week they spend on campus, collects 400 answers and reports their average as the figure for all students at the college. Identify the weaknesses of this design and propose a better sampling frame and a follow-up plan.
Show the answer
The target population is all students enrolled at the college this term, but the frame actually used is the set of people passing the cafeteria at lunchtime on those three days. Students who are rarely on campus at midday, such as online, evening, part-time or commuting students, are undercovered, and the frame may include staff and visitors who are not students. Because the chance of passing the cafeteria at lunchtime tends to grow with the time spent on campus, the average is likely to overstate the figure for all students. People who are approached can decline, so those in a hurry may be missing, which adds nonresponse or self-selection. Recalled weekly hours are also prone to measurement error. Collecting 400 answers does not fix any of these problems.
A better plan uses the registrar’s list of enrolled students as the frame and draws a random sample from it, possibly separately for full-time and part-time students. Selected students are contacted by email and text message, nonrespondents receive reminders and then a different channel such as a phone call, and the response rate is reported by group. The question asks for a specific, neutral quantity, such as the hours spent on campus during the previous week, and is tested on a few students before the survey starts.