On days when many people carry umbrellas, many people also come home soaked. Nobody thinks umbrellas make people wet. Both follow from the weather. Data are full of patterns like this, and the question is always which conclusions a study can actually support. The answer depends far less on the numbers than on how the study was set up.
Key terms
Observational study
Definition An observational study measures variables on individuals as they are, without the researcher deciding which treatment or condition each individual gets. The individuals choose their own behavior, or it is decided by circumstances. An observational study can show an association between variables, meaning that certain values of one variable tend to occur with certain values of the other. On its own it cannot establish that one variable causes the other.
In plain words You watch and record, but you do not interfere. You learn what goes together, not what causes what.
Example A researcher surveys 500 adults about how many hours of television they watch and their resting heart rate, and finds that heavy watchers tend to have higher heart rates. This is an observational study. The data show an association. They do not show that television raises heart rate, because the heavy watchers may differ in many other ways, such as how much they exercise.
Randomized experiment
Definition In an experiment the researcher imposes treatments on the subjects and then measures a response. In a randomized experiment the treatment each subject receives is decided by a chance process. A well designed experiment usually also has a control group, which receives no treatment, a placebo or the standard treatment, so that the treatment group has something to be compared with.
In plain words You decide who gets what, and you let chance make that decision, then you compare the groups.
Example Eighty volunteers with mild headaches are split by a coin flip into two groups. One group takes a new pill and the other takes an identical looking pill with no active ingredient. After an hour the researchers record how many in each group report that the headache is gone. This is a randomized experiment with a placebo control.
Random assignment
Definition Random assignment means using a chance process to decide which subjects receive which treatment. It tends to balance the groups on every other variable, both the ones you thought of and the ones you did not, so that the groups differ in a systematic way only in the treatment. Random assignment is different from random sampling. Random sampling decides who is in the study, and it supports generalizing from the sample to the population. Random assignment decides who gets which treatment within the study, and it supports conclusions about cause and effect.
In plain words Random sampling answers “who does this apply to.” Random assignment answers “did the treatment cause the difference.” A study can have one, both or neither.
Example The four possible designs support different conclusions.
| Design | Cause and effect | Generalize to the population |
|---|---|---|
| Random sample and random assignment | Yes, for the population | Yes |
| Random assignment only | Yes, for subjects like those studied | No |
| Random sample only | No, association only | Yes, for the association |
| Neither | No | No |
A “Yes” for cause and effect still assumes the experiment was run properly and the difference between the groups is too large to be explained by chance alone.
Confounding
Definition A confounding variable is a variable that is related both to the explanatory variable and to the response, so that its effect on the response cannot be separated from the effect of the explanatory variable. Confounding is the main reason an association in an observational study does not prove causation. Random assignment protects against confounding because chance, not the subjects or the researcher, decides who gets the treatment.
In plain words A confounder is a hidden third factor that pushes both variables at once and makes them look connected.
Example In a city, months with high ice cream sales also have many cases of sunburn. Hot sunny weather is a confounder. It raises ice cream sales and it raises sun exposure. Banning ice cream would not prevent sunburn.
Causal inference
Definition A causal inference is a conclusion that changing one variable produces a change in another. At this level, such a conclusion is justified when the study is a randomized experiment, the experiment was carried out properly and the difference between the groups is too large to be explained by chance variation in the assignment alone. Properly carried out means, among other things, that the groups were treated the same apart from the treatment, the response was measured the same way in every group and subjects did not drop out in a way related to the treatment. Association alone, no matter how strong, is not enough.
In plain words To claim cause, you need random assignment, a fair run of the experiment and a difference bigger than luck would give.
Example In the headache experiment, 30 of the 40 pill takers and 18 of the 40 placebo takers report relief. Suppose this difference is large enough that chance alone would rarely produce it. Then it is reasonable to conclude that the pill causes relief for subjects like these volunteers. If instead the researchers had compared people who chose to take the new pill with people who chose to take nothing, those who chose the pill might have expected it to work and reported more relief for that reason. That expectation would be confounded with the pill.
Generalization
Definition Generalization is extending a conclusion from the individuals studied to a larger population. It is justified for the population from which the sample was randomly selected, and only for that population. Results from volunteers or from another convenience sample can be generalized only with great caution, even when the treatments were randomly assigned.
In plain words You may extend a result to the group you randomly sampled from, and no further.
Example A researcher randomly selects 300 of the 2,000 students at one high school and finds that students who eat breakfast tend to have higher grades. The association may be generalized to all students at that school, since they were randomly sampled. It may not be generalized to students in other schools, and because this is observational, it does not show that eating breakfast raises grades.
Check yourself
- Researchers randomly selected 400 residents of a town and asked how much tea they drink and how well they sleep. Residents who drink more tea reported better sleep on average. Which conclusion is best supported? (A) Drinking tea causes better sleep for residents of the town. (B) Among residents of the town, drinking more tea is associated with better reported sleep, but the study does not show that tea causes better sleep. (C) Drinking tea causes better sleep for all adults. (D) No conclusion can be drawn about residents of the town, because the treatments were not randomly assigned.
Show the answer
The correct choice is (B). The random sample allows a conclusion about the town’s residents, but the study is observational, so it shows association and not cause. Choices (A) and (C) claim cause without random assignment, and (C) also reaches beyond the town. Choice (D) is too strong, since random sampling does support a conclusion about association in the town.
- A study recruited 120 volunteers from a college campus. Each volunteer was randomly assigned either to use a new study app or to study without it for a month. The app group scored higher on a final test, and the difference was too large to be explained by chance alone. Which conclusion is most appropriate? (A) The app causes higher test scores for all college students in the country. (B) The app likely causes higher test scores for people similar to these volunteers. (C) The study shows only an association, so it says nothing about cause. (D) The app causes higher test scores for every student at that college.
Show the answer
The correct choice is (B). Random assignment supports a causal conclusion, but the subjects were volunteers, not a random sample, so the conclusion applies to people like them. Choices (A) and (D) generalize to populations that were not randomly sampled. Choice (C) ignores the random assignment.
- An observational study of house fires finds that fires with more firefighters present tend to cause more damage. Name a likely confounding variable and explain why it is one.
Show the answer
The size of the fire is a likely confounder. A larger fire brings more firefighters, and a larger fire also causes more damage. The association does not show that firefighters cause damage.