At one school, the 40 students who signed up for an optional tutoring program averaged 80 points on the final exam, while the 60 students who did not sign up averaged 68. It is tempting to say that tutoring raised scores by 12 points. But the students chose for themselves, and the tutored group already had stronger records before the program began. This lesson separates an observed difference from a causal effect and names the main reasons the two can disagree.
Key terms
Observational study
Definition In an observational study the researcher records the explanatory variable and the response for each unit but does not assign the explanatory variable. Which unit ends up in which group is decided by the units themselves or by circumstances. The units may come from a random sample, which supports generalizing to the sampled population, but without random assignment the groups being compared can differ in other ways that also affect the response. In an experiment, by contrast, the researcher assigns the conditions to the units.
In plain words An observational study watches what happens without intervening. It can show that two variables are associated and describe the association accurately. A causal conclusion from it needs additional design features or explicitly stated assumptions, together with evidence against the competing explanations.
Example The tutoring program is an observational study, since the students decided whether to take part. The comparison of 80 with 68 correctly describes these 100 students. Whether tutoring caused any part of the 12-point difference is a separate question that the two averages cannot answer.
Explanatory and response variables
Definition In a study of how one variable relates to another, the response variable is the outcome being studied, and the explanatory variable is the variable used to explain or predict differences in the response. An explanatory variable may be a treatment, an exposure or a group label, and it is also called a predictor or an independent variable, while the response is also called the outcome or the dependent variable. The roles are chosen by the question being asked. They do not by themselves say which variable, if either, causes the other.
In plain words Calling a variable explanatory states the direction of the question, not a fact about the world. A good prediction can run from a variable that has no causal effect on the response at all.
Example In the tutoring study, tutoring (yes or no) is the explanatory variable and the final exam score is the response. Prior score could be added as a second explanatory variable. In another setting, shoe size is a useful explanatory variable for predicting adult height, although buying larger shoes would not make anyone taller.
Causal effect
Definition For a unit and two conditions, treatment and control, the potential outcomes and are the responses the unit would have under treatment and under control. The individual causal effect is
and the average causal effect for a group of units is the mean of the , which equals the mean of the minus the mean of the . For each unit at most one potential outcome is observed, the one for the condition it actually received, so an individual causal effect is not observed directly.
In plain words A causal effect compares the same units under two different conditions, not two different groups of units. At any one time each unit receives only one of the conditions, so its outcome under the other condition is counterfactual. Estimating a causal effect means finding a design or assumptions that let one group stand in for what the other group would have done.
Example The table below is constructed so that both potential outcomes are known, which real data do not provide. Scores are in points.
| Student | A | B | C | D |
|---|---|---|---|---|
| Without tutoring | 60 | 70 | 50 | 80 |
| With tutoring | 66 | 74 | 58 | 82 |
| Effect | 6 | 4 | 8 | 2 |
The average causal effect is points. If the stronger students B and D choose tutoring, the observed tutored mean is and the observed untutored mean is . The observed difference of 23 points is more than four times the average effect.
Confounding
Definition Confounding is present when the groups defined by the explanatory variable also differ in other factors that affect the response, so that the observed association mixes the effect of the explanatory variable with the effects of those factors. A confounding variable is such a factor. It is associated with the explanatory variable, it affects the response other than through the explanatory variable, and it is not itself a result of the explanatory variable. It may be measured or unmeasured. Adjustment, for example comparing units with similar values of a measured confounder, removes the confounding due to that variable only as far as the compared units are truly similar on it, the variable is measured accurately and any model used is appropriate. Adjustment for measured variables gives no guarantee about unmeasured confounders, and adjusting for a variable that the explanatory variable itself affects can introduce new bias.
In plain words A confounder is a third variable that pushes units toward one group and also changes their outcomes. The raw comparison then credits the explanatory variable with part of the confounder’s influence. Adjusting for the confounders you measured can help, but it gives no guarantee about the ones you did not measure.
Example Split the 100 students of the tutoring program by their score on an earlier test.
| Group | Students | Mean final score |
|---|---|---|
| Tutored, prior 70 or more | 30 | 84 |
| Tutored, prior below 70 | 10 | 68 |
| Not tutored, prior 70 or more | 15 | 80 |
| Not tutored, prior below 70 | 45 | 64 |
The tutored students average and the others . Three quarters of the tutored students had a prior score of 70 or more, against one quarter of the others. Within each prior-score band the difference is only 4 points, so most of the raw 12-point gap goes with the imbalance in prior scores. The 4 points are still not an established causal effect. Students in the same band can differ in their exact prior scores, motivation and study habits, and the ones who chose tutoring may be the more motivated ones.
Reverse causation
Definition Reverse causation is present when the variable treated as the response, or an earlier stage of the condition it measures, affects the variable treated as explanatory, instead of or in addition to the effect being claimed. Measuring the explanatory variable before the response rules out an effect of the later measurement on the earlier one, but it does not rule out an earlier stage of the outcome driving the explanatory variable.
In plain words Ask whether the arrow could point the other way. If struggling students seek tutoring, then weak performance leads to tutoring, and tutoring can look harmful even when it helps.
Example Return to the constructed table and suppose that students A and C, who have the lowest scores without tutoring, have been struggling and therefore sign up. The observed tutored mean is and the observed untutored mean is , a difference of points, although tutoring raises every student’s score in the table and the average effect is points. In a different setting, cities with more police officers can also have more reported crime because cities with more crime hire more officers.
Selection into the analyzed group
Definition Selection can distort an association when the units that end up in the analysis are included in a way that depends on the explanatory variable and also on the response or on other factors that affect it. Restricting the analysis to such a subset can change an association, hide one or create one that is absent in the full group. Stating who was eligible, who enrolled, who stayed and who was excluded makes the selection process visible.
In plain words Before comparing groups, ask how the rows got into the data. A filter such as keeping only the students who finished the course can make the remaining groups differ in ways the full groups did not.
Example Suppose 100 enrolled students split evenly, 25 in each combination of tutored or not and well prepared or not, so tutoring and preparation are unrelated. Well-prepared students score 80 and the others 60 whether or not they are tutored, so tutoring has no effect, and in the full group both tutored and untutored students average 70. Now suppose the students who finish the course are exactly those who were tutored or well prepared, and only finishers are analyzed. The 50 tutored finishers still average 70, but the 25 untutored finishers are all well prepared and average 80. Among finishers tutoring looks 10 points harmful, though it changes nobody’s score.
Check yourself
- A health survey records, for 500 randomly sampled adults, their average daily walking time and their resting heart rate. Is this an observational study or an experiment? For the question whether walking lowers resting heart rate, name the explanatory and the response variable. Does the random sample allow a causal conclusion?
Show the answer
It is an observational study, because nobody assigned the amount of walking. Walking time is the explanatory variable and resting heart rate is the response. The random sample supports generalizing the observed association to the population the sample was drawn from, but it does not make the association causal. Age or health conditions could affect both walking and heart rate, which is confounding, and people with a low resting heart rate, often the fitter ones, may find walking easier and walk more, which is reverse causation.
- In a constructed table of potential outcomes, four students have scores 52, 62, 66 and 74 without a study program and 64, 76, 68 and 78 with it. The first two students, who have the lowest scores without the program, join it, and the other two do not. Find the average causal effect and the observed difference in mean scores. Does the observed difference show that the program has no effect?
Show the answer
The individual effects are , , and points, so the average causal effect is points. The observed mean of the two participants is and the observed mean of the other two is , so the observed difference is . It does not show that the program has no effect. Every student in the table scores higher with the program, but the participants would have averaged only without it, 13 points below the others, and that imbalance exactly hides their average gain of 13 points.
- A survey of students finds that those who sleep more hours per night have higher grade averages. Give two competing explanations besides “more sleep raises grades” and propose a study that would better distinguish them.
Show the answer
One explanation is confounding. Hours worked at a part-time job, health problems or general organization could affect both how long students sleep and how well they do in their courses. Another is reverse causation. Students whose grades are slipping may stay up late studying or worrying before exams, so weak grades lead to less sleep.
A better study randomly assigns volunteer students either to a program that helps them sleep longer, for example coaching and a fixed bedtime schedule, or to a comparison program, measures sleep with a device rather than by recall, and compares grades at the end of the term. Random assignment makes the groups comparable on average across the possible assignments, including on job hours and organization, and since chance alone decides the assignment, neither earlier grades nor later ones can have caused it. The study estimates the effect of the program, which may act partly through routes other than sleep, and volunteers may differ from the student body as a whole, so both limits should be reported.