Paired t-Tests and Before-After Data, statistics lesson

Five devices have their response times measured before a software adjustment and again after it. The times before the adjustment range from 10 to 50 milliseconds, while no device changes by more than 3 milliseconds. Comparing the two columns as if they came from different devices would bury those small changes under the large differences between devices. This lesson analyzes such data through one difference per pair, which reduces the comparison to a one-sample t procedure, and explains what a before-after design can and cannot show about cause.

Key terms

Paired data

Definition Paired data consist of two measurements on each of $n$ pairs, where the design links the two measurements of a pair before any results are seen. A pair can be one unit measured under two conditions or at two times, such as a device measured before and after an adjustment, or two different units deliberately matched on characteristics related to the response, such as two neighboring plots of land. The pair is the unit of analysis, so the sample size is the number of pairs. The two measurements within a pair may be dependent, and usually are, while different pairs are assumed independent of one another.

In plain words Pairing is a fact about how the data were collected, not about how the spreadsheet looks. Two unrelated groups of five do not become paired because their rows line up, and two measurements of the same device do not become independent when they are stored on separate sheets. The usual reason to pair is that the two measurements of a pair share whatever makes that unit high or low, so a comparison within the pair leaves those differences out.

Example Five independently selected devices have these response times in milliseconds.

Device	1	2	3	4	5
Before	10	20	30	40	50
After	11	22	30	43	49

Each after value belongs to the device in the same column. The table holds ten measurements but only five pairs, so $n = 5$ . The before and after times are strongly related, with a sample correlation of about $0.995$ , because a device that is slow before the adjustment is also slow after it.

Within-pair difference

Definition For pair $i$ with measurements $y_{i1}$ and $y_{i2}$ , the within-pair difference is $d_i = y_{i1} - y_{i2}$ , always taken in the same direction, chosen before the analysis. The paired analysis replaces the $2n$ measurements by the $n$ differences, summarized by their mean $\bar{d}$ and their sample standard deviation $s_d$ with the $n - 1$ denominator. The mean difference equals the difference of the two column means, but the spread of the differences depends on how the two measurements vary together,

s_d^2 = s_1^2 + s_2^2 - 2 s_{12},

where $s_1^2$ and $s_2^2$ are the sample variances of the two columns and $s_{12}$ is their sample covariance.

In plain words Each difference compares a unit with itself, or with its match, so any amount that adds equally to both measurements, such as a slow device’s extra baseline time, cancels. When the two measurements of a pair are positively related, the differences have a smaller variance than the sum of the two column variances, and with a strong relation they can vary far less than either column. A fixed direction gives the sign a meaning. With after minus before, a positive difference means the value went up.

Example In the direction after minus before, the device differences are $1$ , $2$ , $0$ , $3$ and $-1$ milliseconds. Their mean is $\bar{d} = 5/5 = 1$ , the deviations from the mean are $0$ , $1$ , $-1$ , $2$ and $-2$ , and their squares add to $10$ , so

s_d^2 = \frac{10}{5 - 1} = 2.5

and $s_d \approx 1.581$ milliseconds. The after and before columns have sample variances $237.5$ and $250$ and sample covariance $242.5$ , and indeed $237.5 + 250 - 2 \times 242.5 = 2.5$ . The column standard deviations, about $15.4$ and $15.8$ milliseconds, are roughly ten times the spread of the differences.

Paired t test

Definition Let $\mu_d$ be the population mean of the within-pair differences. The paired t test of $H_0: \mu_d = \mu_0$ , usually with $\mu_0 = 0$ , is the one-sample t test applied to the $n$ differences,

t = \frac{\bar{d} - \mu_0}{s_d/\sqrt{n}},

with $n - 1$ degrees of freedom, where $n \ge 2$ is the number of complete pairs and $s_d/\sqrt{n}$ is the estimated standard error of $\bar{d}$ . If the differences are independent observations from one normal distribution with positive variance, the statistic has exactly this t distribution when $H_0$ is true. Without normality the t reference is an approximation, which is reasonable when the differences show no strong skewness or outliers for the number of pairs available. The conditions concern the differences and the independence between pairs. The two measurements within a pair need not be independent, and the two columns need not be normal on their own.

In plain words Once each pair is reduced to one difference, the problem is a one-sample problem. The question becomes whether the mean change in the population of pairs differs from the null value, judged against the variation of the changes themselves.

Example For the devices the estimated standard error is

\frac{s_d}{\sqrt{5}} = \sqrt{\frac{2.5}{5}} \approx 0.707

milliseconds, so $t = 1/0.707 \approx 1.414$ with $4$ degrees of freedom. The two-sided p-value is about $0.230$ , so at the level $0.05$ the test does not reject $H_0: \mu_d = 0$ . Five pairs do not give strong evidence of a mean change, which is different from showing that the mean change is zero. Treating the columns as independent samples instead would give the Welch standard error $\sqrt{250/5 + 237.5/5} \approx 9.87$ milliseconds, about fourteen times larger, and a statistic of only about $0.10$ , because that calculation keeps the large differences between devices that the pairing removes.

Paired confidence interval

Definition Under the conditions of the paired t test, a 95 percent confidence interval for $\mu_d$ is

\bar{d} \pm t^{*} \frac{s_d}{\sqrt{n}},

where $t^{*}$ is the $0.975$ quantile of the t distribution with $n - 1$ degrees of freedom. When the differences are independent observations from one normal distribution with positive variance, intervals built this way capture $\mu_d$ in 95 percent of repeated samples of pairs, and approximately so when the t reference is a reasonable approximation. The interval estimates the population mean of the differences. It does not show where 95 percent of individual differences fall, and it is not a prediction for the next pair.

In plain words The interval gives the mean changes that are consistent with the data, in the units of the measurement and in the chosen direction. Whether it is precise enough depends on the size of change that would matter in practice.

Example With $4$ degrees of freedom $t^{*} \approx 2.7764$ , so the margin is $2.7764 \times 0.7071 \approx 1.963$ milliseconds, and the interval runs from about $-0.96$ to $2.96$ milliseconds. It contains zero, in agreement with the p-value above $0.05$ , and it also contains mean increases of almost 3 milliseconds and mean decreases of almost 1. In the direction before minus after every difference changes sign, the mean becomes $-1$ and the interval runs from about $-2.96$ to $0.96$ milliseconds, while $s_d$ , the standard error and the two-sided p-value stay the same.

Before-after design

Definition In a before-after design each unit is measured, then receives an intervention, then is measured again, and the analysis uses the within-unit changes. The design keeps stable differences between units out of the comparison, but without a suitable control group it cannot separate the effect of the intervention from anything else that changes between the two measurements. Such alternative explanations include time trends and outside events, practice or learning from the first measurement, drift in the measuring instrument, and regression to the mean when units are selected because their first measurement was extreme. A randomized control group measured at the same two times addresses these explanations, and so can a crossover design that randomizes the order of the two conditions, provided the effect of the first condition has worn off before the second.

In plain words A paired test can give evidence that the mean measurement changed. Whether the intervention caused the change is a separate question, which the design has to answer, not the p-value. Regression to the mean is a common trap. If units are chosen because their first reading was unusually high, part of that high reading was chance, and their second reading tends to be lower even when nothing was done.

Example Suppose the five devices had been measured under light system load before the adjustment and under heavier load after it. The differences would then mix the adjustment with the change in load, and the paired analysis could not tell the two apart. Now suppose a team times 100 devices, adjusts the 10 slowest and times those 10 again. If each reading is a stable device value plus independent measurement noise with mean zero, the 10 slowest were chosen partly for large positive noise in their first reading, so their mean second reading is expected to be lower even if the adjustment does nothing. Randomly splitting the 10 slowest devices into an adjusted and an unadjusted group and comparing the mean changes of the two groups would separate the effect of the adjustment from this selection effect.

Check yourself

Six people each type a passage on keyboard A and on keyboard B, in an order chosen at random for each person. Their speeds in words per minute are below. Using the difference A minus B, find the mean difference, $s_d$ , the paired t statistic and a 95 percent paired confidence interval, given $t^{*} \approx 2.571$ for 5 degrees of freedom.

Person	1	2	3	4	5	6
Keyboard A	52	47	60	41	55	49
Keyboard B	48	45	55	38	51	49

Show the answer

The differences are $4$ , $2$ , $5$ , $3$ , $4$ and $0$ , so $\bar{d} = 18/6 = 3$ words per minute. The deviations from the mean are $1$ , $-1$ , $2$ , $0$ , $1$ and $-3$ , whose squares add to $16$ , so $s_d^2 = 16/5 = 3.2$ and $s_d \approx 1.789$ . The standard error is $\sqrt{3.2/6} \approx 0.730$ , and

t = \frac{3 - 0}{0.730} \approx 4.11

with $5$ degrees of freedom, a two-sided p-value of about $0.009$ . The margin is $2.571 \times 0.730 \approx 1.88$ , so the interval runs from about $1.12$ to $4.88$ words per minute. Each person used both keyboards, in a random order, so the data support the conclusion that keyboard A gives these six people a higher mean typing speed than keyboard B, by an amount the interval puts between about 1.1 and 4.9 words per minute. The six were not sampled from a wider population, so a broader claim needs more support.

Someone analyzes the keyboard data as two independent samples of six. The column means are about $50.67$ and $47.67$ and the column standard deviations about $6.59$ and $5.79$ words per minute. What standard error and statistic does that analysis give, and why is it the wrong comparison?

Show the answer

The Welch standard error would be $\sqrt{6.59^2/6 + 5.79^2/6} \approx 3.58$ , and the statistic $3/3.58 \approx 0.84$ , with a two-sided p-value of about $0.42$ . The estimated mean difference is the same $3$ words per minute, but the standard error is about five times the paired value $0.730$ , because it includes the large differences in typing skill between people, which range from 41 to 60 words per minute on keyboard A alone. The two columns come from the same six people and are strongly related, with a correlation of about $0.97$ , so they are not independent samples, and the sample size is six pairs, not twelve independent observations. Which analysis to use follows from the design, not from which p-value is smaller.

A clinic measures the blood pressure of 200 adults at a screening, enrolls the 30 with the highest readings in a six-week exercise program, and measures them again. Their mean systolic reading falls by 8 mmHg, and a paired t test gives $p < 0.001$ . Does this show that the program lowers blood pressure?

Show the answer

No. Under its conditions the paired test says that a mean drop this large would be very unusual if the mean change were zero, but it cannot say why the readings fell. The 30 were selected for having the highest first readings, so regression to the mean predicts a drop even without any program. Seasonal changes, getting used to the measurement and changes in medication could also contribute. A design that addresses these explanations selects the 30 people the same way, randomly assigns about half to the program and the rest to no program, measures everyone at the same two times, and compares the mean change in the program group with the mean change in the other group. That last comparison is between two independent groups, so it uses a two-sample method such as Welch’s t test on the changes.

Paired t-Tests and Before-After Data

Key terms

Paired data

Within-pair difference

Paired t test

Paired confidence interval

Before-after design

Check yourself

How this lesson connects

Builds on

Used later in