Comparing Two Independent Means, statistics lesson

Twenty people complete a task with interface A, and thirty different people complete the same task with interface B. Group A averages 18 minutes and group B averages 22 minutes, a gap of 4 minutes between the sample means. Could chance variation in two sample means easily produce a gap of that size, and which differences between the population means are consistent with the data? This lesson builds the standard error of a difference from both groups, uses it in Welch’s t test and confidence interval, and shows when the pooled-variance t test fits instead.

Key terms

Independent samples

Definition Two samples are independent when the values in one sample are statistically independent of the values in the other. In practice this means that no unit contributes observations to both samples, and nothing links an observation in one sample to a particular observation in the other. Two random samples drawn separately from two populations are independent in this sense. Groups formed by randomly assigning units to two conditions, with each unit measured once, are analyzed in the same way. The methods of this lesson also need independent observations within each sample. The target is the difference of population means $\mu_A - \mu_B$ , with the order of the groups fixed before the analysis, and its estimate is the difference of sample means $\bar{x}_A - \bar{x}_B$ .

In plain words Each observation belongs to exactly one group, and knowing a value in group A tells you nothing about any particular value in group B. Two columns of the same length in a spreadsheet are not evidence either way. What matters is how the data were produced. If the same people, or deliberately matched people, appear in both groups, the samples are paired, and the analysis works with the within-pair differences instead.

Example In the opening study each of the 50 people used one interface and was timed once.

Group	Size $n$	Mean $\bar{x}$ in minutes	SD $s$ in minutes
A	20	18	4
B	30	22	6

In the order A minus B the estimate of $\mu_A - \mu_B$ is $18 - 22 = -4$ minutes, and the negative sign says that group A had the lower sample mean. The order is a choice, but once chosen it stays the same in the hypotheses, the interval and the interpretation. If instead each of 25 people had used both interfaces, the two sets of times would be linked person by person, and they would not be independent samples.

Standard error of a difference

Definition Suppose $\bar{x}_A$ is the mean of $n_A$ independent observations from a population with standard deviation $\sigma_A$ , $\bar{x}_B$ is the mean of $n_B$ independent observations from a population with standard deviation $\sigma_B$ , and the two samples are independent. Then the variance of the difference is the sum of the two variances,

\begin{aligned} &\operatorname{Var}(\bar{x}_A - \bar{x}_B) \\[0.4em] &\quad = \frac{\sigma_A^2}{n_A} + \frac{\sigma_B^2}{n_B}, \end{aligned}

and the standard error of the difference is its square root, the standard deviation of the sampling distribution of $\bar{x}_A - \bar{x}_B$ . The population standard deviations are usually unknown, so the methods below use the estimated standard error

\widehat{\operatorname{SE}} = \sqrt{\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}},

where $s_A$ and $s_B$ are the sample standard deviations with the $n - 1$ denominator. If the two means were dependent, the variance of their difference would also contain the term $-2\operatorname{Cov}(\bar{x}_A, \bar{x}_B)$ .

In plain words Subtracting two means does not subtract their uncertainty. Each sample mean can miss its population mean in either direction, and the two misses can add up, so both groups contribute to the uncertainty of the difference. Variances add, standard deviations do not. When both contributions are positive, the standard error of the difference is therefore larger than either group’s own standard error and smaller than the sum of the two. The sample standard deviations describe individual measurements, while the standard error describes how much the difference of means would vary from sample to sample.

Example For the interface data the two estimated variance contributions, in square minutes, are

\begin{gathered} \frac{s_A^2}{n_A} = \frac{16}{20} = 0.8, \\[0.4em] \frac{s_B^2}{n_B} = \frac{36}{30} = 1.2 . \end{gathered}

Their sum is $2$ , so $\widehat{\operatorname{SE}} = \sqrt{2} \approx 1.414$ minutes. Adding the two separate standard errors instead, $4/\sqrt{20} + 6/\sqrt{30} \approx 1.990$ , would overstate the uncertainty, and using a sample standard deviation such as $6$ minutes would confuse the spread of individual times with the precision of a mean.

Welch t test

Definition Suppose the two samples are independent, the observations within each sample are independent, each sample has at least two observations, and each sample mean is approximately normal. The last condition holds when the populations are close to normal, or when the samples are large enough that moderate skewness has little effect. To test $H_0: \mu_A - \mu_B = \Delta_0$ , most often with $\Delta_0 = 0$ , Welch’s t test computes

t = \frac{(\bar{x}_A - \bar{x}_B) - \Delta_0}{\widehat{\operatorname{SE}}}

and compares it with the t distribution whose degrees of freedom $\nu$ come from the Welch-Satterthwaite formula in the next card. For the alternative $H_1: \mu_A - \mu_B \ne \Delta_0$ the p-value is $P(\lvert T_\nu \rvert \ge \lvert t \rvert)$ , where $T_\nu$ has that t distribution, and for a one-sided alternative it is the single tail in the direction of $H_1$ . The test does not assume $\sigma_A = \sigma_B$ . Even for normal populations its t reference is an approximation, because the exact null distribution of the statistic depends on the unknown ratio $\sigma_A / \sigma_B$ . Pairing, clustering and strong skewness or outliers in small samples call for other methods.

In plain words The statistic counts how many estimated standard errors the observed difference lies from the value named by the null hypothesis. The p-value then asks how often a statistic at least that far out would arise if the null hypothesis and the model were true. Welch’s test lets each group keep its own spread, which is why it is a common default for comparing two independent means.

Example For $H_0: \mu_A - \mu_B = 0$ against a two-sided alternative,

t = \frac{-4 - 0}{\sqrt{2}} \approx -2.828,

so the observed difference lies about $2.83$ estimated standard errors below zero. With $\nu \approx 47.997$ degrees of freedom, computed in the next card, the two-sided p-value is about $0.0068$ , and the test rejects $H_0$ at the significance level $0.05$ . The p-value is the probability, computed assuming equal population means and the model, of a statistic at least this far from zero. It is not the probability that the population means are equal.

Welch-Satterthwaite degrees of freedom

Definition Write $v_A = s_A^2/n_A$ and $v_B = s_B^2/n_B$ for the two estimated variance contributions, with $n_A \ge 2$ , $n_B \ge 2$ and at least one of $v_A$ and $v_B$ positive. The Welch-Satterthwaite degrees of freedom are

\nu = \frac{(v_A + v_B)^2}{\dfrac{v_A^2}{n_A - 1} + \dfrac{v_B^2}{n_B - 1}} .

The formula comes from matching the mean and variance of $v_A + v_B$ , computed for normal populations, with those of a multiple of a chi-square distribution, and then replacing the unknown population variances by their estimates. The value of $\nu$ is never below the smaller of $n_A - 1$ and $n_B - 1$ and never above $n_A + n_B - 2$ , and it is usually not a whole number.

In plain words The degrees of freedom describe how much information the estimated standard error carries. When one group’s contribution dominates, $\nu$ is close to that group’s $n - 1$ , because the standard error then rests mostly on one sample variance. The value reaches its maximum $n_A + n_B - 2$ exactly when $v_A/(n_A - 1) = v_B/(n_B - 1)$ . A fractional value is fine, since it labels a t distribution, not a number of observations.

Example For the interface data $v_A = 0.8$ and $v_B = 1.2$ , so

\begin{aligned} \nu &= \frac{(0.8 + 1.2)^2}{0.8^2/19 + 1.2^2/29} \\[0.4em] &\approx 47.997 . \end{aligned}

This is just below the upper limit $20 + 30 - 2 = 48$ , because the two contributions are nearly balanced relative to their degrees of freedom, with $0.8/19 \approx 0.0421$ and $1.2/29 \approx 0.0414$ . The lower limit is $19$ .

Welch confidence interval

Definition Under the conditions of the Welch t test, an approximate 95 percent confidence interval for $\mu_A - \mu_B$ is

(\bar{x}_A - \bar{x}_B) \pm t^{*} \, \widehat{\operatorname{SE}},

where $t^{*}$ is the $0.975$ quantile of the t distribution with $\nu$ Welch-Satterthwaite degrees of freedom. Other confidence levels use the matching quantile. The confidence level describes the procedure. Over repeated samples, intervals built this way capture $\mu_A - \mu_B$ about 95 percent of the time. The interval excludes a value $\Delta_0$ exactly when the two-sided Welch test of $H_0: \mu_A - \mu_B = \Delta_0$ gives a p-value below $0.05$ .

In plain words The interval reports how large the difference might be, which a p-value alone does not. It concerns the difference between two population means. It is not a range for individual times, and a difference in means does not say that every member of one group finishes faster than every member of the other.

Example With $\nu \approx 47.997$ the quantile is $t^{*} \approx 2.0106$ , so the margin is $2.0106 \times \sqrt{2} \approx 2.843$ minutes, and the interval $-4 \pm 2.843$ runs from about $-6.84$ to $-1.16$ minutes. Both endpoints are negative, so in the order A minus B the data are consistent with a population mean for interface A that is lower by roughly 1.2 to 6.8 minutes. The interval excludes zero, in agreement with the p-value $0.0068$ . Reversing the order to B minus A gives the estimate $4$ and the interval from about $1.16$ to $6.84$ minutes, while the standard error, the degrees of freedom and the two-sided p-value stay the same.

Pooled variance and the equal-variance t test

Definition Suppose the two populations have a common standard deviation, $\sigma_A = \sigma_B = \sigma$ . The pooled variance

\begin{aligned} s_p^2 &= \frac{(n_A - 1)s_A^2}{n_A + n_B - 2} \\[0.4em] &\quad + \frac{(n_B - 1)s_B^2}{n_A + n_B - 2} \end{aligned}

estimates $\sigma^2$ by weighting each sample variance by its degrees of freedom. The equal-variance t test, also called the pooled t test, uses the statistic

t = \frac{(\bar{x}_A - \bar{x}_B) - \Delta_0}{s_p\sqrt{1/n_A + 1/n_B}}

with $n_A + n_B - 2$ degrees of freedom. If the samples are independent random samples from normal populations with equal variances, this statistic has exactly that t distribution when $H_0: \mu_A - \mu_B = \Delta_0$ is true. The matching confidence interval replaces $\widehat{\operatorname{SE}}$ by $s_p\sqrt{1/n_A + 1/n_B}$ and takes $t^{*}$ from the t distribution with $n_A + n_B - 2$ degrees of freedom. Equal variances are an extra assumption of this model, and computing $s_p$ does not establish them.

In plain words Pooling assumes one shared spread. In return, under its assumptions, it gives an exact t reference distribution and at least as many degrees of freedom as Welch’s method. If both the spreads and the sample sizes differ, the pooled estimate of the variance of $\bar{x}_A - \bar{x}_B$ is biased. On average it is too small when the smaller sample comes from the population with the larger variance, which tends to make the test reject a true null hypothesis too often, and too large in the opposite case. When $n_A = n_B$ the pooled and Welch standard errors are equal, although their degrees of freedom usually differ. Choose the method from the design and from what is known about the spreads before seeing the results, not by a preliminary test of equal variances and not by whichever p-value is smaller.

Example For the interface data $19 \times 16 + 29 \times 36 = 1348$ , so

s_p^2 = \frac{1348}{48} \approx 28.08

square minutes. Since $1/20 + 1/30 = 1/12$ , the pooled standard error is $\sqrt{1348/576} \approx 1.530$ minutes. The statistic is $-4/1.530 \approx -2.615$ with $48$ degrees of freedom, and the two-sided p-value is about $0.012$ . The pooled standard error is larger than Welch’s $1.414$ here because the larger sample also has the larger standard deviation, and pooling gives group A a variance of about $28.08$ in place of its own $16$ . Both methods reject $H_0$ at the level $0.05$ for these data, but that agreement is a feature of these numbers, not a rule.

Check yourself

A study randomly assigns 29 volunteers to two checkout pages, 9 to page P and 20 to page Q, and records each person’s completion time in seconds. Page P gives $\bar{x}_P = 64$ and $s_P = 6$ , and page Q gives $\bar{x}_Q = 71$ and $s_Q = 10$ . Using the order P minus Q, find the estimate, its standard error, the Welch degrees of freedom and a 95 percent Welch interval, given $t^{*} \approx 2.062$ for those degrees of freedom. What changes if the order is Q minus P?

Show the answer

The estimate is $64 - 71 = -7$ seconds. The variance contributions are $36/9 = 4$ and $100/20 = 5$ square seconds, so the estimated standard error is $\sqrt{9} = 3$ seconds. The degrees of freedom are

\begin{aligned} \nu &= \frac{(4 + 5)^2}{4^2/8 + 5^2/19} \\[0.4em] &= \frac{171}{7} \approx 24.43, \end{aligned}

which lies between $8$ and $27$ as it must. The margin is $2.062 \times 3 \approx 6.19$ seconds, so the interval runs from about $-13.19$ to $-0.81$ seconds. Because the pages were randomly assigned, the interval supports the conclusion that page P lowered the mean completion time for these volunteers, by an amount the interval puts between about 0.8 and 13.2 seconds. In the order Q minus P the estimate becomes $7$ and the interval runs from about $0.81$ to $13.19$ seconds. The standard error, the degrees of freedom and the two-sided p-value stay the same, and the t statistic changes sign.

A classmate analyzes the data of question 1 with both the Welch test and the equal-variance t test. The Welch test gives a two-sided p-value of about $0.028$ and the pooled test about $0.063$ . The classmate reports the Welch result because it is significant and adds that the pooled test shows the two pages perform the same. What is wrong with this?

Show the answer

There are two mistakes. Choosing between analyses after seeing which one gives the smaller p-value gives the procedure two chances to reject, so the reported p-value overstates the evidence. The method should be fixed in advance, and Welch’s test is a reasonable advance choice here, since nothing in the design guarantees equal spreads and the sample sizes differ. The pooled analysis also assumes a common variance and gives page P the pooled variance $2188/27 \approx 81.0$ in place of its own $36$ , which inflates the standard error of the difference to about $3.61$ seconds.

The second mistake is reading a p-value above $0.05$ as evidence of no difference. Even the pooled analysis gives a 95 percent interval from about $-14.4$ to $0.4$ seconds, which contains differences of up to 14 seconds in favor of page P. Not significant means the data did not rule out zero, not that the effect is zero.

Return to the interface data with means 18 and 22 minutes. What does the Welch analysis support in each of the following designs?
- Fifty volunteers were randomly assigned, 20 to interface A and 30 to interface B.
- Random samples were drawn of 20 people who already use interface A at work and 30 who already use interface B.
- Each of 25 people completed the task once with each interface.

Show the answer

In the first design random assignment supports a causal conclusion, that interface A lowers the mean completion time for these volunteers, by roughly 1.2 to 6.8 minutes according to the interval. The volunteers were not sampled from a wider population, so extending the result to other people needs a separate argument.

In the second design random sampling supports a conclusion about the two populations of users, namely that the population mean time is lower among users of interface A. The people did not get their interfaces by chance, though, so the difference could come from experience, job type or other variables. This is an association, not evidence that the interface causes the difference.

In the third design every person contributes a time to both groups, so the samples are paired, not independent, and the Welch standard error does not apply. The analysis should use the 25 within-person differences with a paired t test, ideally with the order of the interfaces randomized for each person. The p-value itself cannot tell which design produced the data.

Comparing Two Independent Means

Key terms

Independent samples

Standard error of a difference

Welch t test

Welch-Satterthwaite degrees of freedom

Welch confidence interval

Pooled variance and the equal-variance t test

Check yourself

How this lesson connects

Builds on

Used later in