One-Way ANOVA and Welch ANOVA, statistics lesson

Three training methods are randomly assigned to nine people, three per method, and each person’s time to finish a task is recorded in minutes. The method means are 4, 6 and 8 minutes, but the times inside each method also spread over several minutes. Are the means further apart than that scatter alone would make them? Running a separate t test for every pair asks three questions instead of one and raises the chance of a false alarm. One-way ANOVA answers the single question of whether all the population means are equal, and Welch ANOVA answers it without assuming equal variances.

Key terms

One-way ANOVA

Definition One-way analysis of variance (ANOVA) compares the means of $k \ge 2$ groups defined by a single factor, a categorical variable such as training method. Write $y_{ij}$ for observation $i$ in group $j$ , where group $j$ has $n_j$ observations and $N = n_1 + \dots + n_k$ is the total. The classical model is

y_{ij} = \mu_j + \varepsilon_{ij},

where $\mu_j$ is the population mean of group $j$ and the errors $\varepsilon_{ij}$ are independent and normally distributed with mean $0$ and the same variance $\sigma^2$ in every group. The test compares

\begin{gathered} H_0\colon \mu_1 = \mu_2 = \dots = \mu_k, \\[0.4em] H_1\colon \text{not all } \mu_j \text{ equal}. \end{gathered}

The groups must come from random samples of the populations being compared or from random assignment of units to treatments, and the conclusions reach only as far as that design allows. Rejecting $H_0$ says that at least two population means differ. It does not say which ones, and it does not say that all of them differ. With $k = 2$ groups the test is the two-sided pooled two-sample t test, and the F statistic defined below equals the square of the pooled t statistic.

In plain words Rather than comparing the groups two at a time, ANOVA asks one question about all of them. It answers by comparing how far the group means spread apart with how much single observations scatter inside their own groups. The name refers to an analysis of variation, but the hypothesis is about means.

Example The constructed completion times are $2$ , $4$ and $6$ minutes for method A, $4$ , $6$ and $8$ for method B, and $6$ , $8$ and $10$ for method C. The group sample means are $\bar{y}_A = 4$ , $\bar{y}_B = 6$ and $\bar{y}_C = 8$ minutes, and the grand mean of all nine times is $\bar{y} = 6$ minutes. The null hypothesis is $H_0\colon \mu_A = \mu_B = \mu_C$ , against the alternative that the three population mean times are not all equal. Because the methods were randomly assigned, a rejection would support the causal conclusion that the methods do not all produce the same mean time for the people in the study.

Within-group sum of squares

Definition Let $\bar{y}_j$ be the sample mean of group $j$ . The within-group sum of squares is

\text{SS}_{\text{W}} = \sum_{j=1}^{k} \sum_{i=1}^{n_j} (y_{ij} - \bar{y}_j)^2 ,

the sum of the squared deviations of all observations from their own group means. It is also called the error or residual sum of squares, and it carries the squared units of the response. For a group with $n_j \ge 2$ observations the inner sum equals $(n_j - 1)s_j^2$ , where $s_j^2$ is the sample variance of that group, and a group with a single observation contributes $0$ .

In plain words This sum measures the variation that the grouping does not explain, the differences among units that received the same treatment. Shifting a whole group up or down leaves it unchanged, so on its own it says nothing about whether the group means differ.

Example In method A the deviations from the group mean $4$ are $-2$ , $0$ and $2$ , and their squares add to $8$ . Methods B and C show the same pattern around their own means, so each also contributes $8$ , and $\text{SS}_{\text{W}} = 24$ square minutes. Equivalently, every group has sample variance $s_j^2 = 4$ , and $(3 - 1) \cdot 4 = 8$ for each group.

Between-group sum of squares

Definition Let $\bar{y}$ be the grand mean of all $N$ observations, which equals $\sum_j n_j\bar{y}_j / N$ . The between-group sum of squares is

\text{SS}_{\text{B}} = \sum_{j=1}^{k} n_j (\bar{y}_j - \bar{y})^2 ,

the squared distance of each group mean from the grand mean, weighted by the group size. It equals $0$ exactly when all the group sample means are equal. With the total sum of squares $\text{SS}_{\text{T}} = \sum_j \sum_i (y_{ij} - \bar{y})^2$ , every data set satisfies

\text{SS}_{\text{T}} = \text{SS}_{\text{B}} + \text{SS}_{\text{W}} .

To see why, write $y_{ij} - \bar{y} = (y_{ij} - \bar{y}_j) + (\bar{y}_j - \bar{y})$ and square. The cross terms add to zero, because the deviations inside each group sum to zero.

In plain words Replace every observation by its group mean and measure how much these replaced values vary around the grand mean. That is the part of the total variation the groups account for. The total splits exactly into this part and the scatter inside the groups, with nothing created or lost.

Example For the training data,

\begin{aligned} \text{SS}_{\text{B}} &= 3(4 - 6)^2 \\[0.4em] &\quad + 3(6 - 6)^2 \\[0.4em] &\quad + 3(8 - 6)^2 = 24 . \end{aligned}

The nine deviations from the grand mean are $-4, -2, 0, -2, 0, 2, 0, 2, 4$ , whose squares add to $\text{SS}_{\text{T}} = 48$ . Indeed $48 = 24 + 24$ .

Mean square

Definition A mean square is a sum of squares divided by its degrees of freedom. One-way ANOVA needs $N > k$ , so that some observations are left for estimating the scatter inside groups, and uses

\begin{gathered} \text{MS}_{\text{B}} = \frac{\text{SS}_{\text{B}}}{k - 1}, \\[0.4em] \text{MS}_{\text{W}} = \frac{\text{SS}_{\text{W}}}{N - k}. \end{gathered}

The between-group sum has $k - 1$ degrees of freedom because the weighted deviations $n_j(\bar{y}_j - \bar{y})$ sum to zero. The within-group sum has $N - k$ because the deviations inside each of the $k$ groups sum to zero. Together they make up the total $N - 1$ . When the observations are independent with a common variance $\sigma^2$ , normal or not,

\begin{aligned} E(\text{MS}_{\text{W}}) &= \sigma^2, \\[0.4em] E(\text{MS}_{\text{B}}) &= \sigma^2 + \frac{\delta}{k - 1}, \end{aligned}

where $\delta = \sum_j n_j(\mu_j - \bar{\mu})^2$ and $\bar{\mu} = \sum_j n_j\mu_j / N$ . So the within-group mean square estimates $\sigma^2$ whether or not $H_0$ holds, while the between-group mean square has expectation $\sigma^2$ when all the $\mu_j$ are equal and a larger expectation otherwise. The within-group mean square is the pooled variance, and with two groups it is the pooled variance of the two-sample t test.

In plain words A sum of squares grows with the number of pieces it adds up, so each one is divided by the number of independent pieces of information it holds. The results are two estimates of the error variance on the same footing. The within-group estimate is unbiased under either hypothesis, and the between-group estimate is pushed upward, on average, when the population means differ.

Example For the training data $k = 3$ and $N = 9$ , so the degrees of freedom are $2$ and $6$ .

Source	SS	df	MS
Between groups	$24$	$2$	$12$
Within groups	$24$	$6$	$4$
Total	$48$	$8$

In a balanced design the within-group mean square is the plain average of the group sample variances, here the average of $4$ , $4$ and $4$ .

F statistic

Definition The one-way ANOVA F statistic is

F = \frac{\text{MS}_{\text{B}}}{\text{MS}_{\text{W}}} .

Under the classical model, with independent normal errors of common variance, and when $H_0$ is true, the statistic has the F distribution with $k - 1$ numerator and $N - k$ denominator degrees of freedom, written $F_{k-1,\,N-k}$ . Large values count against $H_0$ , so the p-value is the upper tail probability

P(F_{k-1,\,N-k} \ge F_{\text{obs}}),

computed assuming $H_0$ . Only the upper tail is used, although the test detects differences between the means in every direction.

In plain words The ratio compares the spread of the group means with the spread that noise alone would give them. With equal population means the ratio varies around $1$ , and a ratio far above $1$ is evidence that not all population means are equal. A significant F says that some means differ, not which ones or by how much, and a nonsignificant F does not show that the means are equal.

Example For the training data $F = 12/4 = 3$ with $2$ and $6$ degrees of freedom. For two numerator degrees of freedom the tail has the closed form $P(F_{2,\nu} \ge f) = (1 + 2f/\nu)^{-\nu/2}$ , so

P(F_{2,6} \ge 3) = 2^{-3} = 0.125 .

At a level of $0.05$ fixed in advance the data do not reject $H_0$ , and the 5 percent critical value of $F_{2,6}$ , about $5.14$ , is well above $3$ . This is not evidence that the methods are equally fast. With three people per method, the data are compatible with equal population means and also with true differences larger than the ones observed.

Welch ANOVA

Definition Welch’s ANOVA tests the same $H_0$ while allowing the group variances $\sigma_j^2$ to differ. It assumes independent observations, independent groups and approximately normal data within each group, and it needs $n_j \ge 2$ and $s_j^2 > 0$ in every group. With weights $w_j = n_j / s_j^2$ , their sum $W$ and the weighted mean $\tilde{y} = \sum_j w_j\bar{y}_j / W$ , let

\Lambda = \sum_{j=1}^{k} \frac{(1 - w_j/W)^2}{n_j - 1} .

The statistic is $F_W = M/C$ , where

\begin{gathered} M = \tfrac{1}{k - 1}\sum_{j} w_j(\bar{y}_j - \tilde{y})^2, \\[0.4em] C = 1 + \frac{2(k - 2)}{k^2 - 1}\Lambda . \end{gathered}

Under $H_0$ it is compared with an F distribution with $k - 1$ and $\nu_2 = (k^2 - 1)/(3\Lambda)$ degrees of freedom, where $\nu_2$ need not be a whole number. With two groups the statistic is the square of Welch’s t statistic and $\nu_2$ is the Welch-Satterthwaite degrees of freedom. The F reference is only an approximation, and the method is not free of assumptions. It still needs independence, and its p-value is less accurate for very small or strongly skewed groups.

In plain words Classical ANOVA pools all groups into one variance estimate. When the variances differ, and especially when the group sizes differ as well, the pooled estimate misstates the precision of some group means, and the classical test can reject a true $H_0$ much more or much less often than its stated level. It rejects too often when the smaller groups have the larger variances. Welch’s version weights each mean by its estimated precision, so a noisy mean counts for less, and it adjusts the degrees of freedom for the estimated variances.

Example Change method B to the times $2$ , $6$ and $10$ minutes and keep A and C. The means stay $4$ , $6$ and $8$ , but the sample variances become $4$ , $16$ and $4$ , so the weights are $3/4$ , $3/16$ and $3/4$ with $W = 27/16$ . The weighted mean is $\tilde{y} = 6$ by symmetry, and $\tfrac34(4 - 6)^2 + 0 + \tfrac34(8 - 6)^2 = 6$ , so $M = 6/2 = 3$ . Then $\Lambda = 19/27$ and $C = 127/108$ , which give

\begin{gathered} F_W = \frac{324}{127} \approx 2.551, \\[0.4em] \nu_2 = \frac{72}{19} \approx 3.789 . \end{gathered}

The p-value from $F_{2,\,3.789}$ is about $0.199$ . The noisy mean of B receives a quarter of the weight of each other mean, and the denominator degrees of freedom fall well below the classical $6$ .

Check yourself

Give every method the same three times, $4$ , $6$ and $8$ minutes. Find the between-group and within-group sums of squares and the F statistic. What does the result say about the population means?

Show the answer

Every group mean equals $6$ , which is also the grand mean, so $\text{SS}_{\text{B}} = 0$ . In each group the squared deviations add to $8$ , so $\text{SS}_{\text{W}} = 24$ and $\text{MS}_{\text{W}} = 24/6 = 4$ . Hence $\text{MS}_{\text{B}} = 0$ and $F = 0/4 = 0$ , the smallest possible value, with p-value $1$ .

The sample means show no difference at all, but no sample can show that population means are exactly equal. With three people per method, a 95 percent confidence interval for any one pairwise difference runs from about $-4.0$ to $4.0$ minutes, so the data are also compatible with differences of a few minutes in either direction.

A study randomly assigns $15$ people to three methods, five per method, and reports $\text{SS}_{\text{B}} = 40$ and $\text{SS}_{\text{W}} = 60$ . Find the mean squares, the F statistic and its p-value, and decide at level $0.05$ . A colleague then writes that all three methods have different mean times. Is that right?

Show the answer

The degrees of freedom are $3 - 1 = 2$ and $15 - 3 = 12$ , so $\text{MS}_{\text{B}} = 20$ , $\text{MS}_{\text{W}} = 5$ and $F = 4$ . The closed form for two numerator degrees of freedom gives

\begin{aligned} P(F_{2,12} \ge 4) &= \bigl(\tfrac{5}{3}\bigr)^{-6} \\[0.4em] &= 0.6^6 , \end{aligned}

which is about $0.0467$ . That is below $0.05$ , so $H_0$ is rejected. Under the model’s conditions the data are evidence that the three population means are not all equal.

The colleague’s sentence does not follow. Rejecting $H_0$ means at least two means differ, which would also be true if one method differed and the other two were equal. Finding out which methods differ needs follow-up comparisons with control of the error rate across the comparisons, and the size of the differences needs estimates with intervals. A significant result also does not show that the differences are large enough to matter.

For the changed data, A with $2, 4, 6$ , B with $2, 6, 10$ and C with $6, 8, 10$ , compute the classical F statistic and its p-value and compare them with the Welch result above. Someone suggests using the classical analysis because a test of equal variances on these data was not significant. Comment.

Show the answer

The means have not changed, so $\text{SS}_{\text{B}} = 24$ and $\text{MS}_{\text{B}} = 12$ . The within-group sum is $8 + 32 + 8 = 48$ , so $\text{MS}_{\text{W}} = 48/6 = 8$ and $F = 1.5$ with $2$ and $6$ degrees of freedom, giving

P(F_{2,6} \ge 1.5) = 1.5^{-3} = \tfrac{8}{27} ,

which is about $0.296$ . The classical analysis pools the variances $4$ , $16$ and $4$ into $8$ , which overstates the uncertainty of the means of A and C and understates that of B. Welch’s analysis uses each group’s own variance and gives $F_W \approx 2.551$ with p about $0.199$ . Neither rejects $H_0$ at $0.05$ .

With three observations per group, a test of equal variances has little power, so failing to reject does not show that the variances are equal. The choice of analysis should come from the design and what is known about the variability, and Welch’s version is a common choice whenever the variances may differ. It still requires independent observations and roughly normal data within groups.

One-Way ANOVA and Welch ANOVA

Key terms

One-way ANOVA

Within-group sum of squares

Between-group sum of squares

Mean square

F statistic

Welch ANOVA

Check yourself

How this lesson connects

Builds on

Used later in