Kruskal-Wallis Tests and Comparing Several Distributions, statistics lesson

Waiting times are recorded for independent random samples of patients at three clinics. Such times are skewed, with a few very long waits, so a comparison built on a normal model for the group means is doubtful. A natural question is whether the three clinics have the same distribution of waiting times, and the Kruskal-Wallis test addresses it using only the order of the observations. This lesson builds the test statistic from pooled ranks, states the null hypothesis it actually tests, compares exact and chi-square p-values, handles ties, and separates the omnibus conclusion from claims about particular pairs of groups.

Key terms

Pooled ranks

Definition Suppose $k \ge 2$ independent groups have sizes $n_1, \dots, n_k$ , and let $N = n_1 + \dots + n_k$ . Pool all $N$ observations, sort them, and give them the ranks $1, 2, \dots, N$ from the smallest to the largest. Observations with equal values receive the average of the ranks they would otherwise occupy, called their midrank. The rank sum $R_j$ of group $j$ is the sum of the pooled ranks of its observations, and its mean rank is $\bar{R}_j = R_j / n_j$ . With or without ties the $N$ ranks add up to $N(N + 1)/2$ , so the overall mean rank is

\bar{R} = \frac{N + 1}{2} .

In plain words Ranking all the observations together puts every group on one common ladder, and the mean ranks show which groups sit mostly low and which mostly high. Ranks keep only the order of the values. A strictly increasing change of scale, such as taking logarithms of positive values, leaves every rank unchanged, and an extreme value never gets a rank above $N$ , however far out it lies.

Example Take the constructed samples $A = \{1, 2, 3\}$ , $B = \{4, 5, 6\}$ and $C = \{7, 8, 9\}$ . All nine values are distinct, so their pooled ranks are $1$ to $9$ and here coincide with the values. The rank sums are $R_A = 6$ , $R_B = 15$ and $R_C = 24$ , which add to $45 = 9 \cdot 10/2$ . The mean ranks are $2$ , $5$ and $8$ , around the overall mean rank $5$ . If the value $9$ were replaced by $900$ , every rank would stay the same.

Kruskal-Wallis test

Definition The Kruskal-Wallis test compares $k \ge 2$ independent groups through their pooled ranks. Its statistic is

\begin{aligned} H &= \frac{12}{N(N + 1)} \sum_{j=1}^{k} \frac{R_j^2}{n_j} \\[0.4em] &\quad - 3(N + 1) . \end{aligned}

Because the ranks always add up to $N(N + 1)/2$ , the same number can be written as

\begin{aligned} H &= \frac{12}{N(N + 1)} \\[0.4em] &\quad \times \sum_{j=1}^{k} n_j \bigl(\bar{R}_j - \bar{R}\bigr)^2 . \end{aligned}

So $H \ge 0$ , with $H = 0$ exactly when all groups have the same mean rank. Large values are evidence against the null hypothesis that the $k$ population distributions are identical. The p-value is the probability under that null hypothesis of a value of $H$ at least as large as the observed one, computed exactly by permutation or approximately from a chi-square distribution, and with ties the statistic is divided by the tie correction factor described below. The test requires independent observations, within groups and between groups, and a response whose values can be ordered. For $k = 2$ the statistic is the square of the standardized Wilcoxon rank-sum statistic, so the test extends the two-sided Mann-Whitney test to more than two groups.

In plain words Rank everything together and ask whether some groups collect mostly small ranks and others mostly large ones. The statistic measures how far the group mean ranks spread around the middle rank. Ranks reduce the pull of extreme values and need no normal model. They do not repair dependent observations, a biased selection of units or measurements that mean different things in different groups. Repeated measurements on the same units call for a method for related samples, such as the Friedman test.

Example For the constructed samples, $N = 9$ and every $n_j = 3$ , so

\begin{aligned} H &= \frac{12}{90}\Bigl(\frac{6^2}{3} + \frac{15^2}{3} + \frac{24^2}{3}\Bigr) \\[0.4em] &\quad - 3 \cdot 10 \\[0.4em] &= \frac{12}{90} \cdot 279 - 30 = 7.2 . \end{aligned}

The mean-rank form gives the same value, since $\tfrac{12}{90} \cdot 3\bigl((2 - 5)^2 + 0^2 + (8 - 5)^2\bigr) = \tfrac{12}{90} \cdot 54 = 7.2$ . This is the largest value the statistic can take for three groups of three without ties, because the three groups occupy three separate blocks of ranks.

Identical-distribution null hypothesis

Definition Let $F_j$ be the population distribution of the response in group $j$ . The usual null hypothesis of the Kruskal-Wallis test is

H_0\colon F_1 = F_2 = \dots = F_k ,

and the alternative is that the $F_j$ are not all equal. If the groups are independent random samples, then under $H_0$ all $N$ observations are independent draws from one distribution, so they are exchangeable. Given the pooled set of observed values, every allocation of the $N$ observed units to groups of the fixed sizes $n_1, \dots, n_k$ is then equally likely, and there are

\frac{N!}{n_1! \, n_2! \cdots n_k!}

such allocations. The exact permutation p-value is the fraction of them whose statistic is at least the observed value. In a randomized experiment the same reference follows from the random assignment itself, under the null hypothesis that each unit would give the same response in whichever group it was placed. The statistic reacts mainly to differences in mean ranks, that is, to some groups tending to give larger values than others. Distributions that differ only in spread around a common center of symmetry give every group the same expected mean rank, and the test is not designed to detect such differences.

In plain words The null hypothesis says that group membership makes no difference at all to the distribution of the response. Then the group labels are arbitrary tags, and shuffling them among the observed values produces data just as plausible as the real ones. A statement about medians needs more. Under a location-shift model, $F_j(x) = F(x - \theta_j)$ for one continuous distribution $F$ , the groups share shape and spread and differ only by shifts, so $H_0$ says the shifts are equal and a rejection is evidence that the medians are not all equal. Without such a model, a rejection says the distributions are not all identical, and the difference may involve location, spread or shape.

Example For the constructed samples there are

\frac{9!}{3! \, 3! \, 3!} = 1680

allocations of the nine values to the labeled groups $A$ , $B$ and $C$ , each of size three. Only six reach a statistic of at least $7.2$ , namely those that give the blocks $\{1, 2, 3\}$ , $\{4, 5, 6\}$ and $\{7, 8, 9\}$ to the three groups in one of the $3! = 6$ orders. The exact p-value is

\begin{aligned} p &= \frac{6}{1680} = \frac{1}{280} \\[0.4em] &\approx 0.00357 . \end{aligned}

Chi-square approximation

Definition Suppose the $k$ groups are independent samples from one continuous distribution. As every $n_j$ grows with $k$ fixed, the distribution of $H$ approaches the chi-square distribution $\chi^2_{k-1}$ . The approximate p-value for an observed value $h$ is $P(\chi^2_{k-1} \ge h)$ , with $h$ replaced by the tie-corrected value when there are ties. This is a large-sample result. A common rule of thumb asks for at least five observations in every group. With smaller groups or heavy ties, an exact permutation p-value is preferred, or a Monte Carlo p-value computed from many random allocations of the labels.

In plain words The chi-square curve is a smooth stand-in for the exact permutation distribution. Without ties the exact null distribution of $H$ has mean $k - 1$ , the same as $\chi^2_{k-1}$ , but with tiny groups the statistic takes comparatively few distinct values, $29$ for three groups of three without ties, and its tail can be far from the tail of the curve, in either direction.

Example With $k = 3$ the reference is $\chi^2_2$ , whose upper tail is $P(\chi^2_2 \ge x) = e^{-x/2}$ for $x \ge 0$ . For the constructed samples the approximate p-value is $e^{-3.6} \approx 0.0273$ , more than seven times the exact value $1/280 \approx 0.00357$ . Within the same design the error also runs the other way. The exact probability of a statistic of at least $3.2$ is $426/1680 \approx 0.254$ , while the curve gives $e^{-1.6} \approx 0.202$ . Cutoffs differ as well. The exact null distribution for three groups of three without ties gives $P(H \ge 5.6) = 0.05$ , while the chi-square cutoff for $0.05$ is about $5.991$ .

Tie correction

Definition A tie occurs when two or more observations share a value, and tied observations receive midranks. Midranks vary less than the untied ranks $1, \dots, N$ , which shrinks the statistic relative to its null scale. Suppose the values that occur more than once form tie blocks of sizes $t_1, \dots, t_m$ . The tie correction factor is

C = 1 - \frac{\sum_{\ell=1}^{m} (t_\ell^3 - t_\ell)}{N^3 - N} ,

and the corrected statistic is $H / C$ . With no ties $C = 1$ . The corrected statistic equals $N - 1$ times the between-group sum of squares of the ranks divided by their total sum of squares, which is the general definition of the Kruskal-Wallis statistic. When the observations are not all equal, $0 < C \le 1$ , so the correction never lowers the statistic. When all $N$ observations are equal, $C = 0$ and the corrected statistic is undefined, since the data contain no ordering information at all.

In plain words Tied observations share their ranks, because putting one ahead of the other would invent information. Sharing makes the ranks less spread out, and dividing by $C$ restores the scale the chi-square approximation expects. The factor $C$ depends only on the pooled values, so it is the same for every allocation and does not change an exact p-value. The exact distribution itself, however, must be recomputed from the tied ranks.

Example Change group $A$ to $\{1, 1, 3\}$ and keep $B$ and $C$ . The two observations equal to $1$ share ranks $1$ and $2$ , so each receives $1.5$ . The rank sums are still $6$ , $15$ and $24$ , so $H = 7.2$ as before. There is one tie block of size $2$ , and

\begin{aligned} C &= 1 - \frac{2^3 - 2}{9^3 - 9} \\[0.4em] &= 1 - \frac{6}{720} = \frac{119}{120} . \end{aligned}

The corrected statistic is $7.2 \cdot \tfrac{120}{119} \approx 7.2605$ , with chi-square p-value $e^{-7.2605/2} \approx 0.0265$ . Recomputing all $1680$ allocations with the tied ranks again finds six at least as extreme, so the exact p-value stays $1/280$ . The count stays at six because both tied observations fall in the same block of three, and a tie that straddles two blocks can change it.

Omnibus test

Definition An omnibus test tests one null hypothesis about all the groups at once, here $H_0\colon F_1 = \dots = F_k$ , against the alternative that the distributions are not all identical. A rejection is a statement about the whole set of groups. It does not say which groups differ, how many differ or in which direction. Questions about particular groups are answered by post-hoc comparisons, such as pairwise rank-sum tests or Dunn’s test, which compares mean pooled ranks. The family of comparisons, for example all $k(k - 1)/2$ pairs, should be stated, and the family-wise error rate controlled, for example with a Bonferroni or Holm adjustment. Running unadjusted pairwise tests whenever the omnibus test rejects does not control the family-wise error rate in general. With four or more groups, a real difference among some groups can make the omnibus test reject with high probability, and several true null pairs are then tested at the unadjusted level.

In plain words The omnibus test asks whether the groups are all alike. Which groups differ is a separate set of questions, with their own error control, and the two answers need not line up. A report is clearer when it shows the group distributions themselves, for example with plots, medians or mean ranks, rather than only the significance label.

Example For the constructed samples, the exact p-value $1/280$ is evidence, under independent sampling, that the three distributions are not all identical. That conclusion fits several patterns, such as $A$ alone differing from two identical distributions for $B$ and $C$ , or all three differing. Telling these apart needs a declared family, here the three pairs $A$ and $B$ , $A$ and $C$ , $B$ and $C$ . With a Bonferroni adjustment at family-wise level $0.05$ , each pairwise p-value is compared with $0.05/3 \approx 0.0167$ .

Check yourself

Independent samples give $A = \{12, 15, 31\}$ , $B = \{8, 10, 14\}$ and $C = \{20, 25, 40\}$ . Find the pooled ranks, the rank sums and the Kruskal-Wallis statistic. For three groups of three without ties, the exact null distribution gives $P(H \ge 5.6) = 0.05$ . Does the exact test reject at the 5 percent level?

Show the answer

Sorted, the values are $8, 10, 12, 14, 15, 20, 25, 31, 40$ with ranks $1$ to $9$ . Group $A$ has ranks $3$ , $5$ and $8$ , group $B$ has $1$ , $2$ and $4$ , and group $C$ has $6$ , $7$ and $9$ . The rank sums are $16$ , $7$ and $22$ , which add to $45$ as they should. Then

\begin{aligned} H &= \frac{12}{90} \cdot \frac{16^2 + 7^2 + 22^2}{3} \\[0.4em] &\quad - 30 \\[0.4em] &= \frac{12}{90} \cdot 263 - 30 \\[0.4em] &= \frac{76}{15} \approx 5.07 . \end{aligned}

The value is below $5.6$ , so the exact test does not reject at the 5 percent level. Enumerating the allocations gives the exact p-value $144/1680 = 3/35 \approx 0.0857$ . This does not show that the three distributions are identical. With three observations per group the test has little power, and the data are compatible with real differences.

For the constructed samples $A = \{1, 2, 3\}$ , $B = \{4, 5, 6\}$ and $C = \{7, 8, 9\}$ , the exact p-value is $1/280$ . A report concludes that group $A$ differs from group $B$ . Is that justified? As a follow-up, what is the smallest two-sided p-value an exact rank-sum test of two groups of three can give?

Show the answer

The omnibus result is evidence that the three distributions are not all identical, under independent sampling. It does not by itself support a claim about $A$ and $B$ in particular, which needs its own stated comparison with multiplicity control.

An exact rank-sum test of two groups of three considers the $\binom{6}{3} = 20$ equally likely allocations of six values. Complete separation in one direction is a single allocation, and the two-sided p-value counts both directions, so the smallest possible value is $2/20 = 0.1$ . Each of the three pairwise tests therefore gives at least $0.1$ , above $0.05$ and far above the Bonferroni level $0.05/3 \approx 0.0167$ . With separate rank-sum tests as the follow-up, no pair can be singled out at the 5 percent level, even though the omnibus test rejects. The two analyses answer different questions. A follow-up built on the pooled ranks of all nine observations, such as Dunn’s test, is a different procedure and can give smaller p-values for the most separated pair.

Change the data to $A = \{1, 2, 4\}$ , $B = \{4, 5, 6\}$ and $C = \{7, 8, 9\}$ . Find the midranks, the statistic, the tie correction factor and the corrected statistic. Can the count of six extreme allocations from the untied data be reused for the exact p-value?

Show the answer

The two observations equal to $4$ would occupy ranks $3$ and $4$ , so each receives $3.5$ . The ranks are $1, 2, 3.5$ in $A$ , $3.5, 5, 6$ in $B$ and $7, 8, 9$ in $C$ , with rank sums $6.5$ , $14.5$ and $24$ . The squared rank sums add to $42.25 + 210.25 + 576 = 828.5$ , so

\begin{aligned} H &= \frac{12}{90} \cdot \frac{828.5}{3} - 30 \\[0.4em] &= \frac{307}{45} \approx 6.822 . \end{aligned}

One tie block of size $2$ gives $C = 1 - 6/720 = 119/120$ , so the corrected statistic is about $6.880$ , with chi-square p-value $e^{-6.880/2} \approx 0.0321$ .

The old count cannot be reused. The group sizes are still $3$ , $3$ and $3$ , so there are still $1680$ allocations, but the possible values of the statistic have changed. The largest value is now about $6.880$ instead of $7.2$ , and it is reached by twelve allocations, because the two observations equal to $4$ can trade places between the lowest and middle blocks without changing any rank. The exact p-value is $12/1680 = 1/140 \approx 0.00714$ , twice the value for the untied data.

Kruskal-Wallis Tests and Comparing Several Distributions

Key terms

Pooled ranks

Kruskal-Wallis test

Identical-distribution null hypothesis

Chi-square approximation

Tie correction

Omnibus test

Check yourself

How this lesson connects

Builds on