Percentiles, Quartiles and Box Plots, statistics lesson

The last eight trips on a bus route arrived $1$ , $2$ , $3$ , $4$ , $5$ , $6$ , $7$ and $20$ minutes late. The mean delay is $6$ minutes, even though the two middle delays are $4$ and $5$ minutes, because one long delay pulls the mean upward. To compare routes fairly we want summaries that show where the bulk of the trips lies and how wide the middle half is, without letting a single trip dominate. This lesson builds such summaries from positions in the ordered data, draws them as a box plot, and adds two resistant measures, the median absolute deviation and the trimmed mean.

Key terms

Percentile

Definition For $0 < p < 100$ , a $p$ th percentile is a value that splits a distribution so that $p$ percent of it lies below the value and $100 - p$ percent lies above it. A quantile is the same idea with the fraction written between $0$ and $1$ , so the 25th percentile is the $0.25$ quantile, and the median is the 50th percentile. For a finite set of values, such as a sample, an exact split may be impossible, or a whole interval of values may give it. A sample percentile is therefore computed with a stated rule, and the rules in common use, which differ in how they interpolate between neighboring ordered values, can give different answers in small samples.

In plain words A percentile marks a position in the ordered data. About $p$ percent of the observations lie below it and the rest above it. In a large dataset the different rules usually agree closely, but in a small one they can disagree, so a reported percentile should come with the rule that produced it, especially when results from different software are compared.

Example In the eight delays, every value strictly between $2$ and $3$ minutes has two delays below it and six above it, an exact split of 25 percent and 75 percent. A rule that takes the midpoint gives $2.5$ minutes, while the linear interpolation rule that many software packages use by default gives $2.75$ minutes. Both are reasonable 25th percentiles, and neither is an arithmetic mistake.

Quartiles and interquartile range

Definition The quartiles $Q_1$ , $Q_2$ and $Q_3$ are the 25th, 50th and 75th percentiles, so they cut ordered data into four parts with about a quarter of the observations in each, and $Q_2$ is the median. Like other sample percentiles they need a stated rule. This page uses the median-of-halves rule for $n \ge 2$ observations. Sort the data and find the median. The lower half consists of the observations before the median position and the upper half of those after it, and when $n$ is odd the single middle observation is left out of both halves. Then $Q_1$ is the median of the lower half and $Q_3$ is the median of the upper half. The interquartile range is

\text{IQR} = Q_3 - Q_1 ,

measured in the units of the data.

In plain words The quartiles bracket the middle half of the ordered data, and the IQR is the width of that middle half. It is a measure of spread built from positions near the middle. Under the median-of-halves rule, in a dataset of at least six values, moving the largest observation farther up or the smallest farther down leaves it unchanged.

Example For the eight delays the median is halfway between the fourth and fifth values, $Q_2 = (4 + 5)/2 = 4.5$ minutes. The lower half $1, 2, 3, 4$ has median $Q_1 = 2.5$ , and the upper half $5, 6, 7, 20$ has median $Q_3 = 6.5$ , so $\text{IQR} = 6.5 - 2.5 = 4$ minutes. The middle half of the delays lies between $2.5$ and $6.5$ minutes, and changing the largest delay from $20$ to $200$ minutes would leave all three quartiles and the IQR exactly as they are. The linear interpolation rule gives $Q_1 = 2.75$ and $Q_3 = 6.25$ instead, with $\text{IQR} = 3.5$ minutes.

Five-number summary and box plot

Definition The five-number summary lists the minimum, $Q_1$ , the median, $Q_3$ and the maximum. A box plot draws these positions along a number line. A box runs from $Q_1$ to $Q_3$ , a line inside the box marks the median, and whiskers extend outward from the box. In the simple version the whiskers reach the minimum and the maximum. In the modified version, the default in much statistical software, each whisker stops at the most extreme observation that is not beyond its outlier fence, and any observation beyond a fence is drawn as a separate point. A box plot should state which version it shows and which quartile rule it uses.

In plain words A box plot compresses a dataset into a few positions. It shows the center, the middle half and the extent of the data at a glance, and it makes several groups easy to compare side by side. It does not show the sample size or the mean, and it can hide gaps, clusters and multiple peaks, so quite different datasets can produce the same box plot.

Example For the eight delays the five-number summary is $(1, 2.5, 4.5, 6.5, 20)$ minutes. A simple box plot has a box from $2.5$ to $6.5$ , a median line at $4.5$ and whiskers out to $1$ and $20$ . In the modified version the fences, found in the next card, are $-3.5$ and $12.5$ minutes. The upper whisker then stops at $7$ , the delay of $20$ minutes appears as a separate point, and the lower whisker still reaches $1$ .

Outlier fence

Definition The lower and upper outlier fences are

\begin{gathered} L = Q_1 - 1.5\,\text{IQR}, \\[0.4em] U = Q_3 + 1.5\,\text{IQR}. \end{gathered}

An observation below $L$ or above $U$ is flagged as a potential outlier. The factor $1.5$ is a convention, and the fences move when the quartile rule changes.

In plain words The fences screen for observations that lie unusually far from the middle half of the data. A flagged value deserves a closer look, at how it was recorded and at whether it came from the same process as the rest. The flag alone does not prove that the value is an error, and it is not a reason to delete it. Even a normal distribution has about 0.7 percent of its values beyond the fences computed from its own quartiles, so a large sample of perfectly ordinary normal data can be expected to contain some flagged points.

Example For the eight delays $\text{IQR} = 4$ , so $L = 2.5 - 6 = -3.5$ and $U = 6.5 + 6 = 12.5$ minutes. No delay lies below $-3.5$ , and only the delay of $20$ minutes lies above $12.5$ , so it is the one flagged. It could be a recording mistake, or a real trip held up by a road closure, and the flag says to find out which. With the linear interpolation quartiles $2.75$ and $6.25$ , the fences would be $-2.5$ and $11.5$ minutes, and the same delay would be flagged.

Median absolute deviation

Definition For observations $x_1, \dots, x_n$ with median $m$ , the median absolute deviation is the median of their distances from $m$ ,

\operatorname{MAD} = \operatorname{median}_{i} \lvert x_i - m \rvert .

This is the raw, or unscaled, MAD, and it is in the units of the data. Some software multiplies it by about $1.4826$ by default, so that for normal data it estimates the standard deviation. A reported MAD should say whether it is raw or scaled.

In plain words The MAD measures spread with two medians, one for the center and one for the typical distance from it. A summary is called resistant when changing a few observations, even to extreme values, can move it only a limited amount. In a dataset of at least three values, moving one observation, however far, changes the MAD by only a limited amount, while it can make the standard deviation as large as you like. Resistance has limits, though. If enough observations move, the MAD moves too.

Example For the constructed values $2, 3, 4, 5, 16$ , the median is $4$ . The distances from $4$ are $2$ , $1$ , $0$ , $1$ and $12$ , which in order are $0, 1, 1, 2, 12$ , so the raw MAD is $1$ . The sample standard deviation of the same values is about $5.70$ , inflated by the single value $16$ .

Trimmed mean

Definition Sort $n$ observations and choose a whole number $k$ with $2k < n$ . The trimmed mean removes the $k$ smallest and the $k$ largest observations and averages the remaining $n - 2k$ . The amount of trimming is usually described by the proportion $k/n$ removed from each tail. When a requested proportion does not give a whole number of observations, software rules differ in how they round, so the rule should be reported.

In plain words A trimmed mean is a compromise between the mean, which uses every value, and the median, which in effect trims everything except the middle one or two. Like the mean, it uses the actual sizes of the values it keeps, and like the median, it ignores the most extreme positions. For a skewed distribution it describes a different quantity from the ordinary mean, so it is not the mean with inconvenient values quietly removed. Choose the trimming rule before making the comparison, report it, and keep the full data on record.

Example For $2, 3, 4, 5, 16$ , trimming one observation from each end, 20 percent per tail, removes $2$ and $16$ and leaves $3, 4, 5$ , whose mean is $4$ . The ordinary mean is $6$ . For the eight delays, trimming one from each end, 12.5 percent per tail, removes $1$ and $20$ , and the remaining six delays average $27/6 = 4.5$ minutes, compared with the ordinary mean of $6$ minutes.

Check yourself

Nine trips on another route took $12$ , $14$ , $15$ , $15$ , $16$ , $18$ , $19$ , $21$ and $40$ minutes. With the median-of-halves rule, find the quartiles, the IQR and the fences. Which trips are flagged, and where do the whiskers of the modified box plot end?

Show the answer

With $n = 9$ the median is the fifth value, $16$ minutes, and it is left out of both halves. The lower half $12, 14, 15, 15$ gives $Q_1 = 14.5$ , and the upper half $18, 19, 21, 40$ gives $Q_3 = 20$ , so $\text{IQR} = 5.5$ minutes. The fences are $L = 14.5 - 8.25 = 6.25$ and $U = 20 + 8.25 = 28.25$ minutes. Only the trip of $40$ minutes lies outside, above $U$ . The whiskers end at $12$ and $21$ minutes, and the trip of $40$ minutes is drawn as a separate point, a value to check rather than to discard.

Compare the eight delays $1, 2, 3, 4, 5, 6, 7, 20$ with the nine values $1, 2, 3, 4, 4.5, 5, 6, 7, 20$ . Show that the two datasets have the same modified box plot. What do the box plots fail to reveal?

Show the answer

For the nine values the median is the fifth value, $4.5$ . The lower half $1, 2, 3, 4$ gives $Q_1 = 2.5$ and the upper half $5, 6, 7, 20$ gives $Q_3 = 6.5$ . With minimum $1$ and maximum $20$ , the five-number summary is $(1, 2.5, 4.5, 6.5, 20)$ , the same as for the eight delays. The IQR is $4$ in both cases, so the fences are $-3.5$ and $12.5$ , the whiskers end at $1$ and $7$ , and $20$ is flagged in both. The two box plots are identical, yet one dataset has $8$ values with mean $6$ and the other has $9$ values with mean $52.5/9 \approx 5.83$ . A box plot does not show the sample size or the mean, and it shows only a rough outline of the shape.

In the values $2, 3, 4, 5, 16$ , change $16$ to $26$ . Recompute the mean, the median, the raw MAD and the mean trimmed by one observation at each end. Which summaries resist the change?

Show the answer

The mean rises from $6$ to $40/5 = 8$ . The median stays $4$ . The distances from $4$ are now $2$ , $1$ , $0$ , $1$ and $22$ , which in order are $0, 1, 1, 2, 22$ , so the raw MAD stays $1$ . Trimming removes $2$ and $26$ and leaves $3, 4, 5$ , so the trimmed mean stays $4$ . The median, the MAD and the trimmed mean resist the change, while the mean moves, and so does the sample standard deviation, from about $5.70$ to about $10.12$ . The resistance of this trimmed mean covers only one extreme value per tail. If $5$ were also changed to $50$ , the trimmed mean would keep $26$ and become $(3 + 4 + 26)/3 = 11$ , while the median would still be $4$ .

Percentiles, Quartiles and Box Plots

Key terms

Percentile

Quartiles and interquartile range

Five-number summary and box plot

Outlier fence

Median absolute deviation

Trimmed mean

Check yourself

How this lesson connects

Builds on

Used later in