A change to a booking system shortens waiting times, and the result is highly statistically significant. Should it be adopted? That depends on how many minutes it saves, how precisely that saving is known, and how large a saving would justify the cost, and a p-value answers none of these by itself. This lesson separates evidence that an effect differs from zero from the question of whether its size matters, using effect estimates, confidence intervals and a threshold fixed before the data are seen.
Key terms
Statistical significance
Definition A result is statistically significant at level when the p-value of the planned test is at most , so that the test rejects . For a null of no effect, , based on an estimate that is approximately normal with standard error , the test statistic is
and the two-sided test at level matches the 95 percent interval , where is the quantile of the standard normal distribution. It rejects when lies outside that interval and fails to reject when lies strictly inside it.
In plain words Significance says that the data would be hard to reconcile with an effect of exactly zero under the model. Because the statistic divides the estimate by its standard error, a tiny effect measured very precisely can be highly significant, while a large effect measured imprecisely can fail to be. For any fixed nonzero true effect, a large enough sample makes a significant result very likely. The label therefore says nothing by itself about whether the effect is large or useful.
Example An intervention aims to cut waiting time, and the effect is the mean number of minutes saved per visit, so a negative value would mean longer waits. Three constructed studies report approximately normal estimates.
| Study | Estimate | SE | 95 percent interval | p-value |
|---|---|---|---|---|
| A | ||||
| B | ||||
| C |
Study A has and a two-sided p-value of about , so it is significant at any conventional level. Yet its whole interval lies below minutes, which is 18 seconds.
Effect size
Definition An effect size is a population quantity that answers the question by saying how large an effect is and in which direction, such as a difference in means or proportions. It can be stated in the original units or in standardized form. The effect estimate is the value computed from the sample, and it is reported with a standard error or a confidence interval. For a difference in means, a standardized effect size divides by a standard deviation of individual observations, and which standard deviation is used has to be stated.
In plain words The original units carry the practical meaning, here minutes saved. Standardizing helps compare an effect with the natural spread of the data, but a standardized effect changes whenever that spread changes, even if the minutes do not. Dividing by a standard deviation of individuals gives an effect size. Dividing by a standard error gives a test statistic, which tends to grow with the sample size when the true effect is not zero, so it is not an effect size.
Example Studies B and C have the same effect estimate, minutes saved. Their standard errors of and give and , so they differ in precision, not in estimated size. As a standardized example, a mean shift of units when individual observations have standard deviation is a standardized effect of .
Practical importance
Definition An effect has practical importance when its true size is large enough to matter for the decision or question at hand. That judgment rests on considerations outside the data analysis, such as costs, harms, benefits and who is affected. Statistics contributes the estimate and its uncertainty, which show which effect sizes are compatible with the data.
In plain words Statistical significance and practical importance answer different questions, and neither implies the other. A result can be significant and unimportant, important-looking but not significant, or significant and important at once. Design matters as well. An association estimated from an observational study is not automatically the causal benefit of an intervention, and a precise estimate of the wrong quantity does not help the decision.
Example Is the saving of about minutes per visit in study A important? The analysis cannot say by itself. Small savings could add up over a very large number of visits, but that claim would need its own calculation with a justified number of visits and a cost model, not a conclusion read off the p-value.
Smallest meaningful effect
Definition The smallest meaningful effect is the smallest true effect size that would matter for the decision, such as the smallest saving that would justify the cost of an intervention. It is fixed before the data are seen, from substantive reasoning, and it serves both to plan the study, through the power at that effect, and to read the result. With a threshold and a 95 percent interval from to for an approximately normal estimate, means that the one-sided test of the null against rejects at level , means that the one-sided test of the null against rejects at that level, and an interval containing leaves the question open.
In plain words Draw the threshold on the same axis as the interval. An interval wholly on the useful side supports a useful effect, one wholly on the other side supports an effect too small to matter, and one that straddles the threshold calls for more data or a cautious decision. Choosing the threshold after seeing the estimate would let the data set their own standard.
Example Suppose the decision makers state in advance that a saving of at least minute is needed. Study A’s interval lies entirely below , so it supports a positive saving that is too small for adoption. Study B’s interval contains , so it settles nothing about usefulness. Study C’s interval lies entirely above , so under the model it supports a saving large enough to be worthwhile. Ranking the studies by p-value puts A second, ahead of B, although A is the one study whose interval rules out a useful saving.
Equivalence margin
Definition An equivalence margin is a number , fixed in advance from substantive reasoning, such that true effects between and count as negligible. An equivalence question asks whether the true effect lies in that range, so the hypotheses are reversed,
A common procedure, two one-sided tests, rejects this at level when the one-sided test of the null and the one-sided test of the null both reject at level . For an approximately normal estimate this happens exactly when the percent confidence interval, the 90 percent interval when , lies within .
In plain words To call two options equivalent, the data must rule out every difference as large as the margin in either direction, and that takes a precise estimate. A margin is two-sided, a band of negligible differences, while a smallest meaningful effect is usually a one-sided bar for adoption. Widening the margin after seeing a wide interval would let almost any result be called equivalent.
Example Take minute. Study A’s 90 percent interval is , about , which lies within . So under the model study A shows, at level , that the effect is within one minute of zero, even though it also differs significantly from zero. Study B’s 90 percent interval, about , extends past , so it does not show equivalence, although it is not significant either.
Absence of evidence
Definition Absence of evidence describes a result that fails to reject a null hypothesis of no effect. Such a result does not show that the effect is zero or negligible. No value strictly inside the 95 percent interval is rejected by the matching two-sided test at level , so an interval that reaches important values leaves them unresolved. A claim that an effect is absent or negligible needs an analysis that rules out effects at least as large as a margin chosen in advance, as an equivalence test does.
In plain words Absence of evidence is not evidence of absence. A nonsignificant result from a small or noisy study often means only that the study could not tell. Look at how wide the interval is and, at the planning stage, at the power against the smallest meaningful effect.
Example Study B has a p-value of about , so it does not reject at level . Its interval from to minutes contains zero, a small increase in waiting time and savings of almost seven minutes. Reporting that the intervention has no effect would be wrong. The fair summary is that the study is too imprecise to say whether the saving reaches one minute.
Check yourself
- A constructed study estimates a saving of minutes with 95 percent interval , and the smallest saving that would justify adoption, fixed in advance, is minute. Write a conclusion that does not treat the word significant as meaning important.
Show the answer
The interval excludes , so under the model the data indicate a positive saving, and the two-sided test of no effect rejects at level . The whole interval also lies below , so the data indicate that the saving is smaller than the adoption threshold. A balanced statement is that the estimated saving is 0.4 minutes, with a 95 percent interval from 0.2 to 0.6 minutes, which supports a small positive saving but not a saving of one minute or more, so the result does not meet the adoption rule.
- A report gives a 95 percent interval from to minutes for the saving, notes that the p-value exceeds , and concludes that the intervention has no effect. What is wrong with the conclusion, and what should the report say?
Show the answer
Failing to reject zero is absence of evidence, not evidence that the effect is zero. The interval contains , but it also contains a 2-minute increase in waiting time and a 3-minute saving, both of which would matter. It extends well beyond and , and so does the 90 percent interval that two one-sided tests at level would use, about if the estimate is approximately normal, so the data cannot show equivalence within a one-minute margin either. The report should say that the study is too imprecise to determine whether the intervention helps, harms or makes no practical difference.
- A cheaper system will replace the current one if the two are equivalent within a margin of minute, fixed in advance. Study D estimates the difference in mean waiting time as minutes with standard error , and study E estimates minutes with standard error , both approximately normal. Neither differs significantly from zero. Using two one-sided tests at , which study, if either, shows equivalence?
Show the answer
Compare each 90 percent interval with . For study D the interval is , about , which lies inside, so study D shows equivalence at level . For study E it is , about , which extends past both margins, so study E does not. The ordinary tests of zero give two-sided p-values of about for D and for E. The larger p-value belongs to the less informative study, which shows why a nonsignificant p-value cannot answer an equivalence question.