Unit 12, lesson 77

Selecting Methods and Reporting Statistical Results

How to build a statistical report from its estimand, an analysis plan and preregistration, keep planned and exploratory analyses apart, avoid selective reporting, and support it with sensitivity analysis and reproducibility.

  • Video 77
  • From Question to Report

A report on two learning interfaces concludes that the new one worked because the p-value was below 0.05. Worked in what sense, by how many minutes or correct answers, for whom, and with how much uncertainty? The threshold answers none of these questions, and the report does not say whether other outcomes were tried before this one was chosen. This lesson builds a report from the quantity the study set out to learn about and a plan written before the results, and it shows how to state an effect with its uncertainty, disclose every comparison, test what the conclusion depends on and let others recheck the numbers.

Key terms

Estimand

Definition The estimand is the quantity a study aims to learn about, specified completely enough to be one definite, if unknown, number. The specification names the population or process, the outcome and how it is measured, any conditions or groups being compared, and the summary, such as a difference of means, a difference of proportions or a mean within-person change. An estimator is the rule applied to the sample, and the estimate is the number that rule produces from the data. A method is then selected to fit the estimand together with the type of outcome, the way units were sampled or assigned, any dependence among the observations and the assumptions the method needs.

In plain words Before choosing a test, write down the number you would want if you could measure every relevant person under every relevant condition. One broad question can lead to several estimands, and they can need different data and different analyses. Starting from a familiar test lets the calculation decide the question instead.

Example In a constructed teaching study, 24 volunteers are randomly assigned, 12 to interface A and 12 to interface B, and each completes one task. Time is measured in minutes from the start of the task to the final submission, whether the submitted answer is correct or not. The primary estimand is

Δ=μB−μA,\Delta = \mu_B - \mu_A ,

where μA\mu_A and μB\mu_B are the mean times, in minutes, that comparable volunteers would have with each interface under the same task conditions. A negative value favors B. Random assignment supports a causal reading of this difference within the study setting, under assumptions such as no interference between participants, but volunteers are not a random sample of all learners. The two groups contain different people, so an independent-groups method such as the Welch interval fits. If the same people had used both interfaces, the estimand would be a mean within-person difference with a paired analysis, and if the question were about correct answers, it would be the difference pB−pAp_B - p_A of success probabilities.

Analysis plan and preregistration

Definition An analysis plan is a written record, made before the outcome data are examined, of the decisions that shape an analysis. It names the eligible units and the design, defines each outcome, identifies the primary outcome with its estimand, gives the method with its interval or test, says how missing data, exclusions and secondary outcomes will be handled, and states any practical threshold. Preregistration places a dated version of the plan in a registry or another independent record before the relevant outcome data are examined, so that later edits cannot silently replace it. A plan can still be revised for a defensible reason, such as an instrument error found during the study, provided the original decision, the change, its reason and its consequences for interpretation all stay on record.

In plain words A plan fixes the choices while nobody can yet know which of them favor a particular answer, and preregistration lets others check when those choices were made. Neither one shows that the design or the method is sound. A plan written after the results are known describes the analysis but no longer protects it from being shaped by them.

Example A plan for the interface study would name the eligible volunteers and the random assignment, define time as above and correctness as a separate outcome recorded as 1 or 0, and declare Δ\Delta the primary contrast with a two-sided 95 percent Welch interval. It would say what happens to missing records and that correctness is reported as an exploratory secondary outcome. It would also state, before any estimate exists, that a mean reduction of at least 2 minutes is the smallest benefit worth acting on. The study is constructed for teaching, so this describes what a plan would contain, not an actual registration.

Planned versus exploratory analysis

Definition An analysis is planned, or prespecified, when its outcome, comparison and method were fixed before the outcome data were examined, and it is confirmatory when the plan also makes it the basis of a formal claim with a stated decision rule. It is exploratory when it was chosen after looking at the data, or picked out of many possible analyses because of what they showed. A plan can also label a prespecified analysis exploratory, as with a secondary outcome reported without a formal claim. The distinction concerns when and why an analysis was chosen, not which formula it uses. When an analysis is chosen by looking at the same data it is applied to, its nominal p-value or confidence level describes one prespecified analysis, not the search that actually produced the result.

In plain words Exploration finds unexpected patterns and new questions, and it is worth doing. The problem is presenting the winner of a search as if it were the single prediction made in advance. Label exploratory results as exploratory and look for confirmation in data collected for that purpose.

Example The constructed times, in minutes, are 14, 16, 15, 18, 17, 13, 19, 16, 15, 20, 14 and 17 with A and 12, 13, 14, 11, 15, 12, 16, 13, 10, 14, 12 and 11 with B. The planned primary analysis gives sample means of about 16.17 minutes for A and 12.75 minutes for B, so the estimated difference in minutes is

Δ^=xˉB−xˉA=−4112≈−3.42.\begin{aligned} \hat{\Delta} &= \bar{x}_B - \bar{x}_A \\[0.4em] &= -\tfrac{41}{12} \approx -3.42 . \end{aligned}

The 95 percent Welch interval runs from about −5.07-5.07 to −1.76-1.76 minutes. Stated as a reduction, A minus B, the estimate is 3.423.42 minutes with an interval from 1.761.76 to 5.075.07, because reversing the subtraction flips both signs and swaps the endpoints. The 95 percent is the approximate rate at which the procedure captures Δ\Delta over repeated samples when its assumptions hold, not a probability that this particular interval contains Δ\Delta. The interval lies below zero but includes reductions smaller than the planned 2 minutes, so under the working assumptions it supports a lower mean time with B without establishing the benefit declared worthwhile. Correctness, the exploratory secondary outcome, was 8 of 12 for A and 10 of 12 for B, a difference of 16\tfrac{1}{6}, about 16.7 percentage points, and it is reported with that label.

Selective reporting

Definition Selective reporting is presenting only some of the outcomes, comparisons, subgroups, exclusion rules or methods that were examined, chosen in light of their results, without disclosing the rest. It hides a problem of multiplicity. If mm tests of true null hypotheses are independent and each rejects with probability exactly α\alpha, the probability of at least one false rejection is

1−(1−α)m,1 - (1 - \alpha)^m ,

which, for 0<α<10 < \alpha < 1, increases toward 1 as mm increases. Familywise error control bounds the probability of any false rejection in a family of tests. The Bonferroni rule tests each of mm hypotheses at level α/m\alpha/m, and when each test is valid at its level it keeps that probability at most α\alpha whatever the dependence among the tests. False discovery rate control bounds the expected proportion of false rejections among all rejections, counted as zero when nothing is rejected, which is a different goal.

In plain words When enough comparisons are tried, chance alone is likely to produce some impressive-looking results, and a reader who sees only the best one cannot tell how surprising it is. Disclosing every comparison lets the reader judge the search, and an adjustment for multiple testing accounts for its size. No adjustment repairs biased sampling or poor measurement, and an exploratory label does not make a broad claim reliable.

Example Suppose ten independent tests are run at α=0.05\alpha = 0.05, every null hypothesis is true, and each test rejects with probability exactly 0.05. The chance of at least one significant result is

1−0.9510≈0.401,1 - 0.95^{10} \approx 0.401 ,

so a single significant result, reported without the other nine, would overstate the evidence. Under the Bonferroni rule each test would use level 0.05/10=0.0050.05/10 = 0.005. A transparent report of the interface study lists time and correctness with their roles, together with any other outcome, subgroup or exclusion rule that was examined, even when its result was unremarkable.

Sensitivity analysis

Definition A sensitivity analysis repeats an analysis under other reasonable choices or assumptions and reports how the results change. The choices can include another defensible method, a different treatment of missing data, other inclusion rules, the removal of single observations, or plausible values for quantities that were not observed. For missing data, a report states how many values are missing in each group and variable, the known reasons and the assumption behind the chosen handling. Analyzing only the complete records can be biased when whether a value is missing is related to the outcome or to other variables that affect it, and imputation also rests on assumptions.

In plain words A sensitivity analysis asks which assumptions the conclusion rests on and how far they would have to move before the conclusion changes. It is a disclosure, not a way to pick the most favorable version, and its scenarios are not probabilities.

Example The constructed data are complete. Removing one record at a time and recomputing the estimated difference gives values from about −3.71-3.71 to −3.07-3.07 minutes, so no single record reverses the direction. That is a check of influence, not a reason to delete a valid record. Now imagine, as a separate scenario, that the last B time of 11 minutes had been lost. The other eleven B times total 142 minutes and the A times total 194, so with the missing value equal to uu minutes the full-data estimate would be

Δ^(u)=142+u12−19412=u−5212.\begin{aligned} \hat{\Delta}(u) &= \frac{142 + u}{12} - \frac{194}{12} \\[0.4em] &= \frac{u - 52}{12} . \end{aligned}

A value of 30 gives about −1.83-1.83 minutes, and a value of 60 gives about 0.670.67, which reverses the sign. These are conditional scenarios, and a real study would justify a plausible range from its context. An interval computed from the observed records alone does not include this kind of uncertainty.

Reproducibility

Definition An analysis is reproducible when the same data and the documented steps regenerate every reported number. In practice that calls for the data dictionary, the original records, every cleaning and exclusion decision, the code or exact calculation steps, the software versions and any random seeds used in simulation. Replication is a different matter. A finding is replicated when a new study with new data addresses the same question and reaches a consistent result. A small p-value guarantees neither, and a fully reproducible analysis can still rest on a poor design or an unsuitable model.

In plain words Another analyst should be able to start from your records, obtain your numbers and see why each decision was made. Keep exact values for calculation and round only for display. Label every uncertainty band in a figure, because a standard deviation, a confidence interval and a prediction interval answer different questions.

Example The upper endpoint −1.76-1.76 comes from unrounded inputs, the estimate −4112-\tfrac{41}{12}, a standard error of about 0.79730.7973 minutes and a t quantile of about 2.07792.0779 for 21.2821.28 Welch degrees of freedom. Rounding the inputs to −3.4-3.4, 0.80.8 and 2.082.08 before combining them gives −3.4+2.08×0.8=−1.736-3.4 + 2.08 \times 0.8 = -1.736, which would be reported as −1.74-1.74. Put together, a report of the constructed study could read as follows. In a randomized comparison of 24 volunteers, 12 per interface with one task each, mean time to final submission, correct or not, was 16.17 minutes with A and 12.75 minutes with B. The estimated difference, B minus A, was −3.42-3.42 minutes, with a 95 percent Welch confidence interval from −5.07-5.07 to −1.76-1.76 and a two-sided p-value of about 0.00032 for equal means, under independent-group assumptions. The interval includes reductions smaller than the planned 2 minutes. Correctness, 8 of 12 with A and 10 of 12 with B, was an exploratory secondary outcome, and no records were missing. The volunteers, the single task and the small groups limit how far the result extends, the data were constructed for teaching, and the data and calculation steps accompany the report.

Check yourself

  1. A summary of the constructed study says that interface B improved accuracy by 16.7 percent. With 8 of 12 correct for A and 10 of 12 for B, what is wrong with that wording, and how should the result be stated?
Show the answer

The success proportions are 812≈0.667\tfrac{8}{12} \approx 0.667 and 1012≈0.833\tfrac{10}{12} \approx 0.833. Their difference, 16≈0.167\tfrac{1}{6} \approx 0.167, is an absolute difference of 16.7 percentage points. A relative increase compares the change with A’s proportion, and here it is 10/128/12−1=14\tfrac{10/12}{8/12} - 1 = \tfrac{1}{4}, or 25 percent, so the phrase “16.7 percent” mislabels the absolute difference. A clear statement gives the denominators and the role of the outcome, for example that 10 of 12 people (83.3 percent) answered correctly with B and 8 of 12 (66.7 percent) with A, a difference of 16.7 percentage points on an exploratory secondary outcome. With 12 people per group this difference is very imprecise, so the word “improved” also claims more than the data establish.

  1. Suppose instead that the A time of 20 minutes had been lost, while every B time was recorded. Write the full-data estimate of the difference as a function of the unknown time vv in minutes. Which values of vv would make the estimated reduction at least 2 minutes, and could any possible value reverse the sign of the estimate?
Show the answer

The remaining A times total 194−20=174194 - 20 = 174 minutes and the B times total 153, so

Δ^(v)=15312−174+v12=−21+v12.\begin{aligned} \hat{\Delta}(v) &= \frac{153}{12} - \frac{174 + v}{12} \\[0.4em] &= -\frac{21 + v}{12} . \end{aligned}

The estimated reduction 21+v12\tfrac{21 + v}{12} is at least 2 exactly when 21+v≥2421 + v \ge 24, that is when v≥3v \ge 3. A time cannot be negative, so every possible value gives Δ^(v)≤−2112=−1.75\hat{\Delta}(v) \le -\tfrac{21}{12} = -1.75, and no value reverses the sign of the point estimate. This settles only the point estimate. The interval also depends on vv, and the scenarios say nothing about how likely each value is.

  1. A short report reads as follows. Learners who chose the new interface scored significantly higher on the final quiz (p = 0.04), which proves that the interface improves learning. The team had compared the two groups on eight outcomes before writing it, and the report gives no group sizes. What problems should an audit flag, and what would a corrected report contain?
Show the answer

The learners chose their interface, so this is an observational comparison. Differences in prior skill or motivation could explain the gap, and the word “proves” claims a causal effect the design cannot support. The quiz was one of eight outcomes and the other seven are not disclosed. If eight independent tests of true null hypotheses each reject with probability 0.05, the chance of at least one significant result is 1−0.958≈0.3371 - 0.95^8 \approx 0.337, so one significant outcome picked from eight is weak evidence of a real difference. There are no denominators, no mean scores and no estimate of the difference in quiz points with an interval, so the reader cannot judge the size or precision of the effect, and the p-value is not the probability that the interface has no effect. A corrected report gives the number of learners in each group and how they came to use each interface, the mean scores in quiz points, the estimated difference with its confidence interval, which outcome was planned as primary, all eight outcomes with their results, and association wording with the limits of an observational study.

How this lesson connects