Does a new learning interface help people finish a task faster, and would that be enough reason to adopt it? No single test answers this. The question fixes what to measure, the design fixes what the data can show, and the analysis, its checks and the final sentences all have to stay consistent with both. This lesson follows one constructed comparison of two interfaces from the question to a conclusion that another analyst could reproduce and check.
Key terms
Statistical investigation and research question
Definition A statistical investigation is the connected process that turns a question into a supported conclusion. It runs through the research question, the design of data collection, measurement, exploratory data analysis, the choice of method, estimation with uncertainty, model checking and a report. A research question is a question stated precisely enough to be answered with data. It names the population or process of interest, the outcome and how it is measured, any conditions or groups compared, and the quantity, or estimand, whose value would answer it.
In plain words Every later step has to fit the question and the way the data were obtained. Asking whether an interface is better leaves open better at what, for whom and by how much, and those answers decide which data and which analysis make sense.
Example The broad question is whether interface B shortens the time people need to submit a task compared with interface A. Made precise, the population target is comparable volunteers doing this task under the study conditions, which is narrower than all learners doing every kind of task. The primary estimand is
the mean time to final submission with B minus the mean time with A, in minutes, so a negative value favors B. Correctness is recorded as well, because a faster submission is not automatically a better one. All data in this lesson are constructed for teaching and describe no real product.
Study design
Definition The study design is the plan for obtaining the data. It specifies how units enter the study, for example by random sampling, by volunteering or by convenience, and how units come to be in each condition, by random assignment, by their own choice or by circumstances the investigator only observes. It also fixes the experimental or observational unit, whether each unit is measured once or under several conditions, and how each variable is measured. Random assignment supports causal comparisons among the units studied, under assumptions such as no interference between units and consistent delivery of each condition. Random sampling supports generalization to the sampled population. Neither one provides what the other provides.
In plain words The design decides what the data can and cannot say before any number is computed. Random assignment balances other causes on average over the possible allocations, though any one small allocation can be noticeably unbalanced, while recruitment decides whom the results describe.
Example In the constructed study, 24 volunteers are randomly assigned, 12 to each interface, and each completes one task. The person is the experimental unit, and the two groups contain different people. Time runs from the start of the task to the final submission, whether that answer is correct or not, and correctness is recorded separately. Timing only the correct submissions would condition the primary outcome on something the interface itself might affect.
| Interface | Times in minutes | Correct |
|---|---|---|
| A | 14, 16, 15, 18, 17, 13, 19, 16, 15, 20, 14, 17 | 8 of 12 |
| B | 12, 13, 14, 11, 15, 12, 16, 13, 10, 14, 12, 11 | 10 of 12 |
Everyone submitted, so no time is missing. If some people had never submitted, recording a time of zero or dropping them would misstate the outcome, and the analysis would need a stated follow-up rule and methods for censored times.
Primary and secondary outcomes
Definition The primary outcome is the outcome, together with its estimand and analysis, that is named in advance as the main basis for the study’s conclusion. Secondary outcomes are further outcomes named in advance and reported with their role stated. Unless the plan made them confirmatory, with a plan for multiplicity, results on secondary outcomes are supporting or exploratory evidence. An outcome chosen after seeing the data is exploratory, whatever it is called.
In plain words Naming the primary outcome in advance guards against promoting, after the fact, whichever outcome looks best. Secondary outcomes are still reported in full, but each claim keeps the role it was given.
Example Here the plan, written before any outcome was examined, makes time the primary outcome, with a mean reduction of 2 minutes as the smallest benefit worth acting on, and correctness an exploratory secondary outcome. Correctness was 8 of 12 for A and 10 of 12 for B, so the sample proportions give
about 16.7 percentage points, an absolute difference rather than a relative improvement of 16.7 percent. With only 4 incorrect answers for A and 2 for B, an exact comparison suits. Fisher’s exact test of equal success probabilities, adding the null probabilities of all tables with the same margins that are no more likely than the observed one, gives a two-sided p-value of about 0.64. That does not show equal accuracy. Newcombe’s 95 percent interval for runs from about to , so differences that would matter in either direction remain compatible with the data. The result is an imprecise exploratory estimate, not evidence that B costs nothing in accuracy.
Exploratory data analysis
Definition Exploratory data analysis examines the data with plots and summaries to understand their distributions, find recording errors, unusual values and missing entries, and judge whether the assumptions of a planned method are plausible. It begins with the records themselves, checking identifiers, one record per unit, group labels, units of measurement and impossible values. Patterns it reveals are new questions and are reported as exploratory. It cannot verify independence or random assignment, which come from the design.
In plain words Look at every observation before reducing the data to one number. An unusual value is a reason to check where it came from, not a reason to delete it automatically. Exploring openly is useful, while quietly searching for the most favorable outcome or subgroup is not.
Example Dot plots of the times on one shared axis show A from 13 to 20 minutes and B from 10 to 16 minutes. The medians are 16 and 12.5 minutes and the means are about 16.17 and 12.75 minutes. B is generally lower, but the groups overlap, and four of the B times exceed the fastest A time of 13 minutes, so a lower mean does not mean every B user was faster than every A user. Each mean lies close to its median and no isolated extreme value appears. Twelve observations per group cannot show that a population is normal, though, and no plot can confirm that the observations are independent.
Method selection and model checking
Definition Method selection chooses an analysis that estimates the declared estimand under the actual design, given the type of outcome, the dependence among observations and the assumptions the method needs. It does not choose whichever analysis gives the smallest p-value. Model checking asks whether those assumptions are reasonable for the data at hand and how much the conclusions depend on them, using plots of the data or residuals, influence checks, comparisons with other defensible methods and knowledge of how the data were collected.
In plain words The target, the outcome and the design point to the method, and the checks ask whether the method’s picture of the data is good enough to trust its answer. Passing a check does not prove an assumption true, and a check aimed at one concern does not settle the others.
Example A numerical outcome, a difference of means and two groups of different people point to Welch’s two-sample method, which allows unequal variances. A paired method would be wrong because no one used both interfaces, and a rank test addresses a different target unless further assumptions tie it to means. The method treats each group’s times as independent draws from the possible times under that interface, a working model that goes beyond the random allocation itself, and with 12 per group it also needs distributions close enough to normal for the t approximation. The sample variances are about and square minutes, so the standard error of the difference, in minutes, is
With about Welch degrees of freedom the t quantile is about , and the 95 percent interval for is
The t statistic for equal means is about with a two-sided p-value of about 0.00032. As a check of influence, removing one record at a time gives estimates from about to minutes, so no single record decides the direction. The dot plots showed no extreme values, which is consistent with the t approximation but does not prove it.
Reproducible report
Definition A reproducible report states the research question, the design and its scope, the group counts, the role of each outcome, the effect estimate in its units with an interval, the method and assumptions behind it, the checks performed and the limitations. It comes with the data dictionary, the records, the decisions and the calculation steps, so that another analyst can regenerate every number and see which assumptions would change the interpretation. Reproducing the numbers from the same data is different from replicating the finding in a new study.
In plain words The conclusion should claim what the design and the analysis support, no more and no less, and the material behind it should let anyone check that.
Example For the constructed study the report says that 12 randomly assigned volunteers used each interface for one task, and that mean time to final submission was 16.17 minutes with A and 12.75 minutes with B. The estimated difference, B minus A, is minutes with a 95 percent Welch interval from to . The interval excludes zero, so under the working assumptions the data indicate a lower mean time with B, but it includes reductions below the 2 minutes the plan called worthwhile, so a reduction of that size is not established. Correctness, 8 of 12 against 10 of 12, is an imprecise exploratory result. The volunteers, the single task, the small groups and the constructed data limit the conclusion. The data file with its dictionary, the plan, the calculation steps with unrounded values, the plots and the influence check accompany the report. A decision to adopt B would also weigh accuracy, cost and accessibility, which the time analysis does not measure.
Check yourself
- Suppose the same 24 times had come from learners who chose their own interface. What changes in the analysis and in the conclusion?
Show the answer
The Welch arithmetic would give the same numbers, but they would estimate something different. The design is now observational, so the difference compares people who chose B with people who chose A, and only for the kind of learners who appear in the data. Those groups may differ in prior experience, motivation or other causes of task time, and such confounding is not removed by a small p-value. The supported conclusion is an association, for example that learners who chose B submitted faster on average by an estimated 3.42 minutes, which may partly or wholly reflect who chose each interface. A causal claim would need further design features or assumptions, such as adjustment for measured confounders together with the assumption that no important confounder is unmeasured.
- Now six volunteers each use both interfaces, in randomized order and with enough separation between tasks that carryover can be assumed negligible. Their B minus A differences in minutes are , , , , and . Using , find the estimated mean difference and a 95 percent interval, and state a conclusion with its limits.
Show the answer
Each person’s difference is one observation, so . The differences sum to , and their sample variance with the denominator is , so
The paired interval is
In this constructed paired study, the estimated mean reduction with B is 2.17 minutes, with a 95 percent interval from 0.94 to 3.39 minutes. The interval excludes zero but includes reductions below 2 minutes, so a reduction of at least 2 minutes is not established. The interval relies on independent differences across people, an approximately normal distribution of differences, and order and carryover effects handled by the design, and six volunteers say little about learners in general.
- An analyst notices that both groups of the original study have 12 people and proposes pairing the A and B records row by row and running a paired t test. Why is this wrong?
Show the answer
No person or matched unit links any A record to any B record, so the pairs would be arbitrary. The mean of the 12 row differences equals in every order, but their standard deviation depends on the order. In the listed order it is about 2.07 minutes, and if both groups are sorted from fastest to slowest it is about 0.51 minutes, so the standard error and the interval would reflect an arbitrary choice rather than the design. A paired analysis needs pairs created by the design, such as the same person under both conditions or deliberately matched people. For independent groups the Welch analysis matches the design.