Unit 6 of 12
Hypothesis Tests and Two-Group Comparisons
- 11 lessons
- Videos 41 to 51
A hypothesis test asks whether data are compatible with a stated null hypothesis. This unit covers p-values, error rates and power, and tests for means and proportions along with rank-based tests.
A p-value is the probability, computed under the null model, of a test statistic at least as extreme as the one observed. It is not the probability that the null hypothesis is true, and statistical significance does not show that an effect is large or important, so effect sizes and intervals are read alongside it. Type I and type II errors, power and permutation tests come next, followed by t tests, proportion tests, paired and two-sample comparisons and rank-based tests, each with the conditions it needs.
Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.
The Logic of Hypothesis Tests
Hypothesis Tests and P-Values
How a hypothesis test compares data with a null model, what the null and alternative hypotheses, test statistic, null distribution and p-value are, and why the tail rule and significance level are fixed in advance.
Video 41
Type I Error, Type II Error and Power
What type I and type II errors are, how power at a stated effect is computed and planned for with a power analysis, what a minimum detectable effect means, and why optional stopping inflates the error rate.
Video 42
Statistical Significance and Practical Importance
Why statistical significance does not measure importance, how effect sizes and intervals are read against a smallest meaningful effect, what an equivalence margin is, and why absence of evidence is not evidence of no effect.
Video 43
Randomization and Permutation Tests
How a randomization test uses the actual random assignment under a sharp null hypothesis, when exchangeability makes a permutation test valid, and how restricted permutations and plus-one Monte Carlo p-values work.
Video 44
Tests for Means and Proportions
One-Sample t-Tests
How the one-sample t test compares a sample mean with a fixed benchmark using the t statistic and n minus 1 degrees of freedom, what the standardized mean difference adds, and how the test matches a confidence interval.
Video 45
One-Proportion Tests
How the exact binomial test and the one-proportion z test with the null standard error test a claimed success probability, why p-values from counts are discrete, and how two-sided exact p-values depend on a stated rule.
Video 46
Comparing Two Independent Means
How to estimate and test a difference between two population means from independent samples, using the standard error of a difference, Welch's t test, its degrees of freedom and interval, and the pooled t test.
Video 47
Paired t-Tests and Before-After Data
How to analyze paired data through within-pair differences, with the paired t test and paired confidence interval, and what a before-after design can and cannot show about cause.
Video 48
Comparing Two Proportions
How to estimate a difference in proportions in percentage points, test equality with the two-proportion z test and the pooled proportion, build the unpooled confidence interval and read a randomized A/B test.
Video 49
Rank-Based Tests
Sign Tests and Wilcoxon Signed-Rank Tests
How the sign test counts positive differences and the Wilcoxon signed-rank test sums signed ranks, what null each one tests, why the signed-rank test needs symmetry, and how zero differences and tied ranks are handled.
Video 50
Mann-Whitney Tests and Rank-Based Comparisons
How the Wilcoxon rank-sum test and the Mann-Whitney U statistic compare two independent samples through pooled ranks, what the exact permutation null assumes, and what the probability of superiority and a location shift mean.
Video 51