Comparing Two Proportions

Definition Suppose group A has $n_A$ units and group B has $n_B$ units, each unit gives one binary outcome, success or failure, defined before the data are collected, and $x_A$ and $x_B$ are the numbers of successes. The sample proportions are $\hat{p}_A = x_A/n_A$ and $\hat{p}_B = x_B/n_B$, and the difference in proportions $p_A - p_B$ is estimated by $\hat{p}_A - \hat{p}_B$. On the probability scale this absolute difference is also called the risk difference, and it is measured in percentage points. If the outcomes are independent Bernoulli trials within each group, with success probabilities $p_A$ and $p_B$, and the two groups are independent, then

When $p_B > 0$, the relative difference $(p_A - p_B)/p_B$ and the risk ratio $p_A/p_B$ describe the same comparison on a relative scale.

Definition Under the null hypothesis $H_0: p_A = p_B$ both groups share one success probability $p$. Its estimate, the pooled proportion, is

the proportion of successes in the two groups combined. It is the weighted average of $\hat{p}_A$ and $\hat{p}_B$ with weights $n_A$ and $n_B$. Putting it in place of both probabilities in the variance formula gives the estimated null standard error

Pooling belongs to tests of equality. A null hypothesis of a nonzero difference, $p_A - p_B = \delta_0$ with $\delta_0 \ne 0$, does not give the groups a common probability, so the pooled proportion does not fit it.

Definition Suppose the outcomes are independent Bernoulli trials with probability $p_A$ in group A and $p_B$ in group B, the two groups are independent, and the expected counts under the null hypothesis, $n_A\hat{p}_{\text{pool}}$, $n_A(1 - \hat{p}_{\text{pool}})$, $n_B\hat{p}_{\text{pool}}$ and $n_B(1 - \hat{p}_{\text{pool}})$, are all reasonably large, with at least 5 or at least 10 each as common guidelines. The two-proportion z test of $H_0: p_A = p_B$ uses

which has approximately the standard normal distribution when $H_0$ is true. For $H_1: p_A \ne p_B$ the p-value is $P(\lvert Z \rvert \ge \lvert z \rvert)$ with $Z \sim N(0, 1)$, and a one-sided alternative uses the single tail in its direction. The guidelines are rules of thumb, not guarantees of accuracy, and for small or sparse counts an exact procedure such as Fisher’s exact test is often preferred.

Definition Without assuming $p_A = p_B$, each group’s variance is estimated from its own proportion,

and the approximate 95 percent confidence interval for $p_A - p_B$ is

where $z^{*} \approx 1.960$ is the $0.975$ quantile of the standard normal distribution. It needs independent Bernoulli outcomes within each group, independent groups and enough observed successes and failures in each group, with at least 10 of each as a common guideline. Its actual coverage can fall well below 95 percent with small counts or with proportions near 0 or 1, and score-based intervals, such as Newcombe’s interval built from a Wilson interval for each group, usually perform better there.

Definition An A/B test is a randomized experiment that compares two versions of something, such as a web page, a message or an interface, by randomly assigning each unit, usually a user, to version A or version B and recording a primary outcome chosen in advance, often a binary one such as a purchase. Random assignment supports a causal comparison of the two versions for the units in the experiment, under its conditions, provided each unit’s outcome depends only on its own version, units receive the version they are assigned and outcomes are recorded for all assigned units. A sound plan fixes the outcome, the measurement window, the sample size and the analysis before the data arrive.