Randomized Experiments, Blocking and Matched Pairs

Definition In an experiment the investigator imposes treatments, the conditions whose effects are being compared, and then measures a response. The experimental unit is the entity to which a treatment is assigned, the smallest division of the material such that any two units could receive different treatments. A person, a classroom, a plot of land or a device can be the unit, depending on what the design assigns. Replication means applying each treatment to more than one experimental unit, and the number of replicates counts experimental units. Repeated measurements of one unit, and measurements of smaller parts inside it such as the students of an assigned classroom, share a single assignment. Analyzing them as if they were independent replicates is called pseudoreplication, and it tends to make the uncertainty look smaller than it is.

Definition Random assignment allocates the treatments to the experimental units by a chance mechanism with known probabilities, such as a random number generator, fixed before any response is seen. Neither the investigator nor the participants choose who receives which treatment. In a completely randomized design with $n$ units, of which $n_1$ receive the treatment and $n_0 = n - n_1$ the comparison condition, where $1 \le n_1 \le n - 1$, each of the $\binom{n}{n_1}$ possible treatment groups has the same probability. Every unit then has the same probability $n_1/n$ of receiving the treatment, whatever its characteristics, and over the possible assignments the mean of any pre-treatment variable in either group has expected value equal to its mean over all $n$ units.

For unit $i$, let $Y_i(1)$ and $Y_i(0)$ be its potential responses under the treatment and under the comparison condition, of which at most one is observed. The average causal effect for the units in the experiment is

Suppose the design is completely randomized, each unit’s response depends only on its own assignment, every unit receives the condition it is assigned and every response is recorded. Let $\bar{y}_1$ and $\bar{y}_0$ be the mean observed responses in the treatment group and in the comparison group. Then, with the expectation taken over the random assignment,

so the difference of the group means is an unbiased estimate of the average causal effect for these units.

Definition A control group is a group of experimental units that receives a comparison condition instead of the treatment under study, such as no treatment, the current standard or a placebo. It should be handled like the treatment group in every respect except the intended difference, with the same tasks, instructions, timing and measurement. A placebo is a dummy condition made to resemble the treatment as participants experience it while lacking the component whose effect is being studied. A response caused by receiving an apparent treatment, through expectation or attention, is called the placebo effect, and a placebo control lets the comparison separate the active component from that effect. Some questions call instead for an active control, an existing treatment or a condition that gives similar attention and practice. Not every experiment needs a placebo, and the question being asked decides which comparison is appropriate.

Definition Blinding, also called masking, means keeping people involved in an experiment from knowing which treatment each unit received, to prevent their expectations from influencing behavior, the delivery of treatments or the measurement of the response. The participants, the staff who deliver treatments or interact with participants, the people who assess outcomes and the analysts can each be blinded or not. Labels such as single-blind and double-blind are used in different ways, so a clear report states exactly who was blinded.

Definition A block is a group of experimental units that are alike in a characteristic, recorded before treatment, that is expected to affect the response. In a randomized block design the treatments are randomly assigned separately within each block, the treatments are compared within blocks, and the analysis keeps track of the blocks. Comparing within blocks keeps the differences between blocks out of the treatment comparison, which can make the estimate more precise when the blocking variable is related to the response. Blocking does not replace randomization, which still balances the characteristics not used to form the blocks, on average over the possible assignments.

A matched pairs design is a block design with blocks of size two. Either two similar units are matched, with one of them chosen at random for the first treatment and the other given the second, or a single subject or object receives both treatments in a randomly chosen order, and its two measurements form the pair. In the second form the design has to deal with order effects and with carryover from the first condition into the second, for example by leaving enough time between them. The analysis uses the within-pair differences $d_i$, one per pair and always taken in the same direction, and their mean $\bar{d}$.

Definition Random sampling uses chance to decide which units of a population enter a study. Random assignment uses chance to decide which treatment each unit in the study receives. A probability sample with adequate response supports generalizing from the sample to the population that was sampled. Random assignment supports a causal comparison of the treatments for the units in the experiment, under the conditions of the experiment. A study may use both, one or neither, and neither feature supplies the conclusion that belongs to the other.

A conclusion that goes beyond its cell needs other support, such as explicit assumptions for a causal claim without random assignment, or an argument that the results carry over to units that were not sampled.