A learning task can be changed in two ways, by rewriting its instructions and by offering hints. Testing each change in a separate experiment cannot reveal whether hints help more with one kind of instructions than with the other. A factorial design tries every combination, so one experiment shows both how much each change matters on average and whether the effect of one change depends on the other. This lesson builds those effects from cell means, explains why the design needs replicated experimental units, and shows how blocking keeps a known source of variation out of the treatment comparisons.
Key terms
Factor and factorial design
Definition A factor is an explanatory variable that an experiment varies deliberately, or uses to define its groups, and its levels are the particular values chosen for it. A treatment, or treatment combination, fixes one level of every factor. A full factorial design uses every combination of the levels, so a factor A with levels and a factor B with levels give treatment combinations, and when the experimenter sets both factors, the experimental units are randomly assigned to them. The design is balanced when every combination receives the same number of units.
In plain words List the settings of each factor and try all of their combinations. Every unit then contributes information about both factors at once, and the effect of one factor can be compared across the levels of the other.
Example Factor A is the instruction format, with level for the current text and for a new worked-example format. Factor B is hints, with for no hints and for hints. The full design has the four combinations , , and . In the constructed experiment, eight learners are randomly assigned, two to each combination, so the design is balanced.
Cell mean and replication
Definition A cell is one treatment combination, and its cell mean is the mean response of the units that received it. Write for the population cell mean at level of A and level of B, estimated by the sample cell mean . Replication means assigning each treatment combination to more than one experimental unit, the entity to which a treatment is assigned. Replicates count experimental units, so repeated measurements of one unit, such as several questions answered by one learner, are not replicates. In the ordinary model for an design that includes the interaction, the residual sum of squares adds the squared deviations of the observations from their cell means and has degrees of freedom, where is the number of experimental units. With a single unit in every cell this number is , and the error variance cannot be estimated in this way.
In plain words The cell means are the summaries every effect is built from. Replication shows how much units treated alike differ from each other, and that is the yardstick for judging whether differences between cell means are larger than noise.
Example The constructed scores are below, with the cell mean after each pair.
| Format | No hints | Hints |
|---|---|---|
| Current | 9 and 11, mean 10 | 11 and 13, mean 12 |
| New | 13 and 15, mean 14 | 19 and 21, mean 20 |
In every cell the two learners score one point below and one point above the cell mean, so each cell contributes and the residual sum of squares is . Eight observations and four fitted cell means leave residual degrees of freedom, so the residual mean square is , the pooled within-cell variance. If each learner had also answered ten questions, the experiment would still have two replicates per cell, not twenty.
Simple effect
Definition A simple effect of a factor compares its levels at one fixed level of the other factor. In a design the simple effects of B are
and the simple effects of A are at and at . They are estimated by the same differences of sample cell means.
In plain words A simple effect answers what changing one factor does while the other is held at a particular setting. It describes the effect under specified conditions, not on average.
Example Turning hints on raises the mean score from to with the current format, a simple effect of points, and from to with the new format, a simple effect of points. The simple effects of the format are points without hints and points with hints.
Main effect
Definition A main effect of a factor compares its levels after averaging the cell means over the levels of the other factor with stated weights. In a design with equal weights, the main effect of A is
which is the average of the two simple effects of A, and the main effect of B is defined in the same way. In an unbalanced or incomplete design, averaging over the other factor can mean different things, for example equal weights for its levels or weights proportional to the cell sizes, and these can give different answers, so the weights have to be stated.
In plain words When the same weights are used at every level of the factor, a main effect is an average of its simple effects. It summarizes a factor well when the simple effects are similar. When they differ, the average can differ from the effect under every one of the conditions, and reporting it alone hides that the effect depends on the other factor.
Example With equal weights, the current format averages and the new format , a main effect of points for A. For hints, and give a main effect of points for B. That lies between the simple effects and , so it does not mean that hints add points under either format.
Interaction
Definition Two factors interact when the simple effects of one differ across the levels of the other. In a design the interaction contrast is the difference of differences
and rearranging shows that it also equals , the difference of the simple effects of A. So the same interaction is found from either factor, as long as the subtractions keep the same direction. More generally, there is no interaction in an design exactly when there are constants with for every cell, so that the effects add. Interaction depends on the scale on which the response is measured or modeled, and effects that add on one scale can interact on another. The estimated interaction varies from sample to sample, and the ordinary F test for it assumes independent normal errors with a common variance.
In plain words Interaction means that the effect of one factor depends on the level of the other, so a question such as how much hints help has no single answer. In a plot of the cell means with one line for each level of A, no interaction gives parallel lines. Interaction gives lines with different slopes, which may or may not cross.
Example For the scores,
Starting from the format instead gives as well. With the contrasts in hand, the ANOVA table for the balanced two-factor model is below.
| Source | SS | df | MS | F |
|---|---|---|---|---|
| A | ||||
| B | ||||
| Interaction | ||||
| Residual | ||||
| Total |
Under the ordinary model the interaction F of with and degrees of freedom has p-value about . With only four residual degrees of freedom, the estimated interaction of points is not strong evidence of an interaction in the population, and it is not evidence of its absence either. To see the role of the scale, take cell means , , and for , , and . Hints add points with and with , an interaction of points, yet they double the mean under both formats, so the logarithms of these cell means show no interaction.
Randomized block design
Definition A block is a group of experimental units that are similar in a characteristic, known before treatment, that is expected to affect the response. In a randomized block design the treatments are randomly assigned separately within each block, and in a complete block design every treatment appears in every block. Treatments are compared within blocks, and the analysis includes the blocks, for example as an additive block term in the model. Randomizing within blocks restricts which assignments could have occurred, so a randomization test has to rearrange treatment labels only within blocks, as the design did. Blocking does not replace randomization, which still balances the characteristics not used to form the blocks, on average over the possible assignments.
In plain words First group similar units, then let chance assign the treatments inside each group. Each comparison is then made among comparable units, and the known differences between the groups are not counted as noise.
Example Suppose the eight learners form two blocks of four by prior experience, measured before assignment, and each block receives each of the four combinations once, at random. The constructed scores are below.
| Block | ||||
|---|---|---|---|---|
| Less experience | 7 | 10 | 11 | 16 |
| More experience | 13 | 14 | 17 | 24 |
The cell means are again , , and , and the block means are and . An analysis that ignores the blocks treats each cell’s two scores as replicates that differ only by noise. Its residual sum of squares is with degrees of freedom, a residual mean square of . With an additive block term, the blocks account for of that, with degree of freedom, leaving a residual sum of squares of with degrees of freedom and a residual mean square of . The treatment comparisons are then judged against far less noise. Those three residual degrees of freedom measure how the treatment differences vary between the blocks, so using them as error assumes that the treatment effects are the same in both blocks apart from noise.
Check yourself
- In the original scores, change only the cell mean from to . Find the simple effects of B, the interaction contrast and the main effects of A and B with equal weights.
Show the answer
The simple effects of B are with and with , so the interaction contrast is and the lines in the interaction plot are parallel. The format means are and , a main effect of points for A. The hint means are and , a main effect of points for B, which now equals both simple effects.
The absence of interaction is a statement about the score scale. As ratios, hints multiply the mean by with and by with , which differ.
- A researcher assigns each of the four combinations to a single learner, records that learner’s scores on ten questions, and analyzes the question scores as a design with ten replicates per cell and residual degrees of freedom. What is wrong?
Show the answer
The treatment combinations were assigned to learners, so the learner is the experimental unit, and each cell has one replicate, not ten. The degrees of freedom measure how one learner’s question scores vary around that learner’s own mean, which says nothing about how different learners given the same treatment would differ. Treating the questions as replicates is pseudoreplication, and it usually makes the uncertainty look much smaller than it is.
At the level of learners, the model with interaction fits the four learner means exactly and leaves residual degrees of freedom, so any difference between cells could simply reflect which learner happened to receive which combination. The remedy is to assign several learners to each combination. Dropping the interaction to free one degree of freedom would need a scientific justification stated in advance.
- For the two-block design in the last card, someone proposes a randomization test that shuffles all eight treatment labels freely among the learners, keeping two learners per combination. How many assignments did the design allow, how many does free shuffling use, and why does the difference matter?
Show the answer
Within each block the four combinations can be arranged among the four learners in ways, and the two blocks are randomized separately, so the design allowed equally likely assignments. Free shuffling uses
assignments, most of which the design could never produce, such as assigning to two learners from the more experienced block. Those assignments mix the large block difference into the reference distribution, so the test no longer matches how the treatments were actually assigned. A valid randomization test rearranges the labels within each block and uses only the assignments the design allowed.