Stacking and Ensemble Comparison, machine learning lesson

A ridge regression, a random forest and a gradient-boosted model have been fitted for the same tabular prediction task, and their validation errors differ by a few percent. Should the team keep the best of them, average all three, or fit a fourth model that learns how to combine them? Each option is a modeling choice, and each needs evidence from data that played no part in making it. This lesson sets out a fair comparison, shows exactly when averaging helps under squared error, and builds a stacked model whose combiner is trained on out-of-fold predictions so that it does not learn from leaked targets.

Key terms

Model comparison

Definition Model comparison chooses among candidate learning procedures for one prediction problem, where each candidate is a complete pipeline with its preprocessing, feature construction and hyperparameter tuning. A fair comparison fixes in advance the target, the population of cases, one plan of data roles and a metric tied to the decision. The plan says which rows train, validate and test, with folds that keep groups together or respect time order when the data require it. Every candidate is then scored on the same evaluation cases, and all fitting and tuning of a candidate happen inside the training part of each split. Because the cases are shared, the difference between two candidates is assessed case by case or fold by fold, and its size is weighed against its uncertainty and against the costs of training, prediction, maintenance and the features each pipeline needs at prediction time. The test set is used once, for the procedure finally chosen.

In plain words Two scores can only be compared if they answer the same question on the same cases. Scoring one model on easy folds and another on hard folds measures the folds, not the models. Pairing removes the shared difficulty, so a small but steady difference can show up even when each model’s score moves a lot from fold to fold. A more complex candidate earns the choice only through a measured improvement that matters for the decision, not by containing more models.

Example Five folds of one constructed development set give these mean squared errors.

Fold	1	2	3	4	5
Ridge regression	$4.0$	$4.4$	$3.6$	$4.8$	$4.2$
Gradient-boosted trees	$3.7$	$4.5$	$3.4$	$4.5$	$4.4$
Ridge minus trees	$0.3$	$-0.1$	$0.2$	$0.3$	$-0.2$

The means are $4.2$ and $4.1$ . Across folds the two scores have standard deviations of about $0.45$ and $0.51$ , mostly because some folds are harder than others for both models. The paired differences vary much less, with standard deviation about $0.23$ , but they still change sign. The gradient-boosted trees win on three folds of five, and their mean improvement of $0.1$ is less than half that standard deviation. The folds share training rows, so the five differences are not independent, and a $t$ interval computed from them tends to understate the uncertainty. This evidence does not single out the gradient-boosted trees, and if the ridge regression is cheaper to run and easier to maintain, it remains a sound choice.

Ensemble selection

Definition Ensemble selection is the decision whether to combine several fitted models and which ones to include, made with development data and judged under the same evaluation plan as any single candidate. For squared error the effect of averaging two models is exact. If models A and B predict $a_i$ and $b_i$ for targets $y_i$ on any set of $n$ cases, and the average predicts $(a_i + b_i)/2$ , then

\begin{aligned} \text{MSE}_{\text{avg}} &= \tfrac12 \text{MSE}_A + \tfrac12 \text{MSE}_B \\[0.4em] &\quad - \tfrac14 D , \end{aligned}

where $D = \frac1n \sum_i (a_i - b_i)^2$ is the mean squared disagreement of the two models. The identity follows from $\bigl(\tfrac{u + v}{2}\bigr)^2 = \tfrac12 u^2 + \tfrac12 v^2 - \tfrac14 (u - v)^2$ applied to the two errors of each case. So the average beats the better of the two models exactly when $D$ is more than twice the gap between their mean squared errors. A widely used procedure of the same name, greedy ensemble selection, starts from an empty ensemble and repeatedly adds, with replacement, the model from a library of fitted models whose inclusion gives the lowest validation loss of the averaged predictions. It runs for a set number of steps or until no addition helps, and keeps the best ensemble found along the way, so a model chosen several times receives a proportionally larger weight.

In plain words Averaging helps only through disagreement. If two models make the same errors, their average makes them too, and if one model is much worse, half of its larger errors can outweigh what the disagreement removes. The selection is itself fitted to the validation cases, so with a large library and few cases it can overfit them, and the chosen ensemble needs a fresh evaluation like any single model.

Example Constructed validation errors, prediction minus target, on four cases are $(1, -1, 1, -1)$ for model A, $(1.1, -1.1, 1.1, -1.1)$ for model C and $(-1.5, 1.5, -0.5, 0.5)$ for model B, with mean squared errors $1$ , $1.21$ and $1.25$ . Greedy selection first picks A. Averaging A with C gives errors of size $1.05$ and mean squared error $1.1025$ , worse than A alone, because their disagreement $D = 0.01$ is far below twice the gap, $0.42$ . Averaging A with B gives the errors $(-0.25, 0.25, 0.25, -0.25)$ and mean squared error $0.0625$ , and the identity agrees, since $D = 4.25$ and

\tfrac12 (1) + \tfrac12 (1.25) - \tfrac14 (4.25) = 0.0625 .

So the selection adds B, although C has the lower error on its own. Adding A, B or C a third time gives about $0.1389$ , $0.2222$ or $0.1622$ , all worse, so the selection keeps A and B with equal weights.

Out-of-fold prediction

Definition Split the development rows into $K$ disjoint folds that together contain every row, and write $k(i)$ for the fold of row $i$ . For each fold $k$ , fit the complete base pipeline, including every imputer, scaler, feature selector and tuning step, on the rows outside fold $k$ , which gives a fitted model $\hat{f}^{(-k)}$ . The out-of-fold prediction for row $i$ is

\hat{z}_i = \hat{f}^{(-k(i))}(\mathbf{x}_i) ,

made by a model that saw neither row $i$ nor its target. Each row receives exactly one out-of-fold prediction from each base pipeline. When rows share a group, such as one customer, the whole group goes into one fold. With time-ordered data each prediction comes from a model fitted on earlier rows only, so the earliest block receives no out-of-fold prediction.

In plain words An out-of-fold prediction imitates how a model will be used, by predicting a case it was not trained on. An in-sample prediction, made by a model fitted on the same rows, can be far better than anything that model will achieve later. Being out of fold is a property of how each number was produced, not the name of a column, and one preprocessing step fitted on all rows is enough to break it.

Example Four constructed rows have inputs $0$ , $1$ , $3$ and $6$ and targets $2$ , $4$ , $3$ and $7$ . Compare a one-nearest-neighbor regressor, which predicts the target of the closest training input, with the model that predicts the mean of the training targets. In sample each row is its own nearest neighbor, so the neighbor model reproduces every target and has mean squared error $0$ , while the mean $4$ has mean squared error $3.5$ . With $K = 4$ folds of one row each, the neighbor model predicts $4$ , $2$ , $4$ and $3$ , with mean squared error $25/4 = 6.25$ , and the mean model predicts the mean of the other three targets, $14/3$ , $4$ , $13/3$ and $3$ , with mean squared error $56/9 \approx 6.22$ . In sample the neighbor model looks perfect, and out of fold the two models are about equally good.

Meta-model

Definition The meta-model of a stack is the second-level model that maps the vector of base predictions for a case to a final prediction. It is trained with the development targets on the out-of-fold predictions of the base pipelines. Common choices have few parameters, such as a linear combination with nonnegative weights, a convex combination whose weights are nonnegative and sum to $1$ , a ridge regression, or for classification a logistic regression on the base probabilities or log-odds. Base predictions are usually highly correlated, and an unrestricted least-squares fit on correlated columns can give large weights of opposite signs that do not carry over to new cases. For two base models A and B with out-of-fold predictions $a_i$ and $b_i$ , the convex stack predicts $b_i + w(a_i - b_i)$ with $0 \le w \le 1$ . If $\sum_i (a_i - b_i)^2 > 0$ , its squared error is a convex quadratic in $w$ with the unrestricted minimizer

w^* = \frac{\sum_i (a_i - b_i)(y_i - b_i)}{\sum_i (a_i - b_i)^2} ,

and the best weight in $[0, 1]$ is $w^*$ clipped to that interval. If $a_i = b_i$ on every row, every weight gives the same fitted predictions, and a convention such as $w = \tfrac12$ has to be declared.

In plain words The meta-model learns how far to trust each base model from the way they err on cases they were not trained on. Keeping it simple matters, because its inputs are close copies of one another and a flexible combiner can fit the quirks of one table. A low error of the meta-model on its own training table is a fitted result, just as the training error is for any other model.

Example A constructed meta-training table has targets $2$ , $4$ , $6$ and $8$ , out-of-fold predictions $1$ , $5$ , $5$ and $9$ from A and $3$ , $3$ , $7$ and $7$ from B. Each base model has mean squared error $1$ , and their errors $(-1, 1, -1, 1)$ and $(1, -1, 1, -1)$ have opposite signs on every row. The differences $a_i - b_i$ are $-2$ , $2$ , $-2$ and $2$ , and the differences $y_i - b_i$ are $-1$ , $1$ , $-1$ and $1$ , so

w^* = \frac{8}{16} = 0.5 ,

which already lies in $[0, 1]$ . The stack predicts $2$ , $4$ , $6$ and $8$ exactly, with meta-training error $0$ , as the averaging identity predicts, since $D = 4$ gives $1 - \tfrac14 \cdot 4 = 0$ . That zero shows the arithmetic of the fit. The weight was chosen with these targets, so the zero says nothing yet about new cases.

Stacking

Definition Stacking, or stacked generalization, combines base pipelines $f_1, \dots, f_J$ through a meta-model $g$ fitted to their predictions. An honest stack is built from development data in four steps.

Compute the out-of-fold prediction $\hat{z}_{ij}$ of every base pipeline $j$ for every development row $i$ .
Fit the meta-model $g$ to the rows $(\hat{z}_{i1}, \dots, \hat{z}_{iJ})$ with targets $y_i$ , sometimes with a few original features added.
Refit every base pipeline on all development rows.
Predict a new input $\mathbf{x}$ by $g\bigl(\hat{f}_1(\mathbf{x}), \dots, \hat{f}_J(\mathbf{x})\bigr)$ , with the refitted base pipelines and the frozen meta-model.

The procedure as a whole is one candidate. Its performance is estimated on outer held-out data, or in an outer cross-validation loop that repeats all four steps inside every outer training part. A meta-model fitted to in-sample base predictions instead tends to put its weight on whichever base model fits its own training rows most closely, which is a form of leakage inside the procedure.

In plain words The base models predict, and the meta-model learns how much to believe each of them from predictions made the way they will be made in use. The refitted base models have seen more rows than the fold models that produced the meta-model’s training table, so their predictions can be distributed a little differently, which is one more reason to evaluate the complete procedure rather than the meta-model’s training fit. In use, every base pipeline and its features must be available, so a stack adds computation, delay and ways to fail.

Example Freeze the weight $w = \tfrac12$ fitted in the meta-model example above and score the stack on two separate constructed evaluation cases with targets $5$ and $9$ . Model A predicts $4$ and $10$ , with mean squared error $1$ , and model B predicts $8$ and $8$ , with mean squared error $5$ . The stack predicts $6$ and $9$ , with mean squared error $\tfrac12$ , so it helps on these cases, although it no longer predicts every target exactly. If B came to repeat the predictions of A, the stack would equal A and gain nothing. If B drifted to predicting $12$ for both cases, the stack would predict $8$ and $11$ , with mean squared error $6.5$ , far worse than A alone.

Check yourself

A frozen stack puts weight $0.2$ on model A and $0.8$ on model B. For a new case, A predicts $12$ , B predicts $7$ and the outcome is $9$ . Find the squared errors of the stack, of A and of B. A colleague notices that the weight $0.4$ on A would have predicted this case exactly and wants to report the stack with that weight as error free. What is wrong with that report?

Show the answer

The stack predicts

0.2 \times 12 + 0.8 \times 7 = 8 ,

so its squared error is $(9 - 8)^2 = 1$ . Model A alone has squared error $(12 - 9)^2 = 9$ and model B alone has $(7 - 9)^2 = 4$ . With the weight frozen before the outcome was seen, these are honest held-out results for one case.

The weight $0.4$ gives $7 + 0.4 \times (12 - 7) = 9$ , but it was chosen after seeing the outcome, so the case has been used for fitting and its error of $0$ is a fitted result. With one free weight, any outcome between the two base predictions can be matched exactly, so the zero shows nothing about new cases. If the team wants to change the weight, the case becomes development data and the new stack needs evaluation on cases that played no part in choosing it.

A team fits a fully grown regression tree A and a ridge regression B on all development rows and fits the convex weight $w$ on A to their in-sample predictions. The inputs are distinct, so the tree reproduces every training target. What weight does the tree receive? On four constructed rows with targets $6$ , $6$ , $4$ and $6$ , the out-of-fold predictions are $7$ , $4$ , $6$ and $5$ for A and $5$ , $6$ , $4$ and $7$ for B. Find the weight fitted to these, and compare it with the in-sample weight.

Show the answer

In sample $a_i = y_i$ , so the numerator and the denominator of $w^*$ are both $\sum_i (y_i - b_i)^2$ . Unless the ridge regression fits every target exactly, the result is $w^* = 1$ , and the meta-model trusts the tree completely, whatever the two models would do on new cases.

Out of fold, the differences $a_i - b_i$ are $2$ , $-2$ , $2$ and $-2$ and the differences $y_i - b_i$ are $1$ , $0$ , $0$ and $-1$ , so

w^* = \frac{2 + 0 + 0 + 2}{16} = 0.25 ,

which lies in $[0, 1]$ . The tree has out-of-fold mean squared error $(1 + 4 + 4 + 1)/4 = 2.5$ and the ridge regression has $(1 + 0 + 0 + 1)/4 = 0.5$ , so the honest evidence favors the ridge regression. The stack with $w = 0.25$ predicts $5.5$ , $5.5$ , $4.5$ and $6.5$ , with mean squared error $0.25$ on these rows. That is a meta-training value, fitted on the same rows, but the comparison is still telling. On these out-of-fold predictions the in-sample weight $w = 1$ reproduces the tree’s error of $2.5$ , ten times as large.

A team estimates two candidates with five outer folds, a ridge regression and a stack of a random forest and gradient-boosted trees with a convex meta-model built from five inner folds. No hyperparameters are tuned. Describe what happens inside one outer fold and count the model fits. The outer-fold mean squared errors are $3.9$ , $4.6$ , $4.1$ , $3.7$ and $4.2$ for the ridge regression and $3.8$ , $4.7$ , $3.9$ , $3.8$ and $4.0$ for the stack. Which candidate should the team prefer?

Show the answer

Inside one outer fold only the outer training rows are used for fitting. They are split into five inner folds. For each base pipeline and each inner fold, the pipeline is fitted on the other four inner folds and predicts the held-out one, which fills two columns of out-of-fold predictions for all outer training rows. The convex weight is fitted to those columns, both base pipelines are refitted on all outer training rows, and the refitted pipelines predict the outer held-out fold. Their predictions are combined with the frozen weight and scored. The ridge regression is fitted on the same outer training rows and scored on the same held-out fold. Each outer fold needs $2 \times (5 + 1) = 12$ base fits and one meta-model fit for the stack, so $65$ fits over the five outer folds, against $5$ for the ridge regression.

The mean squared errors average $4.1$ for the ridge regression and $4.04$ for the stack. The paired differences, ridge minus stack, are $0.1$ , $-0.1$ , $0.2$ , $-0.1$ and $0.2$ , so the stack wins on three folds of five, and the mean improvement $0.06$ , about $1.5$ percent, is less than half the standard deviation of the differences, which is about $0.15$ . The folds share training rows, so these differences are not independent, and their spread tends to understate the uncertainty of the mean. The improvement is small and uncertain, while the stack costs thirteen times as many fits, needs two pipelines and their features at prediction time, and has more ways to fail. Unless a gain of this size is worth those costs for the decision at hand, the ridge regression remains the better choice.

Stacking and Ensemble Comparison

Key terms

Model comparison

Ensemble selection

Out-of-fold prediction

Meta-model

Stacking

Check yourself

How this lesson connects

Builds on

Used later in