Unit 15, lesson 89

Regression Project from Data to Evaluation

How a regression project runs from a data audit and a protected test set through a baseline, a regularized linear model and a tree ensemble to residual analysis, signed mean error and out-of-range inputs.

  • Video 89
  • Regression and Classification Projects

A dispatcher wants to know, at the moment a delivery leaves, how many minutes it will take. A model with a good score somewhere is not yet an answer. The project has to say what one row is, which inputs exist at dispatch, which data stay sealed until the end, what the model is compared with, where its errors fall and where its claim stops. This lesson carries one constructed regression project through all of these steps, from the data audit to a final claim with stated limits.

Key terms

Data audit

Definition A data audit is a documented check of the data against the prediction task, done before any models are compared. It confirms the unit of observation and whether units are independent or come in groups, such as several records of one driver or one customer. It checks that identifiers are unique and that no unit appears in more than one data split, that every field has a stated unit and a plausible range, and which fields are known at the prediction time. It records how missing values arise and how they will be handled, and how the data were collected or generated, which fixes the population the results can describe. Any field recorded after the prediction time, including anything computed from the outcome, is excluded from the inputs.

In plain words The audit asks whether these rows can answer this question before anyone looks at a score. A missing value is not a zero, identifiers that differ from row to row do not prove that the rows are independent, and an input that becomes known only after the event cannot be used to predict it.

Example The task is to predict a delivery’s duration in minutes at dispatch from the planned distance dd in kilometers, the parcel count ℓ\ell and an indicator rr of a declared busy period. All data are simulated for teaching. Each delivery is one independent unit, with dd drawn uniformly between 1 and 10 km, ℓ\ell from 1 to 5 and a busy period with probability 0.350.35, and the duration is generated as

Y=10+1.8d+2ℓ+12r 1{d>6}+3 1{ℓ>3}+ε\begin{aligned} Y &= 10 + 1.8d + 2\ell \\[0.4em] &\quad + 12r \, \mathbf{1}\{d > 6\} \\[0.4em] &\quad + 3 \, \mathbf{1}\{\ell > 3\} + \varepsilon \end{aligned}

with independent noise ε∼N(0,1.22)\varepsilon \sim N(0, 1.2^2). The busy-period term applies only beyond 6 km, which is an interaction, and the parcel term jumps above 3 parcels. A linear model in dd, ℓ\ell and rr can represent neither exactly. The models see only the generated rows, never these rules. After each duration is generated, the parcel count is hidden with probability 0.070.07, independently of everything else, which left 8 training, 4 validation and 3 test rows without a count. Arrival time and realized speed are known only after the delivery, so they stay out of the inputs, and the identifiers of the three splits are disjoint.

Protected test set

Definition A protected test set is a set of units chosen before modeling begins and kept out of every development decision, including fitting, preprocessing, feature selection, tuning and the choice among candidates. It is scored once, after the complete procedure is frozen, for the selected model and for a baseline named in advance, with a metric fixed in advance. If its units are independent draws from the population where the model will be used, its average loss is an unbiased estimate of the frozen model’s expected loss there. Before this comparison the protocol also fixes the independent unit, the setting the model is meant for, and whether the selected model will be refitted on more data first. A test set that has guided any choice has become development data, whatever it is called.

In plain words Training data fit the models, validation data choose among them, and the test set gives the final score of the one that was chosen. Checking test results while still changing the model turns the test into one more round of selection, and its score then tends to flatter the model.

Example The project generated 180 training, 60 validation and 80 test deliveries with separate fixed seeds and distinct identifiers. Missing parcel counts are filled with the training median, 3, and a 0 or 1 indicator column marks which counts were filled, so a filled 3 can be told apart from an observed 3. The means and standard deviations that standardize the inputs of the linear model, computed with divisor nn, also come from the training rows alone. Validation and test rows reuse these saved values and never fit their own. The validation set compared five declared candidates by mean absolute error (MAE), with ties going to the earlier candidate in the declared order. The selected candidate was then frozen as fitted on the 180 training rows, without refitting on training and validation data together, so the model that is scored is exactly the one that was chosen. Refitting would also be a valid plan if it were declared in advance, but it produces a different fitted model. The workflow behind these steps is set out in A Complete Supervised Learning Workflow.

Regularized linear model

Definition A regularized linear model predicts with a linear function of its input columns and is fitted by minimizing the training loss plus a penalty on the coefficients. Ridge regression, the version used here, standardizes each of the pp features with its training mean and standard deviation to give an n×pn \times p matrix ZZ, sets A=[1   Z]A = [\mathbf{1} \;\, Z] with a leading column of ones, and fits θ=(b,w1,…,wp)\boldsymbol{\theta} = (b, w_1, \dots, w_p) by minimizing

J(θ)=1n∥y−Aθ∥22+λ∑j=1pwj2\begin{aligned} J(\boldsymbol{\theta}) &= \frac{1}{n} \lVert \mathbf{y} - A\boldsymbol{\theta} \rVert_2^2 \\[0.4em] &\quad + \lambda \sum_{j=1}^{p} w_j^2 \end{aligned}

with λ≥0\lambda \ge 0 and the intercept bb unpenalized. Setting the gradient to zero gives

(1nA⊤A+λP)θ^=1nA⊤y,\begin{aligned} &\Bigl( \tfrac{1}{n} A^{\top}A + \lambda P \Bigr) \hat{\boldsymbol{\theta}} \\[0.4em] &\quad = \tfrac{1}{n} A^{\top}\mathbf{y} , \end{aligned}

where PP is the diagonal matrix with 0 for the intercept and 1 for each slope. For λ>0\lambda > 0 this system has exactly one solution. Setting λ=0\lambda = 0 gives least squares, whose solution is unique only when AA has full column rank. Because the training columns of ZZ are centered, the fitted intercept is the training mean of the targets. Half of JJ is the ridge objective with the factors 12n\tfrac{1}{2n} and λ2\tfrac{\lambda}{2} used in Ridge, Lasso, and Elastic Net, so the same λ\lambda gives the same fit there.

In plain words The penalty pulls the slopes toward zero and trades some training fit for stability. A larger λ\lambda pulls harder, which can reduce the variance of the fit when data are few or noisy and adds bias when the true relationship needs large slopes. A linear model can only add up the effects of its input columns, so an interaction or a jump that is not among the columns shows up in its residuals.

Example A small constructed practice project has training distances 1, 2, 3 and 4 km with durations 12, 15, 18 and 21 minutes, and uses λ=1\lambda = 1. The distances have mean 2.5 and variance 1.25 with divisor 4, so the standardized feature zz has mean 0 and mean square 1. With one such feature, setting the derivative of JJ with respect to ww to zero gives

w=c1+λ,c=1n∑i=1nziyi,\begin{gathered} w = \frac{c}{1 + \lambda} , \\[0.4em] c = \frac{1}{n} \sum_{i=1}^{n} z_i y_i , \end{gathered}

where cc is the least-squares slope on the standardized scale, so λ=1\lambda = 1 halves it. In original units least squares gives 3 minutes per kilometer and ridge gives 1.5. With the intercept at the mean duration 16.5, the ridge fit in original units is

y^=12.75+1.5x.\hat{y} = 12.75 + 1.5x .

At the validation distances 1.5 and 3.5 km, with observed durations 13.5 and 19.5 minutes, it predicts 15 and 18, an MAE of 1.5 minutes. In the full project the inputs were distance, the filled parcel count, the busy indicator and the missingness indicator. Ridge with λ=0.01\lambda = 0.01 had validation MAE 2.81 minutes and ridge with λ=1\lambda = 1 had 3.81 minutes, so the stronger penalty did worse there.

Tree ensemble

Definition A tree ensemble combines the predictions of many decision trees. The random forest used here fits BB regression trees, each to its own bootstrap sample of nn training rows drawn with replacement. At every node a tree draws a random subset of the features, of a fixed size, and, for each drawn feature jj, considers thresholds tt at the midpoints between consecutive distinct values, sending rows with xj≤tx_j \le t to the left child LL and the others to the right child RR. It chooses the split with the smallest

SSE⁡(j,t)=∑i∈L(yi−yˉL)2+∑i∈R(yi−yˉR)2\begin{aligned} \operatorname{SSE}(j, t) &= \sum_{i \in L} (y_i - \bar{y}_L)^2 \\[0.4em] &\quad + \sum_{i \in R} (y_i - \bar{y}_R)^2 \end{aligned}

among the splits that leave at least a set minimum number of sampled rows, repeats counted, in each child, and splits only if this value is below the sum of squared deviations of the node itself. Growth stops at a maximum depth or when no split qualifies. Each leaf predicts the mean target of its sampled rows, and the forest predicts

f^(x)=1B∑b=1Bf^b(x).\hat{f}(\mathbf{x}) = \frac{1}{B} \sum_{b=1}^{B} \hat{f}_b(\mathbf{x}) .

Every leaf mean lies between the smallest and the largest training target, so every forest prediction does too. Rescaling a feature by a positive factor and a shift moves its thresholds along with it, so trees need no standardization.

In plain words Each tree cuts the input space into boxes and predicts an average inside each box, so it can pick up interactions and jumps without being told about them. Averaging trees fitted to different resamples, with different random features at each split, smooths out their individual quirks. The forest still cannot predict outside the range of the training targets.

Example The full project fitted forests of 31 trees that draw 2 of the 4 input columns at each split and keep at least 5 bootstrap rows in each leaf, with maximum depth 3 or 6. Their validation MAEs were 2.48 and 1.94 minutes. In the practice project, two trees of depth one use given bootstrap samples, all four rows once for the first and rows 1, 1, 3 and 4 for the second, with at least one sampled row in each leaf. The first tree’s thresholds 1.5, 2.5 and 3.5 give sums of squared deviations 18, 9 and 18, so it splits at 2.5 km with leaf means 13.5 and 19.5. The second tree sees distances 1, 1, 3 and 4 with durations 12, 12, 18 and 21, and its thresholds 2 and 3.5 give 4.5 and 24, so it splits at 2 km with leaf means 12 and 19.5. At 1.5 km the ensemble predicts (13.5+12)/2=12.75(13.5 + 12)/2 = 12.75, and at 3.5 km it predicts 19.5, so its validation MAE is (0.75+0)/2=0.375(0.75 + 0)/2 = 0.375 minutes. The training-median baseline of 16.5 minutes has validation MAE 3 and ridge has 1.5, so the ensemble is selected.

Residual analysis and signed mean error

Definition For predictions y^i\hat{y}_i of held-out targets yiy_i, this course writes the residual as ei=y^i−yie_i = \hat{y}_i - y_i, prediction minus observation, so a negative residual is an underprediction. Residual analysis examines these residuals, plotted against the predictions and against each input and summarized within groups of cases, each reported with its count, to find where errors are large or lean one way. The signed mean error of n≥1n \ge 1 cases is

eˉ=1n∑i=1nei,\bar{e} = \frac{1}{n} \sum_{i=1}^{n} e_i ,

the mean prediction minus the mean observation, so a positive value means overprediction on average. By the triangle inequality ∣eˉ∣≤MAE⁡\lvert \bar{e} \rvert \le \operatorname{MAE}, with equality exactly when no two residuals have opposite signs.

In plain words The MAE says how large the misses are, and the signed mean error says whether they lean one way. Positive and negative residuals cancel in the signed mean, so a value near zero does not mean the errors are small, and an average over all cases can hide groups where the model is consistently off.

Example On the 80 protected test deliveries the selected depth-six forest has MAE 2.16 minutes, root mean squared error 2.78 minutes and signed mean error −0.38-0.38 minutes. The first test delivery, 6.92 km with 5 parcels in a busy period, took 48.335 minutes and was predicted at 41.616, a residual of −6.719-6.719 minutes. The 29 busy-period deliveries have MAE 2.64 and signed mean error −1.83-1.83, while the 51 others have MAE 1.89 and signed mean error +0.44+0.44, and the count-weighted average of the two signed means is the overall −0.38-0.38. The 3 deliveries with a missing parcel count were all underpredicted, with MAE 4.98, but 3 cases cannot support a firm conclusion. As an exploratory description, all 15 busy deliveries longer than 6 km were underpredicted, by 3.59 minutes on average, which matches the interaction built into the simulation. The training-median baseline of about 28.12 minutes has test MAE 7.79. Each delivery’s baseline absolute error minus its model absolute error gives a mean improvement of 5.63 minutes, and resampling the 80 deliveries 2000 times, keeping both errors of a delivery together, gives an approximate 95 percent percentile interval from 4.45 to 6.70 minutes. That interval assumes independent, representative test deliveries and treats both fitted models as fixed.

Out-of-range inputs

Definition An input is out of range when it lies outside the region covered by the training data, either because a feature exceeds the values seen in training or because a combination of values never occurred there, even if each value did. A prediction at such an input is an extrapolation, and a test set drawn from the training population contains few or no such inputs, so it says little about their accuracy. How a model extrapolates depends on its form. A linear model continues its fitted slopes. In a tree, every comparison on a feature sends a value above the largest threshold for that feature to the right, and every threshold lies below the largest training value of the feature. So for any two values of a feature at or above its largest training value, with the other inputs fixed, a forest gives the same prediction, and it never predicts beyond the range of the training targets.

In plain words A model has evidence only where it has seen data. Outside that region it still returns a number, and the number can be badly wrong in a consistent direction. The honest responses are to check new inputs against the documented range, to limit the claim, or to collect and evaluate data from the new region.

Example A separately labeled stress population draws distances from 12 to 16 km and keeps every other rule of the simulation. The frozen forest’s MAE rises to 9.32 minutes and its signed mean error is −9.32-9.32 minutes, so every one of the 80 stress deliveries is underpredicted. The largest training distance is just under 10 km, and the forest gives the same prediction at 12 km as at 16 km when the other inputs agree. No forest prediction can exceed the largest training duration, about 50.74 minutes, while the simulated durations keep rising by 1.8 minutes per kilometer on average. The report keeps this failure next to the original test result rather than replacing either one. Its conclusion is that the frozen forest beats the median baseline on simulated deliveries of 1 to 10 km from the same process, that the evidence for longer trips is unfavorable, and that a predictive model of this kind says nothing about what would happen if a dispatcher changed parcel counts or busy-period schedules.

Check yourself

  1. Four held-out deliveries have predictions 20, 25, 31 and 40 minutes and observed durations 22, 24, 35 and 41 minutes. With the residual convention of this lesson, find the MAE, the root mean squared error and the signed mean error, and say what the sign means.
Show the answer

The residuals, prediction minus observation, are −2-2, 11, −4-4 and −1-1 minutes. The MAE is (2+1+4+1)/4=2(2 + 1 + 4 + 1)/4 = 2 minutes. The squared residuals are 4, 1, 16 and 1, so the root mean squared error is 22/4=5.5≈2.35\sqrt{22/4} = \sqrt{5.5} \approx 2.35 minutes, larger than the MAE because the misses have different sizes. The signed mean error is (−2+1−4−1)/4=−1.5(-2 + 1 - 4 - 1)/4 = -1.5 minutes, so on average the predictions are 1.5 minutes too low.

Its size, 1.5, is below the MAE of 2 because one residual is positive and partly cancels the others. A signed mean error near zero would not have shown that the errors were small, only that they balanced.

  1. In the practice project the two-tree ensemble was selected on validation data. Its sealed test deliveries are at 2.5 and 4.5 km with durations 16.5 and 22.5 minutes. Find the ensemble’s test predictions, its test MAE and signed mean error, and the test MAE of the training-median baseline. Why should the miss at 4.5 km be expected, and what do these two cases establish?
Show the answer

At 2.5 km the first tree sends the value at its threshold to the left and predicts 13.5, while the second tree predicts 19.5 because 2.5>22.5 > 2. The ensemble predicts (13.5+19.5)/2=16.5(13.5 + 19.5)/2 = 16.5. At 4.5 km both trees predict 19.5, so the ensemble predicts 19.5. The residuals are 0 and −3-3, so the test MAE is 1.5 minutes and the signed mean error is −1.5-1.5 minutes. The baseline predicts 16.5 for both, with absolute errors 0 and 6 and an MAE of 3 minutes.

The distance 4.5 km lies beyond the largest training distance, 4 km. Both trees put it in their right leaf, whose mean is 19.5, and no prediction of the ensemble can exceed 21, the largest training duration, while the durations in this example rise by 3 minutes per kilometer. The improvement of 1.5 minutes over the baseline comes from two cases, one of them an extrapolation, so it supports no precise claim about performance on new deliveries. It is reported with that limitation.

  1. A teammate computes the median parcel count and the standardization means and standard deviations from all 320 deliveries pooled together. Later, after seeing the test MAE of 2.16 minutes, the teammate fits forests of depth 4, 5 and 8, scores each on the test set and reports the best of them. Which steps break the protocol, and what should the report say?
Show the answer

Both steps break it. Fitting the median and the scaling values on all rows lets the test inputs shape the preprocessing, so the test rows no longer stand in for future deliveries that arrive after these values are fixed. Every preprocessing value must come from the training rows alone and be reused unchanged on validation and test rows.

Choosing a depth by its test score uses the test set for selection. The best of several test scores tends to understate the error of the chosen model on new data, and the test set has become development data. The report should keep the original frozen result, MAE 2.16 minutes against 7.79 for the baseline, and label any later forests as exploratory. A claim about a revised model, refitted with correctly fitted preprocessing, needs new test deliveries that played no part in any choice.

How this lesson connects