A demand forecaster did well on one test period and much worse on the next. The model may be weak, the demand may have changed, or the second test may simply have asked for forecasts farther ahead, and a single split of the data cannot tell these apart. This lesson evaluates a forecasting procedure by replaying, origin after origin, the forecasts it could really have made, with a declared training window and update policy. It then reads the errors by origin and by horizon to separate the effect of a longer horizon from a genuine change in the data.
Key terms
Rolling-origin evaluation and time-series backtesting
Definition Rolling-origin evaluation measures how a forecasting procedure would have performed by replaying it at a sequence of past origins . The procedure includes its features, any fitted preprocessing, its model, its tuning rule and its update policy. At each origin , everything that learns from data is fitted using only information available at , forecasts are made for the declared horizons , and only afterward are they compared with the actual values, which requires for a series of length . Time-series backtesting is the same replay under a name common in finance and forecasting practice, and it is also called time-series cross-validation or evaluation on a rolling forecasting origin. The settings are fixed before the replay. If window lengths or other settings are chosen by comparing backtest scores, those scores become development evidence, and an honest estimate for the chosen procedure needs later origins that played no part in the choice.
In plain words The evaluation stands at each past decision time, hides everything after it, makes the forecasts that could have been made, and only then looks at what happened. Its value comes from reproducing that information boundary at every origin, not from producing many splits. A scaler or imputer fitted on the whole series, a feature built from later values, or a setting tuned on the evaluation origins each break the boundary, even when the model itself sees only earlier targets.
Example The constructed demand series below, in orders per day, keeps the three-day cycle of the lesson on time-series features. Through day 12 each cycle is 2 orders above the one before, and from day 13 on each cycle is 5 orders above the one before. The change is built into the data, so its cause is known.
| Days | First day | Second day | Third day |
|---|---|---|---|
| 1 to 3 | 20 | 24 | 18 |
| 4 to 6 | 22 | 26 | 20 |
| 7 to 9 | 24 | 28 | 22 |
| 10 to 12 | 26 | 30 | 24 |
| 13 to 15 | 31 | 35 | 29 |
| 16 to 18 | 36 | 40 | 34 |
| 19 to 21 | 41 | 45 | 39 |
| 22 to 24 | 46 | 50 | 44 |
The origins are the ends of days 9, 12, 15 and 18, and each origin forecasts the next three days, so every day from 10 to 21 is forecast exactly once. Every procedure uses the lag-3 value as its only input, and for that input is already observed at the origin . The seasonal-naive forecast copies the value three days earlier. The regression is a least-squares line with an intercept on the lag-3 value, refitted at each origin to the six most recent target days. The bagged stumps are nine regression trees with a single split on the lag-3 value, each fitted to a bootstrap sample of the same six rows by choosing the split with the smallest squared error, with the random draws fixed in advance, and the forecast is the average of their leaf means. The table gives each procedure’s mean absolute error over the three horizons, in orders.
| Origin | Seasonal naive | Regression | Bagged stumps |
|---|---|---|---|
| 9 | 2 | 0 | 1.71 |
| 12 | 5 | 3 | 5.01 |
| 15 | 5 | 0.55 | 4.85 |
| 18 | 5 | 0 | 6.07 |
| All 12 forecasts | 4.25 | 0.89 | 4.41 |
The regression wins here because each target in this constructed series is its lag-3 value plus a constant, a relation a refitted line can recover and extend beyond the range of its training targets. A stump predicts a mean of training targets, so the ensemble can never forecast above the largest target in its window, 40 orders at origin 18, and it must fall short of the outcomes 41 and 45 there. Other random draws would change the stump numbers somewhat but not that limit. The ranking describes this designed process only and is not a general verdict on lines and trees.
Sliding and expanding windows
Definition At origin , a sliding window trains on the most recent eligible rows, here the target times , so the amount of training data stays fixed and the oldest rows drop out as the origin moves. An expanding window trains on all eligible rows from the first one up to , so it keeps every earlier row and grows with each origin. A row is eligible when its target and all its features were observed by time . The window length , and the choice between the two kinds of window, are settings of the procedure, fixed in advance or tuned on development origins only.
In plain words A sliding window forgets, which lets it follow a change quickly but leaves fewer rows for each fit, so its estimates vary more. An expanding window remembers everything, which steadies its estimates while conditions stay the same but keeps old relationships in the fit after they have stopped holding. Neither kind is better in general.
Example With lag-3 inputs, the first eligible target day is day 4. At origin 9 both windows contain the target days 4 to 9. At origin 15 the sliding window with holds the target days 10 to 15, with inputs from days 7 to 12, while the expanding window holds the 12 target days 4 to 15. At origin 18 every row of the sliding window comes from after the change, each target equal to its lag-3 value plus 5, so the refitted line is and its three forecasts are exact. The expanding window still holds 9 rows of the old relation among its 15, and its fit, with slope about 1.229 and intercept about , matches neither relation, so its mean absolute error at origin 18 is about 0.81 orders. Over the four origins the expanding window’s mean absolute error is about 1.12 orders, against about 0.89 for the sliding window.
Forecast error
Definition The forecast error of the forecast made at origin for horizon is
the actual value minus the forecast, so every error is indexed by an origin and a horizon. For a set of origins, the mean absolute error at horizon is
and an overall summary states its weights. Equal weight for every origin-horizon pair, for example, gives the mean of the values when every horizon has the same origins. The errors of a rolling evaluation are generally dependent. Neighboring origins share most of their training data and face similar conditions, and when origins are closer together than the longest horizon, the same target day is forecast from several origins.
In plain words An error belongs to a pair, the place the forecast was made and the distance it reached. Comparing a one-day score of one model with a three-day score of another mixes up the method and the distance. Errors are best reported in the units of the decision, here orders, since percentage errors are undefined when demand is zero and unstable when it is near zero. Because the errors are dependent, a handful of origins supports only rough statements. A standard error computed as if each origin’s score were an independent observation can claim far more precision than the data hold when nearby errors move together, and a careful statement needs a method built for dependent data, such as resampling blocks of consecutive origins. An interval for the average error is also a different object from a prediction interval for tomorrow’s demand, which needs its own procedure and assumptions.
Example The regression’s errors, in orders, are 0, 0 and 0 at origin 9, then 3, 3 and 3 at origin 12, about 0.39, and 0.81 at origin 15, and 0, 0 and 0 at origin 18. Its mean absolute errors at horizons 1, 2 and 3 are about 0.85, 0.86 and 0.95 orders, and the equally weighted mean of all twelve is orders. The origins are three days apart and the longest horizon is 3, so each day from 10 to 21 is forecast once. With an origin at the end of every day from 9 to 18, the days 12 to 19 would each be forecast three times, from three different origins, and those three errors would share the same outcome.
Drift
Definition Drift is a change over time in the process that generates the data, that is, in the joint distribution of the features and the target. A change in the distribution of the features alone is often called covariate drift or data drift, and a change in the conditional distribution of the target given the features is called concept drift. The terms are not used uniformly, so a report should say which change it means. Under drift, past errors need not describe future ones, and methods that assume exchangeable cases, such as split conformal prediction, lose their guarantee.
In plain words A rise in errors is a reason to investigate, not proof of drift. Labels may have been revised, an input may have gone missing, a holiday may have changed the mix of days, the evaluation may have moved to longer horizons, or the deployed code may differ from the tested code. Compare the same metric on the same kind of cases at the same horizons before concluding that the process changed. The two kinds of change also come apart in practice. Inputs can shift with no loss of accuracy, and accuracy can fall with no visible change in the inputs.
Example In the constructed series the relation between each target and its lag-3 value changes after day 12, from to . That is a change in the target given the feature, so it is concept drift in the sense above, while the rising level keeps moving the lag-3 values upward before and after the change. The regression’s mean absolute error jumps from 0 at origin 9 to 3 at origin 12, with an error of exactly 3 at all three horizons, so the jump is not a horizon effect. It falls to about 0.55 at origin 15, where the six-row window mixes three rows of each relation, and to 0 at origin 18, where all six rows follow the new one. The fitted slope at origin 15, about 1.208, with intercept about , is a compromise between the two relations and not a new law of demand. The seasonal-naive errors jump from 2 to 5 and stay there, because at every horizon its error equals the rise per cycle.
Update policy
Definition An update policy is the rule that says when and how a forecasting procedure is refitted or updated. It fixes the cadence, such as every origin, every cycle or only after an approved trigger, the training window, which settings are tuned again, and how a new version replaces the old one, including a fallback when data are missing. The update policy is part of the procedure, alongside its features, model and settings, and a backtest has to replay the policy that will actually be used.
In plain words Retraining is an action whose effect has to be measured, not an automatic repair. A backtest that refits more often than the deployed system will overstate how quickly that system adapts. Each forecast should be stored with its time and its model version, so that a later investigation can reconstruct what was actually predicted.
Example Compare two policies on the same origins. Under the first, the regression is fitted once at origin 9, giving , and never refitted. After the change its forecasts are always 3 orders too low, so its mean absolute errors at the four origins are 0, 3, 3 and 3, an overall mean of orders. Under the second, the regression is refitted at every origin on the sliding six-row window, which gives 0, 3, about 0.55 and 0, an overall mean of about 0.89 orders. Both policies make the same errors at origin 12, because no row of the new relation existed yet. They part ways only afterward, when refitting can use the new rows.
Check yourself
- Two origins produce one-day absolute errors of 2 and 4 orders and two-day absolute errors of 5 and 7 orders. Compute the mean absolute error for each horizon and the equally weighted overall mean. A report then says a new procedure improved the mean absolute error from 6 to 3. Why could that be misleading if the 6 came from two-day forecasts and the 3 from one-day forecasts?
Show the answer
The one-day mean absolute error is orders and the two-day value is orders. The overall mean is orders, which is also the mean of 3 and 6 because both horizons have the same two origins.
The numbers 6 and 3 are exactly the two horizon summaries of these same errors. A drop from 6 to 3 can therefore come entirely from switching to an easier horizon, with no change of procedure at all. A fair comparison scores both procedures on the same origins and the same horizons, or reports each horizon separately.
- Continue the regression backtest to origin 21 with the sliding window of six target days. Which target days and lag-3 days does the fit use, what are its coefficients and its forecasts for days 22 to 24, and can the same procedure forecast day 25 from origin 21?
Show the answer
The target days are 16 to 21, with values 36, 40, 34, 41, 45 and 39, and their lag-3 days are 13 to 18, with values 31, 35, 29, 36, 40 and 34. Every target is its lag-3 value plus 5, so least squares gives intercept 5 and slope 1 with zero residuals. The lag-3 inputs for days 22, 23 and 24 are days 19, 20 and 21, with values 41, 45 and 39, so the forecasts are 46, 50 and 44 orders. They match the constructed values, because the series keeps the new relation.
Day 25 is horizon 4 from origin 21, and its lag-3 value is day 22, which is not observed at the origin. The procedure as defined cannot make that forecast. It would need a declared recursive step that uses the forecast 46 for day 22, or a direct model for horizon 4 built on lags of at least 4. Using the actual day-22 value would leak information from after the origin.
- A team runs the four-origin backtest for several sliding-window lengths, picks the length with the lowest overall mean absolute error and reports that error as what to expect in the coming months. They also compute a standard error from the four per-origin values as if these were independent. What is wrong, and what would be more honest?
Show the answer
The window length was chosen with these very scores, so the winning score is an optimistic estimate for the chosen procedure. It is development evidence. An honest estimate comes from later origins that played no part in the choice, used once for the selected procedure, or from a nested replay in which the window is chosen at each origin using only earlier forecasts whose outcomes were already observed.
The four per-origin values are not independent. Neighboring origins share most of their training rows and face similar conditions, so treating them as independent can understate the uncertainty, and four values are too few for a precise statement anyway. A more careful analysis uses a method built for dependent errors, such as resampling blocks of consecutive origins, or reports the full origin table with a plain caution. Even then a backtest average describes the past, and a change in the demand process could make the coming months behave differently. An interval for that average is also not a prediction interval for future demand.