A dispatcher wants to know, at the moment a delivery leaves, how long it will take. Fitting a model is only one part of answering that well. Someone has to say what a row is, which inputs exist at dispatch, which data may guide choices and which stay sealed, what a prediction is compared with, and what the final number can and cannot claim. This lesson carries one small constructed delivery problem through every stage, from the data contract to the final report.
Key terms
Supervised learning workflow
Definition A supervised learning workflow is the complete, ordered procedure that turns a prediction question into a reported model. First the decision, the unit of observation, the prediction time, the target and the metric are defined. Next the data are divided by role, into development data for every choice and final data, often called the test set, for the last evaluation, with a split that matches how the model will be used, for example by time when it will predict later cases. Then a baseline and candidate procedures are fitted and compared within the development data, with every preprocessing step learned only from the training part of each comparison. The selected procedure is refitted on all development data, scored once on the final data, and reported with its uncertainty and limitations. Information flows only forward. No final outcome influences any choice, and a change made after the final evaluation needs new final data.
In plain words The workflow is a plan for who may see which data, and when. Each stage answers one question, and the order makes sure that the last number reported played no part in any decision.
Example Six constructed deliveries, listed in order of dispatch time, carry the whole example. They only show the mechanics, since six records cannot establish how a model performs in practice.
| Delivery | Distance (km) | Duration (minutes) | Role |
|---|---|---|---|
| development | |||
| development | |||
| development | |||
| development | |||
| final | |||
| final |
The metric is the mean absolute error (MAE) in minutes, the baseline is a constant and the candidates are straight lines fitted with a ridge penalty of 0 or 1.
Data contract
Definition A data contract is a written specification, fixed before modeling, of what the data mean and how they may be used. It states the decision the predictions support, the unit of observation, the prediction time, the features with the time at which each becomes available, and the exact definition of the target, with its unit and its edge cases. It also states the population and time period the data represent, the rules for missing values, duplicates and groups, the data roles and the evaluation metric. A field recorded after the prediction time may define the target, but it can never be a feature.
In plain words The contract settles what the numbers mean before anyone looks at results. A perfect model of the wrong target, or a model that uses inputs not available when the prediction is made, answers a question nobody asked.
Example One row is one delivery, and the prediction time is dispatch. The feature is the route distance in kilometers, known at dispatch. The target is the time in minutes from dispatch to completion, so the completion timestamp defines the label but cannot be a feature. The metric is MAE, because the average size of a timing error is easy to read for planning. That is a stated choice, and a decision in which late arrivals cost more than early ones would call for an asymmetric loss. Suppose a second question asks whether a delivery is delayed, defined as a duration strictly greater than 18 minutes, so that exactly 18 minutes is not a delay. That is a new binary target with its own contract. On the development rows its labels are 0, 0, 0 and 1. Calling a delivery delayed when its predicted duration exceeds 18 gives a yes or no decision, not a probability of delay.
Development data and refitting after selection
Definition Development data are all the data used to make any modeling choice, including the training and validation parts used for fitting, preprocessing, feature selection, tuning hyperparameters and diagnosing errors. Candidates are compared by a validation scheme inside the development data, such as cross-validation, or forward validation for time-ordered data, in which each fit uses only earlier rows and predicts a later one. The rule for breaking ties is fixed in advance. Refitting after selection means fitting the selected procedure, with its chosen settings, once more on all development data before the final data are opened. What is selected is a procedure, a model family with its features, preprocessing and hyperparameters, and what the final evaluation scores is this refit, whose parameters can differ from those of every inner fit. The inner validation scores helped make the choice, so the winner’s score tends to be optimistic.
In plain words Development data are where all the trying happens. Inner validation picks the settings, and then the chosen recipe is trained once on everything set aside for development, since more data usually give a better fit for the same settings.
Example Each candidate minimizes with the intercept unpenalized, which gives
where and . Forward validation fits deliveries 1 and 2 to predict delivery 3, then deliveries 1 to 3 to predict delivery 4.
| Penalty | Fitted on | Prediction | Observed | Absolute error |
|---|---|---|---|---|
| 1 and 2 | ||||
| 1 to 3 | ||||
| 1 and 2 | ||||
| 1 to 3 |
The mean absolute errors are for and for , so the procedure selects , with ties going to the smaller penalty. Selection uses absolute error even though each fit minimizes squared error. The refit on deliveries 1 to 4 has , , and , so minutes per kilometer, minutes, and the frozen model is .
Baseline comparison
Definition A baseline comparison evaluates a simple reference procedure next to the candidate, fitted on the same training data, scored with the same metric and on the same cases. For a numerical target a common baseline is a constant computed from the training targets, such as their mean, which minimizes training squared error, or a median, which minimizes training absolute error. The baseline is chosen before the comparison, and its constant comes from development targets only, never from final outcomes. A candidate has shown that its features help only if it beats the baseline on held-out cases, by a margin that matters for the decision and is large compared with the uncertainty of the comparison.
In plain words The baseline answers the question of how well one could do without the features. A model that cannot beat it on the same cases has not shown that distance helps, and a baseline that has seen the final outcomes is not a fair opponent.
Example The development mean is minutes. The baseline also goes through forward validation. The mean of deliveries 1 and 2 is 13, which misses delivery 3 by 4 minutes, and the mean of deliveries 1 to 3 is , which misses delivery 4 by minutes. Its inner MAE is minutes, worse than both candidates. On the final deliveries, the constant 15.75 misses by 6.25 and 9.25 minutes, an MAE of 7.75 minutes.
Final evaluation
Definition The final evaluation scores the frozen procedure, refitted on the development data, once on final data that played no part in any choice, with the metric fixed in the data contract. If the final cases are independent draws from the population where the model will be used, the score is an unbiased estimate of the frozen model’s expected loss there, and its precision depends on the number of final cases. The report states the target population, prediction time, split, metric, baseline, sample size, chosen settings, uncertainty and limitations. If the final result leads to any change, such as a new feature, another split or a different threshold, the final data have become development data, and a fresh claim needs fresh final data. The original result and the history of decisions are kept.
In plain words The final evaluation is the sealed exam, opened once after every choice is made. Its number describes the frozen model on cases like the final cases, and the report says exactly what that number does and does not cover.
Example At 5 and 6 km the frozen model predicts 22.5 and 25.2 minutes. The observed durations are 22 and 25, so the absolute errors are 0.5 and 0.2 and the MAE is 0.35 minutes, against 7.75 for the baseline. That is a comparison on two constructed deliveries, not evidence that future errors will average 0.35 minutes. Both final distances also lie outside the development range of 1 to 4 km, so the model is extrapolating. A prediction interval would need separate calibration data set aside after selection, and the two final cases cannot be reused for that, so this example reports point predictions and errors only.
Reproducibility
Definition A workflow is reproducible when someone else, or a later run, can rebuild the same data, the same splits and the same procedure from the records and obtain the same results, up to randomness that is itself recorded. That requires the data source and the version or extraction time, the data contract, the split rule, the code and software versions, the random seeds, every candidate and setting tried together with its score, the tie rules and conventions such as how the penalty is scaled, and the final result with the decisions taken before and after it. A fixed random seed alone does not make a workflow reproducible.
In plain words A reader should be able to follow the records and arrive at the same model and the same numbers, and to see everything that was tried along the way, not only the version that looked best.
Example The phrase “ridge with ” does not fix the fit. With the sum of squared residuals used above, on the four development rows gives the slope . If the objective were instead the mean squared residual plus , multiplying by shows that the same acts like in the sum form, and the slope would be . Likewise, the split should be recorded as “the first four deliveries by dispatch time”, not “the first four rows”, since the rows of a table can be reordered.
Check yourself
- A colleague computes the constant baseline as the mean of all six durations. Find this constant and its MAE on the two final deliveries. Why is it the wrong baseline to compare with the model’s 0.35?
Show the answer
The six durations add to 110, so the constant is minutes. It misses delivery 5 by and delivery 6 by , so its MAE is minutes.
It looks better than the honest baseline’s 7.75 only because it was computed with the two final outcomes it is scored on. Those outcomes would not have been known at dispatch, and using them gives the baseline information the model never had. The honest baseline is the development mean 15.75 with final MAE 7.75. The model still wins here, but a comparison is valid because of how it is built, not because of which side it favors.
- A project report lists two evaluation cases with outcomes 18 and 24, predictions 20 and 21 from the reported model, and a constant baseline of 19. The team tried twelve models on these two cases and reported the best one as final. Compute both MAEs. What is wrong with calling this a final evaluation, and what should be done?
Show the answer
The model’s absolute errors are 2 and 3, so its MAE is 2.5. The baseline’s errors are 1 and 5, so its MAE is 3. On these two cases the reported model is ahead by 0.5.
The same two cases chose the winner among twelve models, so they served as development data. The best of twelve scores tends to understate the chosen model’s error on new cases, and two cases are far too few to show that a difference of 0.5 would persist on new deliveries. The result should be kept and labeled as a development result. The whole selected procedure should then be frozen, refitted on the development data and evaluated on new cases that played no part in any choice. The report should also state the prediction time, the target and its units, the population, the split, the sample size and the decision the predictions support.
- The team now flags a delivery as delayed when its predicted duration is above 18 minutes, using the delay definition from the data contract. For the two final deliveries, give the confusion counts, the recall and the specificity. A manager then asks for the probability that delivery 5 is delayed. What can be said?
Show the answer
Both final durations, 22 and 25 minutes, exceed 18, so both deliveries are delayed. Both predictions, 22.5 and 25.2, also exceed 18, so both are flagged. That gives two true positives, no false negatives, no false positives and no true negatives. The recall is . The specificity has denominator , so it is undefined, and the data say nothing about how often on-time deliveries are wrongly flagged.
The manager’s question has no answer from this model. Comparing a predicted mean duration with 18 gives a yes or no decision, not a calibrated probability. Estimating the probability that a delivery takes more than 18 minutes is a different task, which needs its own model fitted on development data, an evaluation of its probabilities, enough final cases from both classes and an analysis of what each kind of error costs.