A delivery time model has passed its evaluation and is about to answer real requests. What has to be saved so that the served predictions match the evaluated ones, which inputs should it refuse, and what should happen when the incoming data change or the outcomes arrive days later? This lesson follows a small constructed delivery model from a saved pipeline through input checks, versions and monitoring to the model card that states what it is for and where it stops being trustworthy.
Key terms
Inference pipeline and training-serving consistency
Definition An inference pipeline is the complete function that turns a raw input record into the output a user receives. It consists of input validation, feature construction, the fitted preprocessing transformations, the fitted predictor and any decision rule applied to the prediction, such as a threshold. Every quantity the pipeline learned, including means, scales, category vocabularies and imputation values, is estimated from the training data and stored with the model parameters as one unit, as in the preprocessing pipelines used for honest evaluation. Training-serving consistency holds when the serving path computes the same features, with the same definitions, units, order and stored parameters, as the path that was evaluated, so that both apply the same function to any record. It fails when serving code recomputes a fitted quantity from incoming data, reimplements a feature differently, changes a unit or reorders columns. A model file that loads without error reveals none of these failures, since each one happens outside the stored parameters.
In plain words The model that was evaluated is the whole chain from raw input to answer, not just the coefficient array. Serving has to run that same chain with the same stored numbers, and recomputing any of them from today’s requests quietly swaps in a different model.
Example Eight constructed training deliveries combine distances of 2, 4, 6 and 8 km with package counts of 1 and 3, and their durations follow the rule minutes exactly. These noiseless records show the mechanics and say nothing about real delivery accuracy. The training means are and the population standard deviations are , and the standardized features are . Least squares in these coordinates gives intercept and coefficients and , so the saved pipeline predicts
For 6 km and 3 packages, and minutes. Substituting the stored transformation gives
so the standardized and raw-unit formulas are one function. Now suppose the serving code centers each request with the means of its own batch while keeping the stored scales. When requests arrive one at a time, each one is centered on itself, both standardized features become , and every request gets minutes. The 6 km, 3 package delivery silently drops from 36 to 31 minutes.
Schema validation
Definition A schema is a declared contract for input records. It lists the required fields by name, with their types, units and allowed values, and says what happens to a record that breaks it. Schema validation checks each incoming record against the contract before any feature is computed. It rejects missing or unexpected fields, wrong types, nonfinite numbers and values outside the declared ranges, and it builds the feature vector by field name rather than by position. A record that fails goes to a defined outcome, such as review with a stated reason, rather than receiving a prediction from replacement values that were never fitted or evaluated. Passing validation shows only that a record fits the contract. It does not show that the units upstream were right or that the record resembles the training data.
In plain words Validation stops numbers from losing their meaning on the way in. It catches wrong fields, types and impossible values, but a value that is wrong yet plausible slips through, so the contract also has to be agreed with whoever produces the data.
Example The demonstration schema requires exactly two fields, distance_km, a finite number from 0 to 10 inclusive, and package_count, an integer from 1 to 5. These ranges are choices made for the example. If the feature vector were built by position and the fields arrived in the other order, the 6 km, 3 package request would become and the pipeline would return minutes instead of 36, although the vector has the right length. Building the vector by name prevents this, and the validator would also reject a count of 6. A distance of 6000 sent in meters fails the range check and goes to review with the field and its expected unit. A distance of 4 miles with one package, sent as 4 km, passes and is predicted at 26 minutes, while the true 6.44 km would give about 33.3 minutes under the training rule. A distance of 10 km is accepted although training covered only 2 to 8 km, so a prediction there is an extrapolation.
Model versioning and rollback
Definition Model versioning gives each complete pipeline bundle an immutable identifier and keeps with it what is needed to identify and reproduce it, namely a reference to the exact training data, the code version, the feature definitions and schema, the dependency versions, the fitted parameters and the evaluation record. Each prediction is logged with the version that produced it, its time and a request identifier, and logs keep only what their stated purpose needs. Reloading a saved bundle and comparing its predictions on fixed reference rows with the originals checks the saved inference function. Reproducing the training as well needs the training inputs, the algorithm settings and the computing environment, which a random seed alone does not describe. A content hash detects changes only against an expected hash from a trusted source, and some serialization formats, such as Python’s pickle, can run code while loading, so bundles are loaded only from trusted sources. Rollback switches future requests back to a retained earlier bundle, restoring its preprocessing, schema and decision rule together with its parameters, after checking that it accepts the current inputs. It does not undo decisions already made, and it does not show that the earlier version is better on today’s data. A fallback is a different thing, a declared simpler behavior used while the normal predictor is unavailable.
In plain words Every answer the system gives should be traceable to one exact saved pipeline. Going back means restoring a whole earlier pipeline, not just older coefficients, and it changes future answers only.
Example The bundle demo-v1 stores the field names, units, schema, means , scales , coefficients, intercept and a fallback value. After it is written as JSON and read back, its predictions on reference rows match the originals exactly. A second bundle demo-v2, made for illustration by raising the intercept by 1, predicts 37 minutes for the 6 km, 3 package request, and switching back to demo-v1 restores 36. The fallback for valid records while the predictor is down is the training mean of 31 minutes. Four valid deliveries of 6 and 8 km with 1 or 3 packages take 32, 36, 38 and 42 minutes under the training rule. The fallback misses them by 1, 5, 7 and 11 minutes, an MAE of 6 minutes against 0 for the normal model. Invalid records still go to review, because a mean does not repair a record whose meaning is unknown.
Drift monitoring
Definition Drift monitoring compares the data a deployed model receives, and later its outcomes, with a declared reference, usually the training or validation data, using stated summaries and alert rules. Input drift, a change in the distribution of the features, can be measured as soon as requests arrive. A change in the relation between features and target, called concept shift, shows up only in labeled outcomes. An input alert is a reason to investigate upstream definitions, changes in who is being served and the available outcome evidence. It does not prove that performance declined, and the absence of an input alert does not prove that performance held. Each alert rule needs an owner and a planned response, such as investigating, pausing use, fixing an input path or evaluating a new candidate model, and the signal alone does not say which response is right. Retraining on a broken input field would build the wrong meaning into the next model. A threshold chosen for illustration has no calibrated false alarm rate. Service signals such as latency, timeouts and failures are monitored as well, separately from predictive quality.
In plain words Monitoring compares what the model sees now with what it was built on. An alert is a question to investigate, not a verdict about accuracy, which only the outcomes can settle.
Example The reference is the eight training rows, with mean distance 5 km and half of the distances above 5 km. A new stream of four deliveries has distances 6, 6, 8 and 8 km with 1, 3, 1 and 3 packages, so its mean distance is 7 km and every distance is above 5 km. The standardized mean difference is
a descriptive effect size whose denominator is the feature’s standard deviation, not a standard error, so it is not a test statistic. An illustrative rule fires when the fraction above 5 km changes by more than 0.3, and here it changes by 0.5. If the training rule still holds, the saved model predicts 32, 36, 38 and 42 minutes with zero error, so the alert came with no loss of accuracy. In the opposite case, requests identical to the eight training rows arrive but every delivery takes 8 minutes longer. No input summary moves, and the MAE goes from 0 to 8 minutes.
Delayed label
Definition A label is delayed when the true outcome of a prediction becomes known only some time after the prediction is made. Evaluating such predictions means joining each outcome to the original prediction through a stable identifier, the event time and the logged model version, so that the score describes what the system actually predicted rather than what a newer model would predict. A score computed while labels are still arriving covers only the labeled part of a cohort, and it is reported with its label coverage, the fraction of the cohort’s predictions that have outcomes. Unlabeled predictions are not zero-error cases, and late labels can belong to systematically different cases, so cohorts are compared at similar maturity and the reasons for missing labels are checked. Revised and duplicate labels need a stated rule.
In plain words The answer key arrives late and in pieces. An early score describes only the cases that came back first, and those may be the easy ones.
Example For the four stream deliveries above, with predictions 32, 36, 38 and 42 minutes, suppose the eventual durations are 32, 36, 46 and 50 minutes. At first only the first two outcomes are known. Their errors are 0 and 0, so the MAE on labeled cases is 0 at a coverage of 2 of 4, or 50 percent. When the last two arrive, with errors 8 and 8, the MAE of the full cohort is
minutes. In this construction the slower deliveries were also the ones reported late, so the early figure overstated the performance on the cohort.
Model card
Definition A model card is a short document attached to a model version. It states the intended use and the uses the model is not meant for, the inputs with their units and the output, the training and evaluation data, the metrics with the population and conditions in which they were measured, the performance with its uncertainty and, where relevant, by subgroup, and the known limitations, monitoring and fallback policies. It describes the fitted system. Data documentation, sometimes written as a datasheet, describes the data themselves, where examples and labels came from, their units, collection times, exclusions, missing values and permitted uses, which ties deployment back to the permission questions of the previous lesson. A model card reports evidence and does not replace it, so it contains only results that were actually measured.
In plain words The model card tells a reader what the model is for, how it was checked and where it should not be trusted, in terms specific enough to act on.
Example The card for demo-v1 names the task, delivery duration in minutes, the two inputs distance_km and package_count with their units, the eight synthetic training rows and the version. It says that recovering the noiseless rule exactly is a check of the implementation, not an estimate of real accuracy. It records the schema rule that sends invalid records to review, the 31 minute fallback and the limits of the constructed drift and delayed label scenarios. It claims no deployment history and no subgroup results, because none exist.
Check yourself
- Use the saved pipeline to predict the duration for 4 km and 1 package. A record then arrives with distance 6000 and 1 package. What should happen to it, and why not clip the distance to 10?
Show the answer
The standardized row is , so minutes. The raw formula agrees, since .
The value 6000 is outside the declared range of 0 to 10 km, so the record goes to review with the field and its expected unit. Clipping would produce a confident prediction of minutes for a record whose meaning is unknown. If the value was 6000 meters, the correct input is 6 km and the prediction is 32 minutes. The repair belongs at the source of the unit error, followed by a check of the corrected pipeline on reference records.
- An engineer changes the serving code to subtract the feature means of each incoming batch, keeping the stored scales and coefficients. The next batch is the stream of 6, 6, 8 and 8 km with 1, 3, 1 and 3 packages. What does the pipeline predict, and what is the MAE if the training rule still holds?
Show the answer
The batch means are . The distance term becomes instead of , which lowers every prediction by 6 minutes. The predictions are 26, 30, 32 and 36 minutes against true durations of 32, 36, 38 and 42, so the MAE is 6 minutes, where the saved pipeline had 0.
Nothing fails at load time, because the model file is unchanged. The function served is no longer the one that was evaluated, and the same request now gets different answers in different batches. Each batch is centered on itself, so its average prediction is always 31 minutes, here , and a change in the typical distance can no longer move the average prediction at all.
- A cohort has 10 predictions. Two days later, labels have arrived for 6 of them, with absolute errors 1, 0, 2, 1, 0 and 2 minutes. The other 4 arrive later with absolute errors 6, 8, 5 and 9 minutes. An input drift alert also fired for this cohort. Report the early MAE with its coverage and the complete MAE. Does the early figure show that the alert was harmless?
Show the answer
The early MAE is minute at a coverage of 6 of 10, or 60 percent. The complete MAE is
minutes, since the late errors add up to 28.
No. The early figure describes only the six cases labeled first, and the four missing ones were not zero-error cases. They turned out to be the worst. An early MAE can be compared only with earlier cohorts at the same two-day maturity, and the team should check whether late labels go with long or unusual deliveries. Whether accuracy changed is judged from the complete outcomes, compared with the complete outcomes of earlier cohorts. The alert by itself says only that the inputs moved.