Each evening at six, a parcel depot plans the next day’s staffing. Some of tomorrow’s parcels are already confirmed, but more will arrive overnight, so how many should the team prepare for? The depot could plan for the same number every day or follow a rule of thumb, but it can also fit a prediction rule to past days. This lesson turns that question into a learning task, names the main kinds of learning, and explains why a fitted rule has to beat a simple baseline on new days and why it is not a causal explanation. Every number on this page is constructed for teaching.
Key terms
Machine learning
Definition Machine learning uses data to choose a model for a specified task. In a prediction task, a model is a rule that turns the information known about a case, collected in an input , into a prediction . The rule usually comes from a family of candidate rules indexed by parameters, numbers whose values pick one rule from the family. A learning algorithm chooses the parameter values from training data, usually by making the average loss on those data small, where a loss is a number that measures how wrong one prediction is. Choosing the values is called training, and applying the trained model to a new input is called inference. Generalization is how well the trained model performs on new cases from the setting where it will be used.
In plain words Instead of writing the rule by hand, you fix a family of possible rules and a way to score their mistakes, and the data choose among them. Fitting the training data well is only the first test. A model is useful when it predicts well on cases it has never seen and when its predictions help the decision they were made for.
Example At six in the evening the depot knows , the number of parcels already confirmed for tomorrow, and wants to predict , the total number of parcels that will arrive tomorrow. The family of rules
has one parameter, the offset , measured in parcels. Training on past days, worked out in the next card, selects . Inference for an evening with 50 confirmed parcels then gives a prediction of 60 parcels, computed before tomorrow’s total is known.
Supervised learning
Definition In supervised learning the training data are pairs . Each holds the features of observation , the inputs known at the time of prediction, and is its target, the outcome observed later. A learning algorithm uses the pairs to choose a rule whose predictions should be close to the targets of new observations. It typically does so by making the average training loss
small, where is the loss of predicting when the outcome is . With the absolute error , this average is the mean absolute error, or MAE, measured in the units of the target. A small training loss is a means, not the goal. The goal is a small loss on new data, which the training data alone cannot show.
In plain words The training examples come with answers, so a candidate rule can be scored by comparing its predictions with the recorded outcomes. Those answers come from records, measurements or people, and they can be wrong too. Absolute error treats predicting 10 too many and 10 too few alike, and that is a choice. A depot that suffers more from too few staff than from too many might score its mistakes differently.
Example Three past days, each one observation, give the training data below.
| Day | Confirmed | Total |
|---|---|---|
| 1 | 40 | 52 |
| 2 | 60 | 68 |
| 3 | 80 | 90 |
The candidate offsets 0, 10 and 20 are compared by their training MAE.
| Offset | Predictions | Absolute errors | MAE |
|---|---|---|---|
| 0 | 40, 60, 80 | 12, 8, 10 | 10 |
| 10 | 50, 70, 90 | 2, 2, 0 | |
| 20 | 60, 80, 100 | 8, 12, 10 | 10 |
The offset has the smallest training MAE, so training selects it. It is in fact the best offset among all real numbers, because the training MAE of is smallest when is a median of the differences , which are 12, 8 and 10. None of this shows how well the rule will do on a new day.
Regression and classification
Definition Regression and classification are the two main kinds of supervised task, and the target decides which one a task is. In regression the target is a numerical quantity, such as a count, a duration or a price, and the size of a prediction error is meaningful. In classification the target is one of a finite set of categories, called classes, and a prediction names a class. With exactly two classes the task is binary classification. The meaning of the target decides the task, not the way it is stored. A count is a regression target even though it is a whole number, and classes coded as 0 and 1 are still categories.
In plain words Ask what kind of answer the prediction gives. If it is an amount, so that missing by 2 is better than missing by 20, the task is regression. If it is a choice among named groups, the task is classification, and the numbers used to code the groups are names rather than amounts.
Example Predicting tomorrow’s total parcel count is regression. If the model predicts 60 parcels and 63 arrive, the error is 3 parcels. Predicting whether a parcel will miss its delivery deadline is binary classification, with the classes late and on time. Coding late as 1 and on time as 0 does not make it a regression task, since the codes could be swapped without changing anything about the parcels.
Unsupervised and reinforcement learning
Definition In unsupervised learning the data are inputs without the target values a supervised task would use, and the aim is to describe structure in them, such as groups of similar cases, found by clustering, or a few combined variables that summarize many, found by dimensionality reduction. In reinforcement learning an agent interacts with an environment over time. At each step it observes the current state, chooses an action and receives a numerical reward, and its actions can change the states it meets later. The aim is a policy, a rule for choosing actions, with the largest expected total reward over time, where later rewards are often discounted. The feedback is the reward for the action actually taken, not the correct action.
In plain words Unsupervised learning asks what patterns are present when there is no answer column. A group found by an algorithm reflects the chosen measurements and method, and it needs checking before it is treated as a meaningful category. Reinforcement learning learns from the consequences of its own choices. Because each choice changes what happens next, the problem is more than predicting a recorded outcome.
Example Recording the length, width, height and mass of many parcels and asking an algorithm for four groups of similar parcels is unsupervised learning. The groups need not match categories such as envelopes or furniture. A delivery van that chooses which stop to visit next poses a reinforcement learning problem. Each choice changes where the van is and which stops remain, and a reward such as the negative of each move’s driving time scores the choices only after they are made.
Baseline
Definition A baseline is a simple prediction rule that a proposed model is compared with, on the same evaluation cases and with the same loss. It is fixed using only information available before those cases are seen, usually the training data. A common baseline for regression predicts the training mean for every case, and a common one for classification predicts the most frequent class in the training data. A model that does not beat its baseline on suitable new cases adds complexity without giving better predictions.
In plain words A baseline says what a much simpler rule would achieve, and a model’s error means little until it is set next to that number. The mean is the constant with the smallest training squared error. Under absolute error the training median does at least as well on the training data, and either one is a fair baseline as long as it is fixed before the evaluation cases are seen.
Example The training totals 52, 68 and 90 give
so the baseline predicts 70 parcels for every day, with training MAE parcels. On a later day that was not used in training, 50 parcels are confirmed and 63 arrive. The fitted model predicts 60 and misses by 3 parcels, while the baseline misses by 7. The model did better on this day, but one day is very little evidence. A fair comparison uses many later days and never updates the baseline with their outcomes.
Prediction versus causation
Definition A predictive claim says how well the features let us guess the target for cases like those in the data. A causal claim says how the target would change if someone intervened to change a feature. A model that predicts well supports only the first kind of claim. It learns associations in its training data, and these can come from common causes, from the way the data were collected or from conditions that may change. A causal conclusion needs further evidence, such as a randomized experiment or clearly stated and defensible assumptions about how the data arose.
In plain words Being able to guess from does not mean that changing would change . A predictive pattern also lasts only as long as the conditions that produced it. And a prediction is not yet a decision, which also needs the costs of the different possible mistakes.
Example The fitted rule does not show that one more confirmed parcel causes one more parcel to arrive. Here a confirmation is best read as an early sign of volume that is on its way. Now suppose a new contract brings many unconfirmed parcels, and on a day with 50 confirmed parcels the total is 95. The unchanged model predicts 60 and misses by 35, while the baseline predicts 70 and misses by 25. The relationship the model learned no longer describes the depot, so the change has to be investigated and the rule evaluated again. Even an accurate count would not settle the staffing level, which also depends on how many parcels a worker can handle and on the costs of too few or too many staff.
Check yourself
- Say which kind of learning each task is. (a) Predict how many minutes a delivery will take, from information known when the van leaves. (b) Predict whether a customer will cancel a subscription next month. (c) Divide customers into groups with similar buying habits, with no target given. (d) Teach a warehouse robot to pick orders, where each move changes its position and which shelves it can reach next, and a reward arrives when an order is complete.
Show the answer
(a) Regression, because the target is an amount of time and the size of an error matters.
(b) Binary classification, with the two classes cancels and stays.
(c) Unsupervised learning, specifically clustering, since no target is given. Whether the groups mean anything useful has to be checked separately.
(d) Reinforcement learning, because the robot’s actions change the states it meets later and the feedback is a reward rather than the correct move.
- A café predicts each evening how many lunches it will serve the next day from , the number of reservations known that evening. Three constructed training days have , and . Find the training MAE of the rules for , and , and choose . Compute the mean baseline and compare the two rules on a later day with 25 reservations and 37 lunches. A colleague wants to include the later day in the baseline average before comparing. Should they?
Show the answer
With the predictions are 25, 35 and 45, the absolute errors are 6, 6 and 3, and the MAE is lunches. With the predictions are 30, 40 and 50, the errors are 1, 1 and 2, and the MAE is lunches. With the predictions are 35, 45 and 55, the errors are 4, 4 and 7, and the MAE is again 5 lunches. Training selects .
The baseline predicts the training mean
lunches. On the later day the model predicts and misses by 2 lunches, while the baseline misses by 3.
The colleague should not. The 37 lunches of the later day are the outcome being predicted, and that outcome is unknown when the prediction is made. Averaging it in would give a baseline of and an error of 2.25, from a rule that used the answer it was supposed to predict. Both rules have to be fixed before the evaluation day’s outcome is seen.
- A manager reads the depot rule and concludes that each extra confirmation brings one extra parcel, so a campaign that gets customers to confirm earlier will raise tomorrow’s volume. What is wrong with this reasoning? On a day that would have had 50 confirmed parcels and 63 in total, the campaign gets 10 of the unconfirmed parcels confirmed in advance. What does the rule predict, and how large is its error?
Show the answer
The rule describes an association in past days, where the confirmed count was an early sign of how much volume was on its way. It says nothing about what happens when someone intervenes to change the count. In this scenario the parcels confirmed because of the campaign would have arrived anyway, so the campaign raises without raising .
On the example day the confirmed count rises to while the total stays at 63. The rule predicts 70 and overestimates by 7 parcels, whereas without the campaign it would have predicted 60 and missed by 3. The intervention changed the relationship the model relied on. Whether confirmations affect volume at all would need different evidence, such as an experiment that randomly chooses which customers receive the campaign.