Model Interpretation and Feature Importance, machine learning lesson

A model predicts how many minutes a job will take from two workload indices, and someone asks which index matters more. The question has several honest answers, depending on whether it is about the model’s formula, the model’s accuracy on data, the average prediction as one input is varied, or one particular prediction. This lesson builds each answer for one small constructed model, shows how correlated features and the choice of reference data change them, and separates all of them from the causal question of what would happen if a real workload changed.

Key terms

Interpretability

Definition Interpretability is the degree to which a person can understand how a model turns inputs into outputs, well enough to answer a stated question about its behavior. An explanation is a summary of a fitted model that answers such a question. A global explanation describes the model over a stated population or data set, and a global importance is a number per feature that summarizes how much the model’s predictions or its score depend on that feature there. A local explanation describes one prediction for one input, always relative to a stated reference such as an average prediction. Every explanation describes the fitted model and the data used to probe it. None of them by itself says what would happen in the world if a feature were changed, which is a causal question that needs an experiment or explicit causal assumptions.

In plain words Before choosing an explanation, say who needs it and what they want to know. A developer looking for a shortcut, a person asking about one decision and a manager asking whether to change a real process need different summaries, and the third question cannot be answered by any summary of a predictive model. An explanation can also describe an unreliable model faithfully, so it never replaces checking how well the model predicts on the population it is used for.

Example A constructed data set has two workload indices that always agree, with rows $(0, 0)$ , $(1, 1)$ , $(2, 2)$ and $(3, 3)$ and job durations of 0, 3, 6 and 9 minutes, so the duration is $3x_1$ . The fitted model

f(\mathbf{x}) = 2x_1 + x_2

predicts all four durations exactly, and so do $g(\mathbf{x}) = 3x_2$ and every model $\beta_1 x_1 + \beta_2 x_2$ with $\beta_1 + \beta_2 = 3$ . The coefficient 2 in $f$ says that its prediction rises by 2 minutes when $x_1$ rises by one unit while $x_2$ stays fixed. The data never vary one index while the other stays fixed, so, as with any collinear features, they cannot decide how to split the credit between the two, and they say nothing about the effect of changing a real workload. Units matter as well. Measuring the first index in units 100 times smaller multiplies its values by 100 and divides its coefficient by 100, to 0.02, without changing a single prediction.

Permutation importance

Definition Let $f$ be a fitted model, $D$ an evaluation data set and $L(f, D)$ a loss of $f$ on $D$ , such as the mean squared error, with lower values better. The permutation importance of feature $j$ is the increase in loss when the column of feature $j$ in $D$ is replaced by a random permutation $\sigma$ of itself, with the model and the other columns unchanged,

I_j = L\bigl(f, D_{\sigma}^{(j)}\bigr) - L(f, D),

usually averaged over several random permutations and reported with its spread. With a score where higher is better, the importance is the drop in score. It depends on the fitted model, the loss or score, the evaluation data and the permutations drawn. Computed on training data, it shows what the model uses to fit the training rows, overfit patterns included. Computed on held-out data, it shows the reliance that matters for predicting new data from the same distribution. Permuting a column breaks its association with the target and with the other features, so with correlated features the shuffled rows can be combinations that never occur in real data.

In plain words Shuffle one column and see how much worse the predictions become. A large increase means that this model, on these data, relies on that feature. It does not mean that the feature causes the outcome. When two features carry the same information, a model can rely on either one, so a feature with zero importance may still be strongly related to the target. Permuting a group of correlated features together asks how much the model relies on the group, which avoids splitting credit inside it.

Example With the mean squared error as the loss, $f$ has loss 0 on the four rows. Reversing the first column to 3, 2, 1, 0 and keeping the second gives predictions 6, 5, 4 and 3 against durations 0, 3, 6 and 9, so the errors are 6, 2, $-2$ and $-6$ and the loss is

\frac{36 + 4 + 4 + 36}{4} = 20

squared minutes. Reversing the second column instead gives predictions 3, 4, 5 and 6, errors 3, 1, $-1$ and $-3$ and a loss of 5. The reversal is one chosen ordering. Averaged over all 24 orderings of four rows, the increases are 10 and 2.5. The model $g = 3x_2$ makes the same predictions on the data but ignores $x_1$ , so permuting $x_1$ never changes them and its importance is 0, while reversing $x_2$ raises the loss of $g$ by 45. Two models with identical predictions on the data get opposite importance rankings.

Partial dependence

Definition For a fitted model $f$ , a feature $j$ and evaluation rows $\mathbf{x}_1, \dots, \mathbf{x}_n$ , the partial dependence of $f$ on feature $j$ at the value $v$ is the average prediction after setting feature $j$ to $v$ in every row and keeping the other features of each row,

\operatorname{PD}_j(v) = \frac{1}{n} \sum_{i=1}^{n} f(v, \mathbf{x}_{i,-j}),

where $\mathbf{x}_{i,-j}$ holds the other features of row $i$ . Its population version averages over the marginal distribution of the other features, not over their distribution given $X_j = v$ . When the features are dependent, the average therefore includes combinations of $v$ with other values that rarely or never occur, and the curve need not equal the conditional mean prediction $E[f(\mathbf{X}) \mid X_j = v]$ .

In plain words Partial dependence asks what the model predicts on average if this one input is set to $v$ for every row and nothing else is changed. It summarizes the model’s formula, averaged over the data, and it can show a clean trend in regions where the model has never seen data. It is not an estimate of what would happen if the feature were changed in the world, and it is not the average outcome among cases that actually have that value.

Example For $f$ , setting $x_1 = v$ in every row gives the predictions $2v + x_{i2}$ , and the second column has mean 1.5, so

\operatorname{PD}_1(v) = 2v + 1.5 .

At $v = 2$ this is 5.5 minutes. The only observed row with $x_1 = 2$ has $x_2 = 2$ and prediction 6 minutes. The average of 5.5 also uses the pairs $(2, 0)$ , $(2, 1)$ and $(2, 3)$ , which lie off the line $x_1 = x_2$ where all the data sit. For $g = 3x_2$ the partial dependence on $x_1$ is flat at 4.5, although $g$ makes the same predictions as $f$ on every observed row.

Individual conditional expectation

Definition The individual conditional expectation curve, or ICE curve, of row $i$ for feature $j$ is the model’s prediction as feature $j$ varies while the other features of that row stay at their observed values,

\operatorname{ICE}_i(v) = f(v, \mathbf{x}_{i,-j}).

The partial dependence curve is the average of the $n$ ICE curves. Because the curves are kept separate, differences in their shapes show where the effect of feature $j$ in the model depends on the values of other features, which is an interaction. ICE curves pass through the same constructed combinations as partial dependence, so they share its limits when features are dependent.

In plain words An ICE plot draws one line per row instead of one average line. Parallel lines mean that the model responds to the feature in the same way for every row, apart from a shift, and lines with different shapes mean that it responds differently. An average of opposite trends can be flat, so a flat partial dependence curve does not show that the model ignores the feature.

Example For $f$ the ICE curves of $x_1$ are $2v$ , $2v + 1$ , $2v + 2$ and $2v + 3$ , four parallel lines with slope 2 whose average is the partial dependence curve $2v + 1.5$ . For contrast, take the constructed model $h(\mathbf{x}) = x_1 x_2$ evaluated on two rows with $x_2 = -1$ and $x_2 = 1$ . The ICE curves of $x_1$ are $-v$ and $v$ , and their average is 0 for every $v$ , although the prediction of $h$ changes with $x_1$ in each row.

SHAP values

Definition SHAP values, short for Shapley additive explanations, split one prediction $f(\mathbf{x})$ into a base value plus one contribution per feature. Let $N$ be the set of the $M$ features, and let a value function $v(S)$ give the model’s expected prediction when only the features in a coalition $S \subseteq N$ are known, so that $v(N) = f(\mathbf{x})$ and $v(\varnothing)$ is the base value. Write $\Delta_j(S) = v(S \cup \{j\}) - v(S)$ for the gain when feature $j$ joins the coalition $S$ . The SHAP value of feature $j$ is its Shapley value

\begin{aligned} \phi_j &= \sum_{S \subseteq N \setminus \{j\}} w_S \, \Delta_j(S), \\[0.4em] w_S &= \frac{\lvert S \rvert!\,(M - \lvert S \rvert - 1)!}{M!}, \end{aligned}

a weighted average of the gains, with weights that add to 1. It equals the average gain of feature $j$ over all $M!$ orders in which the features can be added. The contributions add up exactly,

f(\mathbf{x}) = v(\varnothing) + \sum_{j=1}^{M} \phi_j .

With two features, each of the two orders has weight $1/2$ . The values depend on the model, the explained input, the background distribution that defines $v$ and the rule used for unknown features.

In plain words Add the features one at a time, starting from the base value, and record how much the expected prediction moves when feature $j$ arrives. Its SHAP value is that movement averaged over all orders. The contributions say how this model’s prediction for this input differs from a reference prediction, under a stated meaning of an unknown feature. A positive contribution is not advice to raise the feature, and it is not a causal effect.

Example Explain $f$ at $\mathbf{x} = (3, 3)$ , where it predicts 9 minutes, using the four rows as background and filling each unknown feature with its values in the background rows. The base value is the average prediction 4.5. Knowing only $x_1 = 3$ gives the average of $6 + x_{i2}$ , which is 7.5. Knowing only $x_2 = 3$ gives the average of $2x_{i1} + 3$ , which is 6, and knowing both gives 9. With the two features $N = \{1, 2\}$ ,

\begin{aligned} \phi_1 &= \tfrac12 \bigl[v(\{1\}) - v(\varnothing)\bigr] \\[0.4em] &\quad + \tfrac12 \bigl[v(N) - v(\{2\})\bigr]. \end{aligned}

Feature 1 gains $7.5 - 4.5 = 3$ when it comes first and $9 - 6 = 3$ when it comes second, so $\phi_1 = 3$ . Feature 2 gains 1.5 in either order, so $\phi_2 = 1.5$ , and $4.5 + 3 + 1.5 = 9$ . For any linear model $b + \sum_j \beta_j x_j$ , this filling rule gives $\phi_j = \beta_j (x_j - \bar{x}_j)$ , with $\bar{x}_j$ the background mean, here $2 \cdot 1.5 = 3$ and $1 \cdot 1.5 = 1.5$ .

Background distribution

Definition The background distribution $B$ of an explanation is the reference distribution of inputs, usually a sample of rows, that the method uses in place of features it treats as unknown or replaced. It also fixes the base value. Under the marginal rule, also called interventional, the known features keep the values of $\mathbf{x}$ and the others come from background rows $\mathbf{Z}$ ,

v(S) = E_{\mathbf{Z} \sim B}\bigl[f(\mathbf{x}_S, \mathbf{Z}_{\bar{S}})\bigr],

where $\bar{S}$ is the set of features outside $S$ . Under the conditional rule, the unknown features follow their distribution given the known ones,

v(S) = E_B\bigl[f(\mathbf{X}) \mid \mathbf{X}_S = \mathbf{x}_S\bigr],

which in practice has to be estimated. Partial dependence and permutation importance also rely on a reference set, the evaluation rows. A different background generally gives a different base value and different contributions, and the choice should match the comparison the explanation is meant to make.

In plain words A local explanation always compares a prediction with something, and the background is that something. Explaining a prediction against all past jobs, against jobs of the same type or against one typical job gives different numbers, and each can be computed correctly. The marginal rule asks how the model’s formula responds when features are swapped in, even into combinations the data never show. The conditional rule asks how the expected prediction changes as feature values become known, so correlated features tend to share credit.

Example For $f$ at $(3, 3)$ with the four rows as background, the conditional rule notes that the two indices always agree, so knowing either one equals 3 implies the other does too, and $v(\{1\}) = v(\{2\}) = 9$ . Then $\phi_1 = \tfrac12 (9 - 4.5) + \tfrac12 (9 - 9) = 2.25$ , and likewise $\phi_2 = 2.25$ . The model $g = 3x_2$ gets the same 2.25 and 2.25 under the conditional rule, because that rule only uses predictions on rows like the observed ones, while the marginal rule gives $g$ the contributions 0 and 4.5. Replacing the background by the single row $(0, 0)$ under the marginal rule changes the base value of $f$ to 0 and its contributions to 6 and 3. All of these add up to the same prediction 9, and they answer different questions.

Check yourself

A constructed model is $f(\mathbf{x}) = 1 + 2x_1 + 3x_2$ , the background means are 1 for $x_1$ and 2 for $x_2$ , and the input to explain is $(3, 1)$ . Under the marginal rule, find the base value, both SHAP values and the prediction, and check that they add up. Is the positive contribution of $x_1$ a reason to raise $x_1$ ?

Show the answer

For a linear model under the marginal rule, the base value is the prediction at the background means, $1 + 2 \cdot 1 + 3 \cdot 2 = 9$ , and only the background means matter. The contributions are

\begin{aligned} \phi_1 &= 2\,(3 - 1) = 4, \\[0.4em] \phi_2 &= 3\,(1 - 2) = -3 . \end{aligned}

Their sum with the base value is $9 + 4 - 3 = 10$ , which matches the direct prediction $1 + 2 \cdot 3 + 3 \cdot 1 = 10$ .

The contribution 4 says that this model’s prediction for this input is 4 above the reference because $x_1$ is 3 rather than the background mean 1, under the marginal rule. It describes the model, not the world. Whether raising $x_1$ in a real process would change the outcome is a causal question, and the model’s coefficient can differ from the real effect when $x_1$ is correlated with other causes or when the data never varied it alone.

A report on the workload model $f = 2x_1 + x_2$ says that the importances 20 and 5 from reversing each column show that index 1 is four times as important as index 2, and that cutting index 1 would shorten jobs the most. Find the reversal importances for $g = 3x_2$ , which fits the same data exactly, and for both models with the two columns reversed together. What is wrong with the report?

Show the answer

Reversing $x_1$ leaves the predictions of $g$ unchanged, so its importance is 0. Reversing $x_2$ gives predictions 9, 6, 3 and 0 against durations 0, 3, 6 and 9, errors 9, 3, $-3$ and $-9$ , and a loss of

\frac{81 + 9 + 9 + 81}{4} = 45 .

Reversing both columns together keeps every row on the line $x_1 = x_2$ , so both models predict 9, 6, 3 and 0 and both get the importance 45.

The report treats a property of one fitted model as a property of the features. The two indices are identical in the data, so the split of credit between them is arbitrary, and $f$ and $g$ give opposite rankings with the same predictions. Reversing one column also creates rows the data never contain. The grouped importance of 45 is the same for both models and says that the predictions rely on the shared workload information. The causal claim is unsupported in any case. Permutation importance measures how much the predictions of this model degrade under a shuffle, and the data never varied one index alone, so neither the importances nor the coefficients show what cutting a real workload would do.

Three people ask about the workload model. A developer suspects that the model uses a column recorded only after each job finished. A customer wants to know why their job received a 9-minute estimate. A manager wants to know whether reducing index 1 would shorten jobs. Which kind of explanation fits each request, and what must be stated with it?

Show the answer

The developer needs a global view of what the model relies on. Permutation importance on held-out data, repeated over several permutations and with correlated columns permuted as groups, shows whether the predictions depend heavily on the suspicious column. The check should be paired with a review of when each column becomes available, since importance cannot by itself prove a leak.

The customer needs a local explanation of one prediction, for example SHAP values. The report should state the base value, the background it comes from, such as recent jobs of the same kind, and the rule for unknown features, and it should say that the contributions describe the model’s estimate, not the causes of the job’s duration.

The manager is asking a causal question that no summary of a predictive model answers. Coefficients, importances, partial dependence and SHAP values all describe the fitted model on its data. Answering the manager needs evidence about interventions, such as a randomized trial of reduced workloads or a causal analysis with stated assumptions.

Model Interpretation and Feature Importance

Key terms

Interpretability

Permutation importance

Partial dependence

Individual conditional expectation

SHAP values

Background distribution

Check yourself

How this lesson connects

Builds on

Used later in