Robust Regression and Quantile Prediction

Definition In this lesson the residual of a prediction $q$ for an outcome $y$ is $u = y - q$, actual minus prediction, so $u > 0$ is an underprediction, a shortage, and $u < 0$ is an overprediction, an excess. The lesson on regression metrics used the opposite sign. Squared, absolute and Huber losses do not depend on the sign, but asymmetric losses do.

A loss $L(u)$ acts on the fitted parameters through its derivative. Squared loss $u^2$ has derivative $2u$, which grows without bound, so one observation with a huge residual can move a least-squares fit by an arbitrarily large amount. Robust regression fits a regression model with a loss, or a weighting of the observations, chosen so that a few unusual observations have limited influence on the fit. A common choice is a loss with a bounded derivative, such as the absolute loss $\lvert u \rvert$, whose derivative is $\pm 1$ for $u \ne 0$, or the Huber loss below. For a constant prediction without features, squared loss is minimized by the sample mean and absolute loss by a sample median. A bounded derivative limits the influence of large residuals but not of high-leverage points, observations whose features lie far from the rest, because the gradient with respect to a weight multiplies $L'(u)$ by the feature value.

Definition For a threshold $\delta > 0$, measured in target units, the Huber loss of a residual $u$ is $H_\delta(u) = \tfrac12 u^2$ when $\lvert u \rvert \le \delta$ and

when $\lvert u \rvert > \delta$. The two pieces agree in value and in slope at $\lvert u \rvert = \delta$, so the loss is convex and continuously differentiable. Its derivative is $H_\delta'(u) = u$ for $\lvert u \rvert \le \delta$ and $\delta$ times the sign of $u$ beyond that, so it never exceeds $\delta$ in absolute value. Huber regression with a linear model chooses $b$ and $\mathbf{w}$ to minimize the convex objective

The gradient of term $i$ with respect to $\mathbf{w}$ is $-H_\delta'(u_i)\,\mathbf{x}_i$, which is capped in the residual but not in the features. The minimizer of the expected Huber loss is in general neither the conditional mean nor the conditional median, although for a conditional distribution with finite mean that is symmetric about a point, that point is a minimizer.

Definition An asymmetric cost charges different amounts for an underprediction and an overprediction of the same size. With $u = y - q$, a per-unit shortage cost $c_u > 0$ and a per-unit excess cost $c_o > 0$, the linear asymmetric cost is

Dividing by $c_u + c_o$ does not change which prediction minimizes the cost or its expected value, and it turns the cost into the pinball loss below with level

This model assumes that the cost is proportional to the number of units short or over. Fixed fees, capacity limits or stock carried into the next period need a different decision analysis.

Definition For a level $\tau$ with $0 < \tau < 1$, the pinball loss, also called quantile loss, of a residual $u = y - q$ is

Equivalently $\rho_\tau(u) = \max\bigl(\tau u, (\tau - 1)u\bigr)$. It is zero at $u = 0$, positive elsewhere, convex and piecewise linear, with slope $\tau$ for underpredictions and $\tau - 1$ for overpredictions, and $\rho_{1/2}(u) = \tfrac12 \lvert u \rvert$. For outcomes $y_1, \dots, y_n$, a constant $q$ minimizes $\sum_i \rho_\tau(y_i - q)$ if and only if

that is, when $q$ is a sample $\tau$-quantile. The smallest such $q$ is the lower sample quantile.

Definition Let $Y$ be a real-valued target and $X$ the features, with conditional cumulative distribution function, or conditional CDF, $F(y \mid x) = P(Y \le y \mid X = x)$. For $0 < \tau < 1$, a number $q$ is a conditional $\tau$-quantile of $Y$ given $X = x$ if

The set of such numbers is a closed bounded interval, often a single point, and its left end $\inf\{y : F(y \mid x) \ge \tau\}$ is the lower conditional quantile. The level $\tau = 1/2$ gives the conditional median. If $E[\lvert Y \rvert \mid X = x] < \infty$, then $q$ minimizes the expected pinball loss $E[\rho_\tau(Y - q) \mid X = x]$ if and only if $q$ is a conditional $\tau$-quantile, and at any $q$ where $F(\cdot \mid x)$ is continuous, the derivative of this expected loss is $F(q \mid x) - \tau$. Because $\lvert u \rvert = 2\rho_{1/2}(u)$, the same holds for absolute loss and the conditional median. If $E[Y^2 \mid X = x] < \infty$, then $E[(Y - q)^2 \mid X = x] = \operatorname{Var}(Y \mid X = x) + (q - E[Y \mid X = x])^2$, so the conditional mean is the unique minimizer of the expected squared loss.

Definition Quantile regression at a fixed level $\tau$ fits a model of the conditional $\tau$-quantile, for example the linear model $q_\tau(\mathbf{x}) = b + \mathbf{w}^{\top}\mathbf{x}$, by choosing the parameters that minimize the average training pinball loss

The objective is convex but not differentiable where a residual is zero. A minimizer always exists, can be computed by linear programming and need not be unique. When the model has an intercept, every minimizer leaves at most $n\tau$ training points strictly below the fitted function and at least $n\tau$ at or below it. If $E[\lvert Y \rvert] < \infty$ and the model class contains the true conditional quantile function, that function minimizes the expected pinball loss over the class, so the fit estimates it. Fits at different levels are separate problems and can cross. Two fitted quantiles give a candidate prediction interval, but its nominal level does not guarantee coverage on new data.