Logistic Regression

Definition Binary classification is a supervised learning task in which the target $Y$ takes exactly two values, coded $y = 1$ for the positive class and $y = 0$ for the negative class, and the input is a feature vector $\mathbf{x}$ in $\mathbb{R}^p$ known at prediction time. A classifier either outputs a predicted label $\hat{y}$ in $\{0, 1\}$ directly or estimates the conditional probability $P(Y = 1 \mid \mathbf{x})$, and a decision rule then turns that estimate into a label. Which class is called positive is a coding convention, usually the event of interest. Swapping the codes replaces every probability $p$ of the positive class by $1 - p$.

Definition The sigmoid function, also called the logistic function, is

for every real number $z$. Its values satisfy $0 < \sigma(z) < 1$. It is strictly increasing, with $\sigma(0) = 1/2$, $\sigma(z) \to 1$ as $z \to \infty$ and $\sigma(z) \to 0$ as $z \to -\infty$. It has the symmetry $\sigma(-z) = 1 - \sigma(z)$, and its derivative is $\sigma'(z) = \sigma(z)\bigl(1 - \sigma(z)\bigr)$, which is largest at $z = 0$, where it equals $1/4$.

Definition For a probability $p$ with $0 \le p < 1$, the odds of the event are $p/(1 - p)$, a number in $[0, \infty)$. For $0 < p < 1$ the log-odds, also called the logit, are

which can be any real number. Solving for $p$ gives $p = \text{odds}/(1 + \text{odds})$. If the log-odds equal $z$, then $p/(1 - p) = e^{z}$, so $p = e^{z} - p\,e^{z}$ and $p = e^{z}/(1 + e^{z}) = \sigma(z)$. The logit and the sigmoid therefore undo each other, with $\operatorname{logit}(\sigma(z)) = z$ for every real $z$ and $\sigma(\operatorname{logit}(p)) = p$ for $0 < p < 1$.

Definition Logistic regression is a model for a binary target. For a feature vector $\mathbf{x}$ in $\mathbb{R}^p$, an intercept $b$ and a weight vector $\mathbf{w}$ in $\mathbb{R}^p$, it states

so the log-odds of the positive class are the linear score $z$. Linearity of the log-odds in the chosen features is an assumption of the model, and the features may include engineered ones such as squares or products. The log loss of one example with label $y$ and predicted probability $p$ of class $1$ is

which equals $-\ln p$ when $y = 1$ and $-\ln(1 - p)$ when $y = 0$. Given training examples $(\mathbf{x}_i, y_i)$ for $i = 1, \dots, n$, the parameters are usually estimated by minimizing the mean log loss, the empirical risk

where $p_i = \sigma(b + \mathbf{w}^{\top}\mathbf{x}_i)$. The next lesson derives this loss from the likelihood. The function $\hat{R}$ is convex in $(b, \mathbf{w})$, and its gradient with respect to $(b, \mathbf{w})$ is

The empirical risk has exactly one minimizer when no nonzero $(b, \mathbf{w})$ gives $b + \mathbf{w}^{\top}\mathbf{x}_i \ge 0$ for every example with $y_i = 1$ and $b + \mathbf{w}^{\top}\mathbf{x}_i \le 0$ for every example with $y_i = 0$. If a hyperplane separates the two training classes in this way, with at least one score not zero, the loss keeps decreasing as those separating coefficients are multiplied by larger and larger numbers, and it has no minimizer.

Definition The odds ratio comparing two feature vectors is the odds at the first divided by the odds at the second. In logistic regression the odds at $\mathbf{x}$ are $e^{b + \mathbf{w}^{\top}\mathbf{x}}$. If feature $x_j$ enters the model only through the term $w_j x_j$, then raising $x_j$ by $c$ units while every other feature stays fixed multiplies the modeled odds by

at every starting point $\mathbf{x}$, where $\mathbf{e}_j$ has $1$ in position $j$ and $0$ elsewhere. The odds ratio depends on the unit of $x_j$, since measuring the same quantity in a unit $k$ times smaller divides its coefficient by $k$. If $x_j$ also appears in a squared or product term, the odds ratio for a change in $x_j$ depends on the starting values.

Definition A decision rule with threshold $t$, where $0 < t < 1$, predicts class $1$ when the estimated probability satisfies $\hat{p}(\mathbf{x}) \ge t$ and class $0$ otherwise. The sigmoid is strictly increasing, so for logistic regression $\hat{p}(\mathbf{x}) \ge t$ holds exactly when $b + \mathbf{w}^{\top}\mathbf{x} \ge \operatorname{logit}(t)$. The decision boundary is the set of feature vectors with

When $\mathbf{w} \ne \mathbf{0}$ it is a hyperplane in feature space, a point for one feature and a line for two. When $\mathbf{w} = \mathbf{0}$ every case gets the same probability and the same decision. For $t = 1/2$ the boundary is the set where $z = 0$. Points on the boundary are assigned to class $1$ under this rule. If the features include engineered terms such as $x_1^2$, the boundary is a hyperplane in those features and can be curved in the original inputs.