Logistic Regression

Definition For a probability $p$ with $0 < p < 1$, the odds are $p/(1 - p)$ and the logit, or log odds, is

where $\log$ is the natural logarithm. The logit is strictly increasing, takes every real value exactly once as $p$ runs over $0 < p < 1$, equals $0$ at $p = 1/2$ and satisfies $\operatorname{logit}(1 - p) = -\operatorname{logit}(p)$. It is not defined at $p = 0$ or $p = 1$.

Definition The logistic function, also called the inverse logit, is

for every real $\eta$. Its values lie strictly between $0$ and $1$. It is strictly increasing, equals $1/2$ at $\eta = 0$ and satisfies $\operatorname{logit}^{-1}(-\eta) = 1 - \operatorname{logit}^{-1}(\eta)$. It undoes the logit, since $\operatorname{logit}(\operatorname{logit}^{-1}(\eta)) = \eta$ for every real $\eta$ and $\operatorname{logit}^{-1}(\operatorname{logit}(p)) = p$ for $0 < p < 1$. The formula comes from solving $p/(1 - p) = e^{\eta}$ for $p$.

Definition Logistic regression models a binary response $Y$, coded $1$ for the event and $0$ otherwise, given predictors $x_1, \dots, x_k$. With $p(x) = P(Y = 1 \mid x)$, the model states

so $p(x) = \operatorname{logit}^{-1}(\eta(x))$, where $\eta(x)$ is the linear predictor on the right. The responses are assumed independent given their predictors. Because $Y$ is binary, $\operatorname{Var}(Y \mid x) = p(x)\bigl(1 - p(x)\bigr)$, and no normal errors or constant variance are assumed. The coefficients are usually estimated by maximum likelihood, maximizing

over $\beta$, where $p_i = p(x_i)$. The predicted probability for a profile $x$ is $\hat{p}(x) = \operatorname{logit}^{-1}(b_0 + b_1 x_1 + \cdots + b_k x_k)$, with the estimates $b_j$ in place of the $\beta_j$. It estimates $P(Y = 1 \mid x)$ under the model, and it is not a prediction that the event will happen.

Definition Suppose a predictor $x_j$ enters a logistic regression only through the term $\beta_j x_j$. Raising $x_j$ by one unit while every other predictor stays fixed adds $\beta_j$ to the log odds and multiplies the odds by

whatever the starting values. This number $e^{\beta_j}$ is the odds ratio for a one-unit increase in $x_j$, adjusted for the other predictors in the model, and an increase of $c$ units multiplies the odds by $e^{c\beta_j}$. If $x_j$ also appears in an interaction or a squared term, the odds ratio for a one-unit increase depends on the value of the interacting predictor or on the starting value of $x_j$.

Definition The outcomes of a data set are completely separated when some linear predictor $b_0 + b_1 x_1 + \cdots + b_k x_k$ is positive for every record with $y = 1$ and negative for every record with $y = 0$. Multiplying those coefficients by a constant $c$ and letting $c$ grow sends every likelihood contribution toward $1$. Every finite choice of coefficients gives each record a probability strictly between $0$ and $1$, so the likelihood approaches $1$ without reaching it, and the ordinary maximum likelihood estimate does not exist as a finite vector. A weaker pattern, quasi-complete separation, also rules out a finite estimate. In that pattern some linear predictor is positive or $0$ for every record with $y = 1$, negative or $0$ for every record with $y = 0$, and $0$ for some but not all records.

Definition A model’s predicted probabilities are calibrated in a population when, among the cases that receive predicted probability $q$, the proportion of events equals $q$, for every value $q$ that the model produces. In symbols, $P(Y = 1 \mid \hat{p} = q) = q$. Calibration is checked on observations not used to choose or fit the model, usually by grouping cases with similar predicted probabilities and comparing each group’s observed event rate with its average predicted probability.