Maximum Likelihood and Cross-Entropy

Definition Fix observed training data $(\mathbf{x}_1, y_1), \dots, (\mathbf{x}_n, y_n)$ and a model that gives, for each parameter value $\boldsymbol{\theta}$ in an allowed set $\Theta$, the probability $P_{\boldsymbol{\theta}}(y \mid \mathbf{x})$ that the label equals $y$ when the features are $\mathbf{x}$. The likelihood is the probability that the model gives to the observed labels, viewed as a function of $\boldsymbol{\theta}$ in $\Theta$ with the data held fixed. If the labels are modeled as conditionally independent given their features and $\boldsymbol{\theta}$, it is the product

Without the independence assumption the joint probability of the labels need not factor this way. For a continuous target the probabilities are replaced by densities.

Definition A label $Y$ in $\{0, 1\}$ has a Bernoulli distribution with parameter $p$, where $0 \le p \le 1$, when $P(Y = 1) = p$ and $P(Y = 0) = 1 - p$. Both cases fit in one formula,

for $y$ in $\{0, 1\}$, with $0^0$ read as $1$. If a classifier gives example $i$ the probability $p_i = P_{\boldsymbol{\theta}}(1 \mid \mathbf{x}_i)$ of class $1$, and the labels are conditionally independent given their features and $\boldsymbol{\theta}$, the Bernoulli likelihood is

For logistic regression $\boldsymbol{\theta} = (b, \mathbf{w})$ and $p_i = \sigma(b + \mathbf{w}^{\top}\mathbf{x}_i)$.

Definition A maximum likelihood estimate (MLE) is a parameter value $\hat{\boldsymbol{\theta}}$ in $\Theta$ with $\mathcal{L}(\hat{\boldsymbol{\theta}}) \ge \mathcal{L}(\boldsymbol{\theta})$ for every $\boldsymbol{\theta}$ in $\Theta$. An MLE need not exist, and when one exists it need not be unique. Take $n$ labels with $k$ ones, modeled as independent Bernoulli labels with one shared parameter $p$ in $[0, 1]$. For $0 < p < 1$ the logarithm of the likelihood is $\ell(p) = k \ln p + (n - k)\ln(1 - p)$, and setting its derivative to zero gives

Multiplying by $p(1 - p) > 0$ gives $k(1 - p) = (n - k)p$, so $k = np$ and $\hat{p} = k/n$. When $0 < k < n$ the function $\ell$ is strictly concave on $0 < p < 1$, so the sample proportion is the unique MLE. When $k = n$ the likelihood $p^n$ is largest at the endpoint $\hat{p} = 1$, and when $k = 0$ at $\hat{p} = 0$. For logistic regression the MLE of $(b, \mathbf{w})$ has no closed form and is computed numerically, and it does not exist as a finite vector when a hyperplane separates the two training classes.

Definition The log likelihood is $\ell(\boldsymbol{\theta}) = \ln \mathcal{L}(\boldsymbol{\theta})$, defined where $\mathcal{L}(\boldsymbol{\theta}) > 0$. With conditionally independent labels it is a sum,

and for the Bernoulli model the term of example $i$ is $y_i \ln p_i + (1 - y_i)\ln(1 - p_i)$. The negative log likelihood (NLL) is $-\ell(\boldsymbol{\theta})$. The natural logarithm is strictly increasing, so maximizing $\mathcal{L}$, maximizing $\ell$ and minimizing $-\ell$ give the same parameter values. Dividing $-\ell$ by the positive number $n$ changes its values but not its minimizers.

Definition For a label $y$ in $\{0, 1\}$ and a predicted probability $p$ of class $1$ with $0 < p < 1$, the log loss is

which equals $-\ln p$ when $y = 1$ and $-\ln(1 - p)$ when $y = 0$. For labels $y_i$ and predicted probabilities $p_i$, the binary cross-entropy, or mean log loss, is the Bernoulli NLL divided by $n$,

The name comes from the cross-entropy $H(q, p) = -q \ln p - (1 - q)\ln(1 - p)$ between two distributions on $\{0, 1\}$ with probabilities $q$ and $p$ of class $1$. Each loss $L(y_i, p_i)$ equals $H(y_i, p_i)$, with the label acting as the distribution that puts all its probability on the observed class. For a logistic model with score $z$ and $p = \sigma(z)$, the loss of one example can be written from the score as $L\bigl(y, \sigma(z)\bigr) = \ln(1 + e^{z}) - yz$, its derivative with respect to the score is $\sigma(z) - y = p - y$, and the first term can be computed as

Definition A prior distribution is a probability distribution $\pi(\boldsymbol{\theta})$ on the parameters, chosen before the data at hand are used, that describes which values are considered plausible. By Bayes’ rule the posterior satisfies $\pi(\boldsymbol{\theta} \mid \text{data}) \propto \mathcal{L}(\boldsymbol{\theta})\,\pi(\boldsymbol{\theta})$. A maximum a posteriori (MAP) estimate is a mode of the posterior, a value of $\boldsymbol{\theta}$ that maximizes $\mathcal{L}(\boldsymbol{\theta})\,\pi(\boldsymbol{\theta})$, or equivalently minimizes $-\ln \mathcal{L}(\boldsymbol{\theta}) - \ln \pi(\boldsymbol{\theta})$.

Suppose the weights $w_1, \dots, w_p$ get independent priors and the intercept is left out of the prior. Dividing the MAP objective by $n$ gives the mean NLL $\hat{R}(\boldsymbol{\theta})$ plus a penalty, up to a constant that does not depend on $\boldsymbol{\theta}$. A normal, or Gaussian, prior with mean $0$ and variance $s^2 > 0$ has $-\ln \pi(w_j) = w_j^2/(2s^2)$ plus a constant, so the MAP estimate minimizes

which is the ridge penalty. A Laplace prior with density $e^{-\lvert w_j \rvert / a}/(2a)$ and scale $a > 0$ has $-\ln \pi(w_j) = \lvert w_j \rvert / a$ plus a constant, which gives the lasso objective $\hat{R}(\boldsymbol{\theta}) + \lambda \lVert \mathbf{w} \rVert_1$ with $\lambda = 1/(na)$. Leaving the intercept unpenalized corresponds to a flat prior on $b$, which is not a probability distribution on the whole real line. The penalized fit is still well defined when its objective has a minimizer, but calling it a posterior mode then needs a separate check that the posterior is a proper distribution.