Likelihood and Maximum Likelihood Estimation

Definition A statistical model is a stated family of probability distributions for the data, one distribution for each value of a parameter $\theta$ in a stated parameter space $\Theta$. For discrete data each distribution gives the probability of every possible data set, and for continuous data it gives a joint probability density. The model includes its assumptions about how the observations relate to one another, for example that they are independent. In the running example the model says that the eight trials $X_1, \dots, X_8$ are independent and that each one is $\operatorname{Bernoulli}(p)$, with $X_i = 1$ for a success and $X_i = 0$ for a failure. The parameter space is $0 \le p \le 1$. Under this model the probabilities of the eight outcomes multiply, so a particular sequence with $k$ successes and $8 - k$ failures has probability $p^{k}(1 - p)^{8 - k}$.

Definition Fix the observed data and a statistical model with parameter $\theta$ in $\Theta$. The likelihood function assigns to each $\theta$ in $\Theta$ the probability of the observed data when the parameter equals $\theta$,

or, for continuous data, the joint density evaluated at the observed values. The data are held fixed and $\theta$ varies. When the model makes discrete observations $X_1, \dots, X_n$ independent and their observed values are $x_1, \dots, x_n$, the joint probability is a product,

and for independent continuous observations the individual densities are multiplied in the same way. Multiplying a likelihood by a positive constant that does not depend on $\theta$ gives an equivalent likelihood, with the same ratios between parameter values and the same maximizing values.

Definition For two parameter values $\theta_1$ and $\theta_0$ in $\Theta$ with $L(\theta_0) > 0$, the likelihood ratio of $\theta_1$ to $\theta_0$ is

For discrete data it says how many times as probable the observed data are under $\theta_1$ as under $\theta_0$, and for continuous data it compares the densities at the observed values. A ratio above $1$ favors $\theta_1$, a ratio below $1$ favors $\theta_0$, and a ratio of exactly $1$ means the data do not distinguish the two values. Any positive factor that does not depend on the parameter cancels from the ratio.

Definition A maximum likelihood estimate (MLE) is a parameter value $\hat{\theta}$ in $\Theta$ with $L(\hat{\theta}) \ge L(\theta)$ for every $\theta$ in $\Theta$. An MLE need not exist, and when one exists it need not be unique. For $k$ successes in $n \ge 1$ independent Bernoulli trials with parameter space $0 \le p \le 1$, the likelihood is $p^k(1 - p)^{n - k}$, with $0^0$ read as $1$, and it has exactly one maximizer, the sample proportion

This includes $k = 0$ and $k = n$, where the maximum sits at an end of the parameter space.

Definition The natural logarithm $\ln y$ of a positive number $y$ is the exponent $u$ with $e^u = y$. It is strictly increasing, and it turns products into sums, since $\ln(ab) = \ln a + \ln b$ for positive $a$ and $b$. Where the likelihood is positive, the log-likelihood is

For independent observations it is the sum of the logarithms of the individual probabilities or densities. Because the logarithm is strictly increasing, $\ell(\theta_1) > \ell(\theta_0)$ exactly when $L(\theta_1) > L(\theta_0)$. So if the likelihood is positive somewhere, maximizing $\ell$ over the values with positive likelihood gives exactly the maximizers of $L$. A constant factor $c > 0$ in the likelihood becomes an added constant $\ln c$, and a difference of log-likelihoods is the logarithm of a likelihood ratio.

Definition A maximum likelihood estimate is a boundary estimate when it lies on the edge of the parameter space rather than strictly inside it. With $k = 0$ successes in $n \ge 1$ trials and parameter space $0 \le p \le 1$, the likelihood $(1 - p)^n$ decreases as $p$ increases, so the MLE is the boundary value $\hat{p} = 0$. Whether an MLE exists here depends on the declared parameter space. If the model allows only $0 < p < 1$, the likelihood rises toward $1$ as $p$ approaches $0$ but never reaches it, and no maximum likelihood estimate exists.