Statistical estimation and likelihood

Definition A statistical model for data $x = (x_1, \dots, x_n)$ is a family of joint densities or probability mass functions $f_\theta$, indexed by a parameter $\theta$ in a parameter set $\Theta$. The likelihood function is $\theta \mapsto L(\theta; x) = f_\theta(x)$, the joint density at the observed data, read as a function of $\theta$ with $x$ held fixed. When the model makes the observations independent with a common density $f_\theta$, the joint density factors and

On the set of $\theta$ with $L(\theta; x) > 0$, the log-likelihood is

and because the logarithm is strictly increasing, $\ell$ and $L$ are largest at the same parameter values. Independence is what produces the product. For dependent observations, such as a time series, the joint density factors instead into conditional densities,

where $x_{1:j-1} = (x_1, \dots, x_{j-1})$. The likelihood is not a probability distribution over $\theta$. Its integral over $\Theta$ need not be $1$ or even finite, and it becomes a posterior density only after it is multiplied by a prior and normalized.

Definition A maximum likelihood estimate for data $x$ is a value $\hat\theta(x) \in \Theta$ with

The maximum likelihood estimator is the rule $x \mapsto \hat\theta(x)$, and applied to the random sample it gives a random variable $\hat\theta(X)$. A maximizer need not exist and need not be unique, so each model needs its own argument. For independent observations from $N(\mu, \sigma^2)$ with $\sigma^2 > 0$ known and $\mu \in \mathbb{R}$, the log-likelihood and its derivative are

The derivative is zero only where $\sum_j x_j - n\mu = 0$, that is, at $\mu = \bar x$. The second derivative is $-n/\sigma^2 < 0$ for every $\mu$, so $\ell$ is strictly concave and $\bar x$ is its unique global maximizer. The estimator is $\hat\mu = \bar X$. If $\sigma^2$ is unknown as well, with parameter set $\mathbb{R} \times (0, \infty)$, the likelihood is maximized at $\mu = \bar x$ and

provided this average squared residual is positive. If all observations are equal, the likelihood grows without bound as $\sigma^2 \to 0$ and no maximizer exists. Because $\mathbb{E}\bigl[\sum_j (X_j - \bar X)^2\bigr] = (n - 1)\sigma^2$, the estimator $\hat\sigma^2$ has mean $(n - 1)\sigma^2/n$, while the sample variance $s^2$, with divisor $n - 1$ and defined for $n \ge 2$, is unbiased.

Definition Let $\hat\theta_n$ be an estimator computed from the first $n$ observations of a specified sampling scheme. The sequence $(\hat\theta_n)$ is consistent for $\theta$ if, for every $\theta \in \Theta$ and every $\varepsilon > 0$,

which means that $\hat\theta_n$ converges to $\theta$ in probability. For independent observations with common mean $\mu$ and common finite variance $\sigma^2$, the sample mean has $\mathbb{E}[\bar X_n] = \mu$ and $\operatorname{Var}(\bar X_n) = \sigma^2/n$, and Chebyshev’s inequality gives

The bound tends to zero, so the sample mean is consistent. Unbiasedness, $\mathbb{E}_\theta[\hat\theta_n] = \theta$ at a fixed $n$, is a different property. A biased estimator can be consistent, as $\hat\sigma^2$ is in the normal model, and an unbiased estimator can fail to be consistent. Without independence the variance of the mean is

so the covariance terms decide whether it shrinks at all.

Definition The weak law of large numbers states that if $X_1, X_2, \dots$ are independent and identically distributed with $\mathbb{E}\lvert X_1 \rvert < \infty$ and $\mu = \mathbb{E}[X_1]$, then $\bar X_n \to \mu$ in probability. Finite variance is not needed. A proof outline truncates the observations. For a level $M > 0$ put $Y_j = X_j \mathbf{1}_{\{\lvert X_j \rvert \le M\}}$ and $\delta_M = \mathbb{E}\bigl[\lvert X_1 \rvert \mathbf{1}_{\{\lvert X_1 \rvert > M\}}\bigr]$, which tends to $0$ as $M \to \infty$ because $X_1$ is integrable. Then

Markov’s inequality gives $\mathbb{P}(\lvert \bar X_n - \bar Y_n \rvert > \varepsilon) \le \delta_M/\varepsilon$ for every $n$. The truncated variables are bounded by $M$, so Chebyshev’s inequality gives $\mathbb{P}(\lvert \bar Y_n - \mathbb{E}[Y_1] \rvert > \varepsilon) \le M^2/(n\varepsilon^2)$. The last term satisfies $\lvert \mathbb{E}[Y_1] - \mu \rvert \le \delta_M$. When $\delta_M \le \varepsilon$, these combine to

Given $\eta > 0$, first choose $M$ with $\delta_M \le \min(\varepsilon, \eta\varepsilon)$, and then every $n \ge M^2/(\eta\varepsilon^2)$ makes the probability at most $2\eta$. Since $\varepsilon$ and $\eta$ are arbitrary, this proves the law. The order matters, since the truncation level is fixed before the sample grows. The strong law, which says that $\bar X_n \to \mu$ with probability $1$, holds under the same hypotheses and is stated here without proof.

Definition The central limit theorem states that if $X_1, X_2, \dots$ are independent and identically distributed with mean $\mu$ and variance $\sigma^2$, where $0 < \sigma^2 < \infty$, then

that is, the probability that the left side is at most $z$ tends to $\Phi(z)$ for every real $z$. A proof outline uses characteristic functions. The standardized observation $Z = (X_1 - \mu)/\sigma$ has mean $0$ and variance $1$, and its characteristic function $\psi_Z(u) = \mathbb{E}[e^{iuZ}]$ satisfies

as $u \to 0$, which needs only the finite second moment. By independence the standardized sum $n^{-1/2}\sum_j Z_j$ has characteristic function $\psi_Z(u/\sqrt{n})^{n}$, and for every real $u$

the characteristic function of $N(0, 1)$. Lévy’s continuity theorem, stated here without proof, says that pointwise convergence of characteristic functions to the characteristic function of a distribution implies convergence in distribution. No moment generating function is required. With infinite variance the theorem does not apply, and other scalings and limits can occur. With a finite third absolute moment $\rho = \mathbb{E}\lvert X_1 - \mu \rvert^3$, the Berry-Esseen theorem bounds the error of the normal approximation at every $z$ by $C\rho/(\sigma^3\sqrt{n})$, with an absolute constant $C$ below $0.5$, so the quality of the approximation still depends on the size of $\rho$.

Definition The standard error of an estimator $\hat\theta$ is the standard deviation of its sampling distribution, $\operatorname{SE}(\hat\theta) = \sqrt{\operatorname{Var}_\theta(\hat\theta)}$. For the mean of $n$ independent observations with variance $\sigma^2$ it is $\operatorname{SE}(\bar X_n) = \sigma/\sqrt{n}$, and the estimate $\widehat{\operatorname{SE}} = s/\sqrt{n}$ replaces $\sigma$ by the sample standard deviation. A confidence interval at level $1 - \alpha$ is a pair of statistics $A(X) \le B(X)$ with $\mathbb{P}_\theta\bigl(A(X) \le \theta \le B(X)\bigr) = 1 - \alpha$, or at least $1 - \alpha$, for every $\theta \in \Theta$. For independent $N(\mu, \sigma^2)$ observations with $\sigma$ known, $\sqrt{n}(\bar X_n - \mu)/\sigma$ is exactly standard normal, so the interval

covers $\mu$ with probability exactly $0.95$. In the same normal model with $\sigma$ unknown and $n \ge 2$, a standard result states that $\sqrt{n}(\bar X_n - \mu)/s$ has the Student t distribution with $n - 1$ degrees of freedom, and the interval $\bar X_n \pm t_{n-1,\,0.975}\, s/\sqrt{n}$ is exact. Outside the normal model, for independent identically distributed observations with finite positive variance, the interval $\bar X_n \pm z_{0.975}\, s/\sqrt{n}$ has coverage that tends to $0.95$ as $n \to \infty$, by the central limit theorem combined with consistency of $s$, with no guarantee at a fixed $n$. The probability statement concerns the random endpoints before the data are seen. Once a sample is observed, the interval is two fixed numbers that either contain the fixed $\mu$ or not, and the frequentist calculation assigns no probability $0.95$ to $\mu$ itself.