Naive Bayes Classification

Definition A generative classifier models how the features arise within each class. For classes $c = 1, \dots, K$ it specifies class probabilities $\pi_c = P(Y = c)$ and, for each class, a distribution $P(\mathbf{x} \mid Y = c)$ of the feature vector, a probability mass function for discrete features or a density for continuous ones. Together they give the joint distribution $P(\mathbf{x}, Y = c) = q_c(\mathbf{x})$, where $q_c(\mathbf{x}) = \pi_c\, P(\mathbf{x} \mid Y = c)$, and Bayes’ rule turns it into class probabilities,

for every $\mathbf{x}$ at which the denominator is positive. The products $q_c(\mathbf{x})$ are unnormalized scores. When all errors cost the same, predicting the class with the largest posterior probability minimizes the probability of an error, provided the model is correct. A discriminative classifier such as logistic regression instead models $P(Y = c \mid \mathbf{x})$ directly and says nothing about how $\mathbf{x}$ varies within a class.

Definition The class prior $\pi_c = P(Y = c)$ is the probability of class $c$ before any features are seen, with $\pi_c \ge 0$ and $\pi_1 + \cdots + \pi_K = 1$. If $n_c$ of the $n$ training examples belong to class $c$, the maximum likelihood estimate is $\hat{\pi}_c = n_c/n$. That estimate suits prediction only when the training examples were sampled with the class proportions of the population where the classifier will be used. When the classes were collected in fixed numbers or rebalanced, the priors should be set to the deployment proportions instead. Changing the priors multiplies the posterior odds of class $c$ against class $k$ by the same factor as their prior odds and leaves the likelihoods $P(\mathbf{x} \mid Y = c)$ untouched.

Definition The naive Bayes assumption states that the features are conditionally independent given the class. For every class $c$ and every feature vector $\mathbf{x} = (x_1, \dots, x_p)$,

It is a modeling assumption, not a fact about the data, and it says nothing about independence of the features when the classes are pooled. Its value is economy. For $p$ binary features a general distribution within one class needs $2^p - 1$ probabilities, while the assumption needs only $p$, one for each feature.

Definition For a binary feature $j$ and a class $c$ with $n_c$ training examples, of which $r_{cj}$ have $x_j = 1$, the smoothed estimate of $\theta_{cj} = P(X_j = 1 \mid Y = c)$ is

with a smoothing constant $\alpha > 0$. Laplace smoothing is the choice $\alpha = 1$, which adds one imagined example with the feature present and one with it absent to every class. It is the posterior mean of $\theta_{cj}$ under a uniform prior. With $\alpha = 0$ the formula is the maximum likelihood estimate $r_{cj}/n_c$, which equals $0$ when the feature never appears in the class and $1$ when it always does. For every $\alpha > 0$ the estimate lies strictly between $0$ and $1$. The constant is a modeling choice, fixed in advance or tuned on validation data, and the counts come from the training data only.

Definition The Bernoulli naive Bayes classifier works with binary features $x_j$ in $\{0, 1\}$ for $j = 1, \dots, p$. It combines class priors $\pi_c$ with the probabilities $\theta_{cj} = P(X_j = 1 \mid Y = c)$ under the naive Bayes assumption,

so a present feature contributes $\theta_{cj}$ and an absent one contributes $1 - \theta_{cj}$. It is fitted with $\hat{\pi}_c = n_c/n$, or with deployment priors, and smoothed estimates $\hat{\theta}_{cj}$. It predicts the class with the largest score $\hat{\pi}_c\, \hat{P}(\mathbf{x} \mid Y = c)$, which is the class with the largest estimated posterior probability, and ties are broken by a fixed rule. Unequal error costs call for a different decision rule applied to the posterior probabilities. Multinomial naive Bayes is a different model, built for word counts, in which the likelihood comes from how often each vocabulary word occurs, so the feature representation must be stated.

Definition The log-probability score of class $c$ for a feature vector $\mathbf{x}$ is the natural logarithm of its unnormalized score. In the Bernoulli naive Bayes model, with all $\theta_{cj}$ strictly between $0$ and $1$, the present features add $\ln \theta_{cj}$ and the absent ones add $\ln(1 - \theta_{cj})$,

The logarithm is strictly increasing, so the class with the largest $\ell_c(\mathbf{x})$ is the class with the largest score. To recover probabilities, subtract the largest log score $m = \max_k \ell_k(\mathbf{x})$ before exponentiating,

The common factor $e^{-m}$ cancels, so the result is exactly the normalized posterior, and the largest term in the denominator equals $1$, so the division cannot become $0/0$.