A repair service receives short messages and wants every fault report to reach the repair queue while scheduling requests go to the booking desk. A classifier on word counts can learn this from labeled examples, but a high score on held-out messages means little if copies of those messages were in training, if the stored text contains a tag written after the routing decision, or if the model learned where messages came from instead of what they say. This lesson builds the whole pipeline on constructed messages, compares two classifiers on validation data, scores the chosen one once on test messages, and audits what that score can and cannot support.
Key terms
Text classification
Definition Text classification assigns each document one of predefined classes. A classifier is learned from training documents with known labels and then applied to new documents. A complete specification states the classes and which one is positive, the moment at which the text is available for prediction, where the labels come from, the tokenization and representation, the classifier, the decision rule, and the roles of the data, with training documents for fitting, validation documents for choosing between candidates and test documents for one final estimate. The documents must be ones that may be used for this purpose, and personal or sensitive content has to be handled according to the permissions under which it was collected.
In plain words The classifier is only one step. What the classes mean, which text exists when the decision is made and how the data were split decide what an accuracy figure can support. A classifier that sorts the stored messages well can still fail on the messages it will actually meet.
Example A constructed project separates fault reports, label 1, from scheduling requests, label 0. Twelve two-word training messages such as “motor stopped” and “book visit”, six validation messages and eight test messages were written for this lesson, half of each set fault reports. Both candidate classifiers of the next two cards classify all six validation messages correctly with the rule “predict a fault when ”, so accuracy cannot choose between them. The declared criterion is the mean validation cross-entropy, about 0.1690 for naive Bayes and 0.1212 for logistic regression, so logistic regression is chosen. Scored once on the test messages, it classifies all eight correctly with mean cross-entropy about 0.1462. Eight constructed messages show that the workflow runs as intended. They say almost nothing about accuracy on real messages from real customers.
Multinomial naive Bayes
Definition Multinomial naive Bayes models a document by its count vector over a vocabulary of terms, with tokens in the vocabulary. Given the class and the length , it assumes that the tokens are drawn independently from one class-specific distribution over the vocabulary, with every and , so that
It also assumes that the length does not depend on the class. The multinomial coefficient is then the same for every class and cancels, and the posterior probability is proportional to with the log score
where is the class prior. Let be the total count of term in the training documents of class and the number of their tokens. The smoothed estimate is
with a constant , and is Laplace smoothing. The denominator counts tokens over the whole vocabulary, unlike the Bernoulli model of naive Bayes classification, which counts documents. Tokens outside the vocabulary are ignored.
In plain words Each class gets its own word distribution, and a message is scored by how probable its tokens are under each one, together with the class prior. A word that occurs twice contributes its log probability twice. The model ignores word order and treats the tokens as independent given the class, which is false for real language, yet its scores often separate the classes well. Its probabilities can be too extreme when words repeat or travel together, so they need a calibration check before anyone reads them as probabilities.
Example Each class of training messages has tokens and the vocabulary has terms, so with every denominator is . In the fault class motor occurs twice and fault once, so their estimates are and . Neither word occurs among the scheduling requests, so both estimates there are . The priors are equal, so for the message “motor fault” the posterior odds of a fault are
and the model’s probability of a fault is . The message “motor booking” gets odds exactly 1, because booking occurs twice among the scheduling requests and never among the faults, which cancels the factor 3 from motor. Its probability is exactly 0.5, and the declared rule sends this tie to the fault class.
Regularized logistic regression
Definition Regularized logistic regression models with the logistic sigmoid . For training count vectors , the rows of the count matrix , with labels in and a penalty weight , it chooses and to minimize
where and . Each is the binary cross-entropy of the probability , and the intercept is not penalized. With collected in the vector , the gradient is
The objective is convex. When and the training labels include both classes, it has exactly one minimizer. When and a hyperplane strictly separates the training classes, it has none, because multiplying a separating pair by ever larger factors drives the loss toward 0 without reaching it.
In plain words Logistic regression does not model how words are generated. It fits one weight per vocabulary term so that the weighted sum of counts separates the classes in the training data, and the penalty keeps the weights from growing without bound. Because the weights are fitted together, two words that always appear together split one combined weight between them, while naive Bayes counts the evidence of each word in full. A weight measures association within this fitted model and these data, not the effect of writing a word.
Example On the twelve training messages with , 1000 full-batch gradient steps of size 0.5 from zero weights leave a gradient norm of about , so the weights below are the minimizer to the digits shown. Every word that occurs once among the fault messages gets about 0.7643, the words motor, leaking and broken, which occur twice, get about 1.5286, the scheduling words get the same values with negative sign, and the intercept is 0. For “motor fault” the score is , and , against 0.8571 from naive Bayes. The training classes share no words, so a hyperplane strictly separates them, and without the penalty no minimizer would exist.
Sparse pipeline
Definition A sparse text pipeline is the ordered sequence of steps that maps raw text to a decision, here tokenization, the vocabulary, optional weights such as IDF or a feature selection, the classifier and the decision threshold, with the features kept in sparse form between the steps. Every step that learns from data is fitted on the training documents only, all steps as one unit, and the fitted pipeline is then applied unchanged to validation, test and new documents. Cross-validation refits the whole pipeline inside each training fold. The saved model is the whole pipeline, because a coefficient refers to a column, and a column means a term only through the tokenization rule and the vocabulary order.
In plain words Fitting the steps together and on training data only keeps every evaluation honest about what the model knew. Saving only the classifier’s weights would separate them from the word list that gives them meaning. A sparse pipeline also avoids steps that fill the matrix with nonzero values, such as centering the columns.
Example The vocabulary of the twelve training messages has 18 terms, and not is not among them. The messages “motor broken” and “motor not broken” therefore give the same count vector, with 1 for motor and 1 for broken, and every classifier downstream must treat them alike. Naive Bayes gives both the fault probability 0.9 and logistic regression gives both about 0.9551. No classifier can restore the negation after the representation has erased it. Adding not to the vocabulary would record its presence but not what it negates, while bigrams such as not broken or a sequence model might do better. Any such change is a new candidate, to be compared on new development messages and never adjusted on the eight test messages.
Text leakage
Definition Text leakage is information that reaches the fitting or evaluation of a text model although the model would not have it, or would not have it in that form, when it is used. Three routes are common. Text written after the decision being predicted, such as a resolution note, a reply or a tag added after routing, ends up among the inputs. Copies and near copies, such as forwarded messages, templates and messages from one thread or author, fall on both sides of a split, although the model will meet new authors or later messages. Preprocessing such as the vocabulary, IDF weights or a feature selection is fitted on documents that are later used for evaluation. The inputs must be restricted to text that exists at the moment of the decision, and the split must follow the deployment question, for example a group split by customer or thread when the model will meet new customers, or a chronological split when it will predict later messages from the same sources.
In plain words A text model can look excellent for the wrong reason. If the answer is written somewhere in the input, or if a test message is nearly a copy of a training message, the evaluation measures recognition of stored messages instead of prediction for new ones. The repair is to rebuild the data and the split, not to adjust the model.
Example Suppose each stored fault report also carries the technician’s closing note, such as “replaced valve seal”, while scheduling requests carry none. A model fitted on message plus note can classify nearly perfectly from words like replaced, yet at routing time no note exists, so that accuracy describes no situation the model will face. Likewise, if a customer forwards “pump fault” and the copy “fw pump fault” lands in training while the original lands in test, the test message differs from a training message only by the token fw, and a correct answer on it shows little.
Collection artifact
Definition A collection artifact is a property of how the documents were gathered, stored or labeled, rather than of the content the labels describe, that is associated with the label in the collected data. Examples are signatures or footers added by one channel, the templates of one source, a queue name inserted after routing, a file name that encodes the class, and differences in time period or formatting between the classes. A classifier can reach high accuracy by learning an artifact, and that accuracy lasts only as long as the association does. When the artifact is produced by the labeling or routing itself, it is also text leakage.
In plain words The model learns whatever separates the classes in the data it receives, not what the people who collected the data meant. If fault reports and scheduling requests arrived through different channels, the model may learn the channel. When a new channel appears or an old one changes, the accuracy goes with it.
Example Suppose that in a constructed collection every fault report came from the technicians’ app, which appends the word technician, and no scheduling request did. Refitting naive Bayes with gives a vocabulary of 19 terms and 18 fault tokens, so technician gets the estimate among faults and among scheduling requests, a likelihood ratio of . The scheduling request “change time” gets a fault probability of about 0.149. Sent from the app as “change time technician”, the same request gets about 0.507 and is routed to repairs. The words of the request did not change, only its channel did.
Check yourself
- In a different two-class text problem the vocabulary has 5 terms, the training documents of class 1 contain 15 tokens in total, and the term urgent occurs twice among them. Estimate the probability of urgent in class 1 under multinomial naive Bayes with . Why would dividing by the number of class 1 documents be a mistake?
Show the answer
The estimate is
The multinomial model describes the probability that one token of a class 1 document is urgent. Its denominator therefore counts all tokens of the class plus for each of the 5 terms, which makes the five estimates of class 1 add to . Dividing by the number of documents would estimate how often a document contains the word, the quantity of the Bernoulli model, and the results would not form a distribution over tokens.
- The chosen logistic regression classifies all eight test messages correctly. Before the test labels or predictions were looked at, the messages were grouped by vocabulary coverage. Seven use only vocabulary terms, four fault reports and three scheduling requests. The eighth, the scheduling request “no fault book visit”, contains the unknown token no. Give the accuracy and the fault recall of each group, and say what the second group shows.
Show the answer
The fully covered group has accuracy and fault recall . The second group has accuracy . Its fault recall is the number of detected faults divided by the number of true faults, , which is undefined, not 0 and not 1. Its fault precision is undefined too, because the model predicted no faults in it. A group with no messages at all, such as empty texts, gives no evidence whatsoever.
Both nonempty groups show perfect accuracy, but their denominators are 7 and 1, and one correct prediction shows almost nothing. The model gives this message a fault probability of about 0.178 because book and visit outweigh fault, a correct answer reached by counting words, not by reading no. The only partly covered message is also the longest and the only negated one, so the audit cannot separate the effects of coverage, length and negation. Because the groups were fixed from the text and the vocabulary before the outcomes were seen, they were not chosen to make the results look good. A report should show each group with its counts.
- A team reports 98 percent test accuracy for routing real support messages, constructed here as a scenario. They split 20,000 messages at random by row. Many customers forward the same message several times, every stored message carries the queue name that the routing desk added to its header, and the largest weights belong to the signatures of a few customers. What is wrong, and what split and error review would be defensible?
Show the answer
Three problems inflate the score. The queue name is written after the decision that the model is supposed to make, so it is text leakage and has to be removed from the inputs. The forwarded copies fall on both sides of a random row split, so many test messages are near copies of training messages. The signatures are collection artifacts that identify customers, so the model may route by sender rather than by content.
A defensible plan keeps only the text that exists when a message arrives and splits by customer or by thread, so that every copy stays in one data role, or by time if the model will route future messages from the same customers. The whole pipeline, from tokenization and vocabulary to weights and classifier, is refitted on the training groups, the candidates and the threshold are chosen on validation groups, and the test groups are scored once, next to a baseline such as always predicting the most common class.
The error review reports precision and recall per class and per declared group with their denominators, and inspects misrouted messages for negation, abbreviations, several requests in one message, unknown tokens and different writing styles. It also checks whether the largest weights belong to artifacts such as signatures. The threshold should reflect the different costs of a missed repair and an unnecessary review, and the real messages may be used only with permission and with sensitive content protected.