Text Features and TF-IDF

Definition Tokenization is a fixed rule that splits a document, a string of characters, into a sequence of tokens, the units that will later be counted. The rule on this page first lowercases the text and then takes every maximal run of the letters a to z as one token, in order. Every other character, including digits, punctuation, apostrophes, spaces and letters with accents, only separates tokens and is dropped. No word is reduced to a stem and no common word is removed. Other rules split at whitespace, keep numbers, remove stop words such as the, reduce words to stems or cut rare words into subword pieces, and each choice defines a different representation. The rule is part of the model, and the same rule must run at training time and at prediction time.

Definition The vocabulary $V$ is the list of distinct terms that the representation keeps, in a fixed order, so that term $j$ owns column $j$ of every feature vector. On this page it is fitted on the training documents only. It contains every token that occurs in at least one training document, sorted alphabetically, and its size is $p = \lvert V \rvert$. Applying the fitted representation to validation, test or future documents never changes $V$, and a token outside $V$ is ignored, which is this page’s declared policy for unknown tokens. Rules that prune the vocabulary, for example by keeping only terms found in at least a set number of training documents, are part of the fit as well. Other policies add a column that counts unknown tokens or spell words with subword pieces, and each of them is a different representation that has to be declared.

Definition A bag-of-words representation describes a document by the number of times each vocabulary term occurs in it, regardless of where. For $n$ documents and a vocabulary of $p$ terms, the document-term matrix is the $n \times p$ matrix $C$ whose entry $c_{ij}$ is the number of tokens of document $i$ equal to term $j$. Row $i$ is the count vector of document $i$, every entry is a nonnegative integer, and the row sum is the number of the document’s tokens that lie in the vocabulary. Two documents get the same row exactly when their tokens in the vocabulary agree as a multiset, that is, the same terms with the same counts, in whatever order.

Definition The term frequency of term $j$ in document $i$ measures how prominent the term is inside that one document. On this page it is the raw count $c_{ij}$. Common alternatives are the relative frequency $c_{ij} / \sum_k c_{ik}$ for a document with at least one known token, the indicator of $c_{ij} > 0$, and $1 + \ln c_{ij}$ when $c_{ij} > 0$ with 0 otherwise. The document frequency of term $j$ is the number of training documents that contain it at least once,

where the indicator $\mathbf{1}\{\cdot\}$ equals 1 when its condition holds and 0 otherwise. When the vocabulary comes from the same $n$ training documents, every term has $1 \le \operatorname{df}_j \le n$. Like the vocabulary, the document frequencies are computed from the training documents and then kept fixed.

Definition TF-IDF, short for term frequency times inverse document frequency, is a family of weightings that multiply each term frequency by a weight that decreases as the term’s document frequency grows. This page uses raw counts and the smoothed inverse document frequency

with the natural logarithm and the $n$ training documents, so the TF-IDF matrix has entries $t_{ij} = c_{ij} \operatorname{idf}_j$. Because $\operatorname{df}_j \le n$, every weight satisfies $\operatorname{idf}_j \ge 1$, with equality exactly for a term found in every training document, which therefore keeps weight 1 instead of disappearing. An optional row normalization divides each nonzero row $\mathbf{t}_i$ by its Euclidean length $\lVert \mathbf{t}_i \rVert_2$ and leaves a zero row as the zero vector, since a zero row has no direction to keep. Other members of the family use $\ln(n / \operatorname{df}_j)$, which gives weight 0 to a term in every document, other logarithm bases or other term frequencies. The name TF-IDF alone does not fix the numbers, so the formula has to be stated with the model.

Definition A matrix is sparse when most of its entries are zero, and a sparse storage format keeps only the nonzero entries and their positions. Their number is written $\operatorname{nnz}$. Compressed sparse row (CSR) storage of an $n \times p$ matrix uses three arrays. Counting from 0, they hold the nonzero values row by row, the column index of each value, and $n + 1$ row pointers, where the entries of row $i$ occupy the positions from pointer $i$ up to but not including pointer $i + 1$. The matrix and its shape $n \times p$ stay the same, only the storage changes. Memory and the cost of a matrix-vector product grow with $\operatorname{nnz} + n$ instead of $np$. Operations that send zeros to zeros, such as multiplying columns by IDF weights or dividing rows by their lengths, keep the matrix sparse. Subtracting the column means does not, since it turns every zero of a column with a nonzero mean into a nonzero entry.