Feature Scaling and Categorical Encoding

Definition Feature scaling replaces a numerical feature $x_j$ by

where the center $c_j$ and the scale $s_j > 0$ are numbers learned from the training values of that feature. Standardization and min-max scaling are the most common choices, and robust scaling uses the median and the interquartile range. Each such map is increasing, so the order of the values within a column stays the same. What changes is how much each feature counts when features are combined. The centers cancel in differences, so for two records $\mathbf{x}$ and $\mathbf{z}$ in $\mathbb{R}^p$ the squared Euclidean distance after scaling is

and feature $j$ is weighted by $1/s_j^2$. Models that compare records through distances, such as k-nearest neighbors, can give different answers under different scalings.

Definition Let $x_1, \dots, x_n$ be the training values of one numerical feature, with $n \ge 1$. Their mean and standard deviation are

so the standard deviation is the square root of the average squared deviation from the mean, in the units of the feature. Standardization maps a value $x$ to

using the saved training values of $\bar{x}$ and $s$. This page divides by $n$, the convention of many scaling routines. The sample standard deviation divides by $n - 1$ instead, which needs $n \ge 2$ and gives a larger value whenever $s > 0$, so a fitted scaler should record its divisor. When $s > 0$, the standardized training values have mean 0 and standard deviation 1 under the same divisor. Multiplying a feature by a constant $a > 0$, as a change of units does, multiplies its mean and standard deviation by $a$ too, so the standardized values do not change. If every training value is the same, then $s = 0$ and the division is undefined. A documented convention uses the scale 1 after centering, so every training entry becomes 0. Standardization moves and rescales a distribution without changing its shape, so it does not make data normally distributed, and a single extreme value can move both the mean and the standard deviation a great deal.

Definition Let $m$ and $M$ be the smallest and largest training values of a feature, with $M > m$. Min-max scaling maps a value $x$ to

Training values land in the interval $[0, 1]$, and a later value outside $[m, M]$ lands outside it. If $M = m$, the formula divides by zero and needs a declared convention. Clipping the results to $[0, 1]$ would be an extra modeling decision that hides how far a new value lies beyond the training range. The name normalization is used for min-max scaling, for standardization and for row normalization, which divides a whole record vector $\mathbf{x} \neq \mathbf{0}$ by its length $\lVert \mathbf{x} \rVert_2$, so a pipeline should state its formula rather than the name.

Definition A train-only fit computes every quantity a preprocessing step learns from data, such as means, standard deviations, minima and maxima, medians, category vocabularies and column orders, from the training data alone. The fitted values are saved and applied unchanged to training, validation, test and future records. In cross-validation each fold fits on its own training part. Fitting on all the data, or refitting on a held-out batch, lets held-out information into the representation and makes the input given to one record depend on which other records arrived with it.

Definition Let a categorical feature have the vocabulary $c_1, \dots, c_K$, learned from training data and stored in a fixed order. One-hot encoding represents a value by $K$ indicator columns, where column $k$ equals 1 if the value is $c_k$ and 0 otherwise, so each known category has exactly one entry equal to 1. Any two distinct known categories are then at Euclidean distance $\sqrt{2}$, so the encoding puts no order on the categories and makes no pair closer than another. An ordinal category is a categorical feature whose values have a meaningful order, such as small, medium and large. Integer codes 0, 1 and 2 keep that order, but when a model uses them as numbers they also assert equal gaps, which is an extra assumption. Integer codes for categories without an order, such as service types or colors, invent both an order and distances.

Definition An unknown category is a value of a categorical feature that does not appear in the vocabulary fitted on the training data. The encoder needs a declared rule for it. Common rules raise an error, set all the known indicator columns to 0, or set a dedicated unknown column to 1. The training data contain no record of the new value, so they hold no evidence about how it behaves, and detecting it does not supply that evidence. The vocabulary is learned inside each training fold like any fitted quantity, so a category that appears only in a fold’s held-out part is unknown there as well.