Exploratory Data Analysis and Data Quality

Definition Exploratory data analysis is the examination of a data set through summaries, tables and plots before a model is chosen, to learn how the values are distributed, how variables relate and where the data have quality problems. Checks of structure, such as the schema (what one row represents, which columns exist, their types, units and allowed values), identifiers, timestamps and permissions, can be run on every record, because they do not use outcomes to tune any choice. Exploration of relationships with the target that will guide modeling choices, such as which features or versions of a feature to use, belongs to the training data, since looking at test outcomes and acting on them makes the test set part of development. A pattern found in exploration describes these data and is not by itself evidence of cause.

Definition Data quality is the fitness of a data set for a stated task. Its main parts are accuracy, meaning that each value measures the quantity it claims to in its stated unit, consistency of units, codes and formats across records, completeness, meaning which values are missing and how many, uniqueness, meaning that each real event is recorded once, timeliness, meaning that each value is available at the prediction time, and representativeness of the deployment population. Repairs keep the raw values beside the derived ones. A unit conversion is recorded next to the raw reading, a copy confirmed by the source collapses to one record, a missing value stays marked as missing and never becomes zero, and an unusual value is changed only when evidence shows it is wrong. Each change goes into a log with its reason, so that someone else can retrace it.

Definition The empirical distribution of observed values $x_1, \dots, x_n$ gives weight $1/n$ to each observation, so a value observed $k$ times gets weight $k/n$. Its empirical cumulative distribution function is

the proportion of observations at or below $t$. It is a step function that rises from 0 to 1 and jumps at each observed value. A histogram groups the observations into bins on the number line and draws one bar per bin, whose height is the count in the bin or the proportion of observations in it. When bins have unequal widths, the height is the density, the proportion divided by the bin width, so that areas show proportions. The bins must not overlap and must cover every value, so the rule for values on an edge is part of the definition. This lesson uses bins that include their left edge and exclude their right edge, except that the last bin also includes its right edge.

Definition For observations $x_1, \dots, x_n$ the mean is

Sort the values as $x_{(1)} \le x_{(2)} \le \dots \le x_{(n)}$. The median is the middle sorted value $x_{((n+1)/2)}$ when $n$ is odd, and the average of the two middle values $x_{(n/2)}$ and $x_{(n/2 + 1)}$ when $n$ is even. Among all constants $c$, the mean minimizes the sum of squared deviations $\sum_{i} (x_i - c)^2$, and the median minimizes the sum of absolute deviations $\sum_{i} \lvert x_i - c \rvert$, a minimum that for even $n$ every value between the two middle values shares. Both summaries use only the observed values, so they are reported together with the number of values they rest on.

Definition For a proportion $q$ with $0 < q \le 1$, a $q$ quantile of observed values is a value $t$ with at least a fraction $q$ of the observations at or below $t$ and at least a fraction $1 - q$ at or above it. Several values can qualify, and software uses different conventions to pick one, which can disagree on small samples, so the convention must be named. This lesson uses the nearest-rank quantile

the sorted value at position $\lceil nq \rceil$, which is also the smallest observed value $t$ with $F_n(t) \ge q$. The lower quartile is $Q(0.25)$, the upper quartile is $Q(0.75)$, and the interquartile range is $\operatorname{IQR} = Q(0.75) - Q(0.25)$, the width of the middle half of the data under this convention. For odd $n$ the nearest-rank $Q(0.5)$ equals the median, but for even $n$ it is the lower of the two middle values, while the median averages them.

Definition An outlier is an observation that lies unusually far from the bulk of the data, or from the pattern the other observations follow, judged against a stated comparison. A common exploratory flag uses Tukey’s fences

and marks the values outside them. A flag calls for investigation, not removal. An outlier may be a recording error, a unit mistake, a case from outside the intended population or a genuine extreme, and only evidence about the measurement decides which. A confirmed value stays in the data. When an extreme value has a large influence on a result, a sensitivity analysis reports the result with and without it, and any exclusion is reported with its reason and the population it leaves out.