Python Arrays and Tables for Machine Learning, machine learning lesson

Six constructed parcel records list a weight in kilograms and a route distance in kilometers, both known at booking, and a delivery duration in minutes that is recorded later. A few lines of Python turn this table into arrays and compute averages and predictions from it. The same few lines can also pair one parcel’s inputs with another parcel’s duration and run without any error. This lesson shows how pandas tables and NumPy arrays hold features and targets, what shapes and axes mean, and how to check that each operation computes the quantity you intend.

Key terms

pandas DataFrame

Definition A pandas DataFrame is a two-dimensional table whose columns have names and may hold different data types, and whose rows carry labels called the index. Selecting one column with df["name"] gives a Series, a one-dimensional labeled column, and selecting a list of columns with df[["a", "b"]] gives a DataFrame. The accessor .loc selects rows by label and .iloc selects them by integer position. The method .to_numpy() returns the values as a NumPy array and drops the column names and the index.

In plain words A DataFrame keeps names on the data, so the weights are found under their column name rather than as column 0. Sort, filter and join the whole table while the identifiers are still attached, and split it into features and target only at the end. A join can also change the rows, since a repeated key multiplies rows and an unmatched key creates missing values. A row’s label and its position agree when a table is first built with the default index, but not after sorting.

Example The six parcels are below.

Parcel	Weight (kg)	Distance (km)	Duration (min)
P1	1	4	19
P2	2	8	28
P3	3	6	25
P4	4	10	34
P5	5	12	39
P6	6	14	44

The code builds the table, sorts it by distance and shows that a label and a position can point to different rows.

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "id": ["P1", "P2", "P3", "P4", "P5", "P6"],
    "weight_kg": [1, 2, 3, 4, 5, 6],
    "distance_km": [4, 8, 6, 10, 12, 14],
    "duration_min": [19, 28, 25, 34, 39, 44],
})
by_distance = df.sort_values("distance_km")
by_distance["id"].tolist()   # ['P1', 'P3', 'P2', 'P4', 'P5', 'P6']
by_distance.loc[2, "id"]     # 'P3', the row whose label is 2
by_distance["id"].iloc[2]    # 'P2', the row in position 2

After sorting, the label 2 still belongs to P3, the third parcel of the original table, while position 2 now holds P2.

NumPy array and shape

Definition A NumPy array is a grid of values of a single data type, together with a shape, the tuple of its lengths along each dimension. The number of dimensions is the length of the shape. A feature matrix with $n$ observations and $p$ features is stored as a two-dimensional array of shape (n, p), and a target with $n$ entries usually as a one-dimensional array of shape (n,). A one-dimensional array is neither a row nor a column, and the shapes (n,), (n, 1) and (1, n) are three different arrangements of the same $n$ values.

In plain words The shape says how many entries lie along each direction, and operations use it to decide how entries are combined. Before computing anything, check that the feature array has one row per target entry. Equal lengths are necessary but not sufficient, since they cannot show that row $i$ and target $i$ describe the same parcel.

Example Selecting the two feature columns in a fixed order, and the target column, gives the arrays a model would use.

X = df[["weight_kg", "distance_km"]].to_numpy(dtype=float)
y = df["duration_min"].to_numpy(dtype=float)
X.shape                # (6, 2)
y.shape                # (6,)
y.reshape(6, 1).shape  # (6, 1)

The first length in the shape of X counts the six parcels and the second counts the two features, so $n = 6$ and $p = 2$ . The double brackets keep a table, and the order inside them fixes which column of X means weight and which means distance. That order has to be the same whenever the model is used.

Indexing and slicing

Definition Indexing selects entries of an array by position, counting from 0 along each axis. For a two-dimensional array, X[i, j] is the entry in row i and column j, which in mathematical notation, counting from 1, is $x_{i+1,\,j+1}$ . A slice a:b selects the positions $a, a + 1, \dots, b - 1$ , a bare colon selects every position along its axis, and a negative index counts from the end. An integer index removes its axis from the result, while a slice keeps the axis, even when it has length 1. A list of integers selects those positions in the order given.

In plain words Python counts from zero, so the first parcel is row 0. Whether you index with a single number or slice with a range decides whether the result keeps a dimension, and many model-fitting functions expect a two-dimensional feature array even when there is only one feature.

Example For the parcel array,

X[0, 1]       # 4.0, the distance of P1
X[:, 0]       # array([1., 2., 3., 4., 5., 6.]), shape (6,)
X[:, 0:1]     # the same weights, shape (6, 1)
X[-1]         # array([ 6., 14.]), the row of P6
X[[2, 0, 1]]  # the rows of P3, P1 and P2, shape (3, 2)

The entry X[0, 1] is $x_{12} = 4$ kilometers. Selecting rows with a list is safe only when the same positions are applied to the target as well, so that y[[2, 0, 1]] gives the matching durations 25, 19 and 28 minutes.

Array axis

Definition Axis $k$ of an array is its dimension number $k$ , counted from 0, and its length is entry $k$ of the shape. In a two-dimensional array, axis 0 runs down the rows and axis 1 runs across the columns. A reduction such as sum, mean or max with axis=k combines the entries along axis $k$ and, unless keepdims=True is passed, removes that axis from the shape. For a feature array of shape (n, p), X.mean(axis=0) has shape (p,) and holds the column means

\bar{x}_j = \frac{1}{n} \sum_{i=1}^{n} x_{ij}

for $j = 1, \dots, p$ , while X.mean(axis=1) has shape (n,) and holds one mean per row.

In plain words The axis you name is the one that disappears. For a feature array, averaging along axis 0 averages over observations and gives one value per feature, in that feature’s unit. Averaging along axis 1 mixes different features in one sum, which is meaningless when, as here, they have different units.

Example The weights add to 21 kilograms and the distances to 54 kilometers, so

\begin{aligned} \bar{x}_1 &= 21/6 = 3.5 \text{ kg}, \\[0.4em] \bar{x}_2 &= 54/6 = 9 \text{ km}, \end{aligned}

and X.mean(axis=0) returns array([3.5, 9. ]). The call X.mean(axis=1) also runs without error and returns six numbers. The first of them is 2.5, the average of 1 kilogram and 4 kilometers, which is not a quantity of anything.

Elementwise multiplication and broadcasting

Definition For two arrays of the same shape, A * B multiplies the entries in matching positions and returns an array of that shape, with entries $a_{ij} b_{ij}$ in the two-dimensional case. This elementwise product differs from the matrix product A @ B. For a feature array $X$ of shape (n, p) and a one-dimensional array w of shape (p,), the product X @ w has shape (n,) and entries

(X\mathbf{w})_i = \sum_{j=1}^{p} x_{ij} w_j .

Broadcasting lets elementwise operations combine arrays of different shapes. NumPy lines up the two shapes from the right, treats missing leading dimensions as length 1, and requires each aligned pair of lengths to be equal or to include a 1. A dimension of length 1 is then repeated to match the other length, and if some pair fails the rule, NumPy raises an error.

In plain words The asterisk multiplies matching entries and keeps every product. The at sign multiplies and then adds along the shared feature dimension, which is the weighted sum a linear prediction needs. Broadcasting saves writing loops, but it also turns some shape mistakes into wrong answers instead of errors.

Example With $x_1$ the weight and $x_2$ the distance, take the rule

\hat{y} = 10 + 1 \cdot x_1 + 2 \cdot x_2

in minutes, with coefficients in minutes per kilogram and minutes per kilometer, stored in the saved feature order.

w = np.array([1.0, 2.0])   # minutes per kg, minutes per km
X * w                      # shape (6, 2), first row [1., 8.]
X @ w + 10                 # array([19., 28., 25., 34., 39., 44.])

For P1 the elementwise product gives the separate contributions $1 \cdot 1 = 1$ and $4 \cdot 2 = 8$ minutes, and the matrix product adds them to 9 before the intercept brings the prediction to 19 minutes. In X * w the shape (2,) is aligned with the last length of (6, 2) and repeated down all six rows. A per-parcel array of shape (6,) cannot be combined with X this way, because 6 is aligned with 2, while its (6, 1) version is repeated across the two columns. A coefficient array of length 3 fails in both products, since 3 does not match the 2 features. The six predictions equal the recorded durations exactly because the constructed data were built from this rule. The rule was chosen, not fitted, and says nothing about real deliveries.

Notebook

Definition A notebook, such as a Jupyter notebook, is a document of text cells and code cells whose code is run by a kernel, a running Python process. The kernel keeps every variable created by the cells that have run, in the order they were actually run, which need not be their order on the page, and it keeps them until it is restarted. A notebook’s results can be trusted to follow from the code on the page only once restarting the kernel and running all cells from top to bottom gives them again. Reproducing them elsewhere also needs the same data and package versions, and fixed random seeds where randomness is used.

In plain words The kernel remembers. A cell edited but not rerun, a cell run twice, or a later cell run before an earlier one can leave variables that no longer match what the page shows. A plain Python script run from the start does not carry this hidden state.

Example Suppose one cell converts distances to meters with

df["distance_km"] = df["distance_km"] * 1000

and is accidentally run twice. The page shows one conversion, but the kernel holds distances multiplied by a million, so the mean distance comes out as 9,000,000 instead of 9,000 meters. Restarting the kernel and running all cells in order removes the error. Writing the result to a new column, as in df["distance_m"] = df["distance_km"] * 1000, makes the cell safe to rerun and keeps the column name true to its unit.

Check yourself

The rows X_sel = X[[2, 0, 1]] belong to P3, P1 and P2. A learner wants one mean per feature and writes X_sel.mean(axis=1). What does that return, what is the right call, and what are the correct means with their units?

Show the answer

The selected rows hold the features $(3, 6)$ , $(1, 4)$ and $(2, 8)$ . The call with axis=1 averages within each row and returns three numbers, 4.5, 2.5 and 5, each mixing kilograms with kilometers.

The right call is X_sel.mean(axis=0), which averages down the rows and returns two numbers. The mean weight is $(3 + 1 + 2)/3 = 2$ kilograms and the mean distance is $(6 + 4 + 8)/3 = 6$ kilometers.

A learner sorts the features by distance with order = np.argsort(X[:, 1]) and X_sorted = X[order], but keeps y as it was. Which rows are now paired with the wrong duration, does NumPy report an error, and how should the code be fixed?

Show the answer

The positions in order are 0, 2, 1, 3, 4 and 5, so row 1 of X_sorted now holds P3, with features $(3, 6)$ , while y[1] is still 28, the duration of P2. Row 2 holds P2, with features $(2, 8)$ , but is paired with 25, the duration of P3. The other four rows happen to stay matched.

No error appears, because the shapes (6, 2) and (6,) still agree. The rule $\hat{y} = 10 + x_1 + 2x_2$ , which reproduces every true duration, now seems to miss by 3 minutes on both swapped rows, a false error created only by the misalignment.

The fix is to reorder both arrays with the same positions, X[order] and y[order], or better, to sort the DataFrame and only then select the feature and target columns.

The predictions for the selected rows, pred = X_sel @ w + 10, have shape (3,), and the matching targets are y_sel = y[[2, 0, 1]]. A learner computes residuals with pred - y_sel.reshape(3, 1). What shape does the result have, what is its first row, and what is the fix?

Show the answer

The shapes (3,) and (3, 1) broadcast to (3, 3). Entry $(i, j)$ of the result is prediction $j$ minus target $i$ , so the result compares every prediction with every target instead of giving three residuals.

The predictions are 25, 19 and 28 minutes, and so are the targets. The first row is therefore $25 - 25$ , $19 - 25$ and $28 - 25$ , that is 0, $-6$ and 3.

The fix is to give both arrays the same shape, as in pred - y_sel with both of shape (3,), or with both reshaped to (3, 1). That returns the three residuals 0, 0 and 0. They are zero because the constructed data follow the chosen rule exactly, which checks the code and the row alignment but says nothing about how well the rule would predict real deliveries.

Python Arrays and Tables for Machine Learning

Key terms

pandas DataFrame

NumPy array and shape

Indexing and slicing

Array axis

Elementwise multiplication and broadcasting

Notebook

Check yourself

How this lesson connects

Builds on

Used later in