Design and Defend a Machine Learning Study

Example The corrected study predicts the setup duration of a batch, in minutes, before setup begins, from the declared batch size, for new machines from a simulated population. Each machine contributes four setups, and 36 training, 12 validation and 16 final machines give 144, 48 and 64 rows. The baseline is the training median duration, about 29.27 minutes. The candidates are ridge-penalized polynomial regressions in batch size of degree 1 and 2 with $\lambda = 0.01$ and of degree 9 with $\lambda = 10^{-8}$, each minimizing the mean squared training error plus $\lambda$ times the sum of squared slopes, with an unpenalized intercept and with every standardization fitted on training rows only. Selection uses the validation mean over machines of each machine’s mean absolute error (MAE), and the selected fit is frozen as trained on the 36 training machines. The data are simulated, so no personal or confidential records are involved. A real study would record permissions, licenses, purpose, retention and access, and predictive convenience alone would not justify collecting data about individual workers or using a model to assess them.

Example The flawed study has three setup events with durations 14, 18 and 24 minutes, and its inputs include the recorded duration. Fitting $\hat{y} = b + w \cdot (\text{recorded duration})$ by least squares to the three training rows gives $b = 0$ and $w = 1$, because that line fits every row with zero error and, with three distinct durations, the fit is unique. The validation predictions copy the observed durations, so the MAE is zero apart from floating-point rounding, about $4 \times 10^{-15}$ minutes. The corrected protocol keeps only batch size, which is known before setup, and excludes recorded duration, completion timestamps and any other post-outcome field. Machine identifiers keep each machine’s rows together and serve to audit the split, but they are never inputs, since a new machine’s identifier carries nothing the model could have learned.

Example The simulation draws batch sizes $x$ uniformly between 1 and 8 size units and generates

with a machine offset $U_g \sim N(0, 2.5^2)$ shared by all rows of machine $g$ and independent noise $\varepsilon_{gi} \sim N(0, 0.4^2)$. The models never see the offsets or these coefficients. The final cohort therefore holds 16 independent machines, not 64 independent rows. The selected model’s final equal-machine MAE is 2.12 minutes. Resampling the 16 machines 2000 times with their rows gives an approximate 95 percent percentile interval from 1.48 to 2.77 minutes, while resampling the 64 rows as if independent gives a narrower 1.80 to 2.45 minutes. In the flawed study each of the three events was copied into two rows, one in training and one in validation, so the two splits contained exactly the same events.

Example The validation equal-machine MAEs were 13.45 minutes for the median baseline, 3.00 for the linear model, 1.68 for the quadratic model and 1.80 for the degree-9 model. The quadratic model was selected, and the squared term is a justified improvement over the linear model under the declared rule. The degree-9 model is the failed alternative, since its extra flexibility left the validation error about 0.12 minutes worse than the quadratic one. A gap of that size on 12 validation machines does not show that quadratic models are better in general or that extra polynomial terms always hurt, and the report keeps the degree-9 result with that caveat. Once the quadratic fit is frozen, its validation score has done its job of selection and is not the estimate used to defend the final model.

Example The flawed study is fully reproducible, since rerunning it returns zero validation error every time, yet its claim of reliable performance on new machines fails both the timing check and the independence check. The corrected study lets a reviewer rerun every number. On the 16 final machines the selected quadratic model has equal-machine MAE 2.12 minutes against 13.47 for the training-median baseline, an observed improvement of 11.35 minutes with whole-machine bootstrap interval from 9.31 to 13.59 minutes, computed from each machine’s paired baseline and model errors. For the first final machine the four residuals, prediction minus observation, are about $-0.096$, $0.782$, $1.926$ and $0.058$ minutes, so its MAE is $2.862/4 \approx 0.716$ minutes, and the mean of the 16 machine values is the reported 2.12. The root mean squared error is 2.52 minutes and the signed mean error is $+1.08$ minutes, an average overprediction that the MAE alone cannot show.

Example A declared stress scenario leaves every final input unchanged and adds 5 minutes to every observed duration, standing for an extra setup step that was absent during development. The predictions do not change, so every residual falls by exactly 5 minutes, and the signed mean error moves from $+1.08$ to $1.081 - 5 \approx -3.92$ minutes while the equal-machine MAE rises from 2.12 to 3.92 minutes. A monitor that watches only the inputs would see nothing, which is why delayed checks of the outcomes matter. The concise claim reads as follows. On 16 new simulated machines, a quadratic model chosen on validation data reduced the average machine absolute error from 13.47 to 2.12 minutes relative to a constant training-median baseline, the result depends on the simulated process and the fitted pipeline, and a 5-minute shift in setup times made the error worse. Real use would need real data with permission for that use, evaluation on the intended machines and a plan for changed outcomes, and this study supports no scheduling decision and no assessment of workers.