wtf( )unctionsystem design, drawn
← all problemsML EngineeringMedium

The scaler that had already seen the test set

The pipeline loads one labelled table, standardises the numeric columns, target-encodes the categorical ones, then splits off a test set and reports the score.

Every number in that report is contaminated. The scaler's mean and the encoder's category statistics were computed over rows that are supposed to be unseen, so the model was handed a summary of its own test set before it ever trained.

Order the three stages: what runs first, what learns the parameters, and what only reuses them.
Components — tap one, then tap a slot on the diagram
!Cross-validated AUC 0.94. The first week in production came in at 0.71.

Boundaries, outermost first: Training pipeline: an empty slot for the runs before any fit, an empty slot for the learns the parameters, an empty slot for the reuses them, fits none Outside every boundary: Labelled data (one table), Cross-validation (5 folds), Held-out score (what you report; FAILED: 0.94 offline, 0.71 live) Connections: Labelled data calls runs before any fit (step 1) runs before any fit controls Cross-validation — the split moves each fold runs before any fit calls learns the parameters (step 2) learns the parameters calls reuses them, fits none (step 3) reuses them, fits none calls Held-out score (step 4)

Labelled dataone table
Cross-validation5 folds
Held-out scorewhat you report0.94 offline, 0.71 live