wtf( )unctionsystem design, drawn
← all problemsML EngineeringMedium

The model that was reviewed was not the model that shipped

Two environments, one repository. Staging runs the training pipeline, an evaluation scores the result, a reviewer approves it. Production runs the same pipeline, from the same commit, on its own schedule — and serves whatever that run produced.

Same code and same query is not the same model. The two runs read the warehouse at different moments, initialise differently, and land on different weights. Nothing in the system claims otherwise, because nothing in the system is comparing them.

Tap the components in the order one set of weights travels, from where it is produced to where it serves.
!The model reviewed on Tuesday scored 0.91. The one answering requests on Wednesday scores 0.86, from the same commit, and nobody can say why.

Outside every boundary: Model registry (immutable versions), Analyst notebook (where it started), Production endpoint (serves v47), Training job (runs once), Staging endpoint (serves v47), Production trainer (runs again nightly; FAILED: a second artifact), Evaluation (scores that build) Connections: Training job publishes to Model registry — writes the weights Model registry calls Evaluation — reads v47 back Evaluation calls Staging endpoint — if it clears Staging endpoint calls Production endpoint — the same v47 Analyst notebook controls Training job — the code Production trainer must NOT reach Production endpoint — what happens today

Tap the components in the order a request travels them.