A better model on paper
The new model beats the current one on every offline metric, on a held-out split, by a comfortable margin. The plan is to swap it in on Thursday.
Offline evaluation says how a model scores on data someone assembled. It says nothing about latency under real load, nothing about the feature values that actually arrive at request time, and nothing about the inputs the held-out set never happened to contain.
Boundaries, outermost first: Shadow path: Candidate model, an empty slot for the what the candidate gets, an empty slot for the what happens to its output Outside every boundary: Live requests, Comparison (both, same inputs), Response, Current model (serves the answer) Connections: Live requests calls Current model (step 1) Current model calls Response (step 2) Live requests calls what the candidate gets — a copy (step 3) what the candidate gets calls Candidate model (step 4) Candidate model calls what happens to its output (step 5) what happens to its output calls Comparison (step 6) Candidate model must NOT reach Response — never returned