wtf( )unctionsystem design, drawn
← all problemsML EngineeringMedium

The feature that means two different things

A fraud model uses "average transaction value over the last 30 days". The training job computes it with a SQL query over the warehouse. The scoring service computes it again in application code, because it has a 20 ms budget and cannot query a warehouse.

Someone excluded refunds from one of them, months ago. Both implementations still look correct on their own, both are tested, and neither team has a reason to read the other's code.

Place the component both the training job and the scoring path must read the feature from.
Components — tap one, then tap a slot on the diagram
!Offline the model scored 0.91 AUC. In production it barely beats the rule it replaced.

Boundaries, outermost first: Feature platform: Online store (latest per entity), Offline store (history for training), an empty slot for the computed once, read twice Outside every boundary: Scoring API (20 ms budget; FAILED: recomputes its own), Training job (runs nightly), Transaction events (the raw stream) Connections: Transaction events calls computed once, read twice (step 1) computed once, read twice publishes to Offline store — materialised (step 2) computed once, read twice publishes to Online store (step 2) Offline store calls Training job — training rows (step 3) Online store calls Scoring API — at request time (step 4)

Scoring API20 ms budgetrecomputes its own
Training jobruns nightly
Online storelatest per entity
Transaction eventsthe raw stream
Offline storehistory for training