Every edit rewrote a hundred rows, and the bill arrived
The product's central table is a tree of very small records, and one user action changes a large number of them at once. Analytics, search and the AI features all need that data within minutes.
The current pipeline reloads the changed subtree into the warehouse on every edit. It is correct and it is ruinous: the warehouse is priced and built for data that arrives, and this is data that changes.
- R1The workload is update-heavy rather than append-heavy: an existing record must be replaceable in place without rewriting everything stored beside it. Several downstream systems then read that landing area directly instead of each going back to the operational database.
- R2The transformation walks a tree of records and denormalises it. That is awkward to express as queries, so the thing doing it must be able to run general code over the data rather than only ask questions of it.
- R3Reporting is still insert-heavy and still wants to be asked questions in the usual way. It keeps its existing home, fed from the landing area rather than from the operational database — replacing it would cost money and buy nothing.
Outside every boundary: Change log (one topic per table), Feature store, Operational database (sharded, the truth), Search and vectors, an empty slot for the where a changed record is replaced, tier 1, an empty slot for the walks the tree, in real code, tier 1, an empty slot for the reporting's existing home, tier 2 Connections: Operational database publishes to Change log Change log publishes to where a changed record is replaced where a changed record is replaced holds an open connection to walks the tree, in real code — read and rewrite in place where a changed record is replaced publishes to Search and vectors where a changed record is replaced publishes to Feature store where a changed record is replaced publishes to reporting's existing home