Every region computed the price, and two of them published it
A courier marketplace prices each pickup from the last minute of demand and supply inside a small map cell. The computation holds a live window for every active cell — far too much state to copy between regions, so every region runs the whole job all the time and they all reach roughly the same answer.
That is the design working as intended. The incident is that for four minutes two regions both believed they were the one that publishes.
- R1Two regions must never publish different multipliers for the same cell at the same time. Exactly one region is the writer at any moment, and which one it is has to be something that can be taken away — a region that has stopped responding must lose the right to write without being asked to give it up.
- R2If an entire region becomes unreachable, pricing must keep being produced and served everywhere else. Whatever holds the current multiplier must survive losing a region rather than merely being backed up somewhere.
- R3Pricing a customer's request must be one fast lookup of a value that already exists. No aggregation on the request path, no fan-out across cells, and nothing that has to ask the running computation what it currently thinks.
Boundaries, outermost first: EVERY REGION RUNS THE WHOLE JOB: Window job (region B), Window job (region A), an empty slot for the says which region may write, and can take it back, tier 1 Outside every boundary: Customer app (asks a price), an empty slot for the holds the current multiplier per cell, tier 1, an empty slot for the answers the request path, tier 2 Connections: Window job controls says which region may write, and can take it back — may I write? Window job controls says which region may write, and can take it back — may I write? says which region may write, and can take it back calls holds the current multiplier per cell — the one holder writes holds the current multiplier per cell calls answers the request path Customer app calls answers the request path — one lookup