wtf( )unctionsystem design, drawn
← all problemsMicroservicesHard

Three teams, one database, and every migration is a meeting

Orders, catalog and billing are three teams, three services, one database and one deploy train. Renaming a column takes six weeks, because two other teams read that column and neither can say when they could stop. Nobody has done anything wrong; the schema is a shared mutable global and everybody is holding a reference to it.

They want out, and "give each service its own database" is the answer they have already been given. It is also not, by itself, an answer — there are four more things in the way, and three of them are not about databases at all.

  1. R1Each service's store must be touched by exactly one service, so its shape is that team's private business and a migration needs no meeting. A second reader is what turns a schema into a negotiation, and the number of database servers is not what settles this — who may connect to them is.
  2. R2Instances come and go continuously: the platform replaces them on every deploy and whenever one fails a health check. A caller must never be handed the address of an instance that is not healthy right now, and no address may be written down anywhere in advance.
  3. R3Everything from outside enters through exactly one place, which is where a public caller is authenticated and where per-client request limits are applied. No service authenticates a public caller itself, which means whatever sits there has to know which client this is rather than only where to send it.
  4. R4A finance report needs data from all three services at once. Producing it must put no load on any of the three and may be up to an hour behind — so it cannot be assembled by reading them, and whatever holds it has to be able to turn three private shapes into one report rather than merely storing bytes.
Give each team its own data and its own front door. Tier 1 is the edge, tier 2 is how services find each other, tier 3 is the stores and the reporting path.
Components — tap one, then tap a slot on the diagram
?A column another service reads is a column you cannot drop. Everything in this problem follows from that one sentence, including the parts that look unrelated to it.

Outside every boundary: Finance (reads the report), Billing (its own deploys), Orders (its own deploys), Partners (public callers), Catalog (its own deploys), an empty slot for the one way in, authenticates and throttles, tier 1, an empty slot for the answers where healthy instances are, tier 2, an empty slot for the only the orders team may change it, tier 3, an empty slot for the only the catalog team may change it, tier 3, an empty slot for the only the billing team may change it, tier 3, an empty slot for the an hour stale, and free to the three, tier 3 Connections: Partners calls one way in, authenticates and throttles one way in, authenticates and throttles calls Orders one way in, authenticates and throttles calls Catalog one way in, authenticates and throttles calls Billing one way in, authenticates and throttles controls answers where healthy instances are Billing controls answers where healthy instances are — which are healthy Orders calls only the orders team may change it Catalog calls only the catalog team may change it Billing calls only the billing team may change it only the orders team may change it publishes to an hour stale, and free to the three only the catalog team may change it publishes to an hour stale, and free to the three only the billing team may change it publishes to an hour stale, and free to the three an hour stale, and free to the three calls Finance Billing must NOT reach only the orders team may change it

Financereads the report
Billingits own deploys
Ordersits own deploys
Partnerspublic callers
Catalogits own deploys