The router came back in nine seconds and nobody could push for an hour
The same self-hosted source-control product, and a different kind of state. Repositories live on storage nodes, and a router in front of them knows which copy of each repository is the authoritative one — a small, fast-moving map that is not source code and is not the product's own metadata either.
The router restarts cleanly and comes back in nine seconds. Pushes fail for the next hour.
- R1Nothing may reach a repository storage node except through the component that routes to it. A web node that writes to a storage node directly is writing behind the router's back, and the router is the only thing that knows which copy is authoritative.
- R2The router keeps its own bookkeeping about which copy of each repository is current. It must survive the router process being replaced, and it must be recoverable on its own — losing it is a different incident, with a different runbook, from losing the product's data.
- R3The product's own metadata — users, projects, permissions, review history — is queried constantly by the application and is what people mean when they say the database. Whatever holds it has to answer questions about it, not merely keep it.
Boundaries, outermost first: THE CUSTOMER'S OWN HARDWARE: Storage nodes (repositories on disk), Web nodes (serve git over HTTP), an empty slot for the the only way to the storage nodes, tier 1, an empty slot for the which copy is authoritative, tier 1, an empty slot for the users, projects, permissions, tier 2 Connections: Web nodes calls the only way to the storage nodes — every push the only way to the storage nodes calls Storage nodes — reads and writes the only way to the storage nodes calls which copy is authoritative — which copy? Web nodes calls users, projects, permissions — users, permissions