wtf( )unctionsystem design, drawn
← all problemsAWS SA ProHard

The writer failed, and the reporting team never noticed

A Multi-AZ DB cluster: one writer and two reader instances across three Availability Zones. Everything that talks to it is drawn below, and every one of those arrows is a database dependency.

They are not the same dependency. Some of these send writes and some only ask questions, and the difference decides who is in the incident.

The writer instance fails. Tap everything that stops working until failover completes — not the writer itself.
!A writer instance is lost at 09:14. Checkout starts erroring immediately. The reporting dashboards, which everybody assumed would be the first casualty, do not so much as blink.Reads and writes fail differently here, and one of the survivors is the reason the outage lasts as long as it does.

Boundaries, outermost first: Multi-AZ DB cluster — three zones: Reader 1 (zone b), Reader 2 (zone c), Writer (one instance; FAILED: lost at 09:14) Outside every boundary: Reports API (reads only), Checkout API (writes orders), Payment consumer (writes receipts) Connections: Checkout API calls Writer — INSERT Payment consumer calls Writer — INSERT Reports API calls Reader 2 — SELECT Writer publishes to Reader 1 — semisynchronous Writer publishes to Reader 2 — semisynchronous

The writer instance has failed. Tap everything that stops working with it — not the failure itself.