wtf( )unctionsystem design, drawn
← all problemsResilienceMedium

The burst was survived, and the notifications arrived at four in the morning

Chat, and the path out to the mobile platform that actually rings people's phones. One event in one busy room can turn into hundreds of thousands of notifications inside a second, against a baseline a small fraction of that.

The burst was survived — nothing crashed, nothing was lost, every notification was delivered. They arrived between one and four in the morning, hours after the conversation they were about, and for the whole of that time the senders were the only thing the hosts were doing.

  1. R1The vendor holds a hard limit on how many requests may be outstanding on one connection at once. Exceeding it does not return an error we can handle — it degrades the connection. Only the side holding that connection knows how many are outstanding right now.
  2. R2A notification that is minutes late is worthless. Dropping some during a once-a-month catastrophe is acceptable and understood. The notification system stalling for everybody is not, and neither is delivering a burst hours after the thing it was about.
  3. R3Notification volume must never be able to take resources the messaging path itself needs — a burst in one must not become a shortage in the other. Adding concurrency is not an available answer: the bottleneck is the rate the vendor accepts work at, not our capacity to produce it.
Compose the egress path. Tier 1 is what sets the rate and what sits between the producers and the senders, tier 2 is where the senders draw their capacity.
Components — tap one, then tap a slot on the diagram
?The vendor's limit is on requests already outstanding on a connection, and going past it degrades the connection rather than returning an error you could catch.

Outside every boundary: The messaging path (must not suffer), Devices (somebody's phone), Producers (mentions, calls), The push vendor (one connection), an empty slot for the what sets the rate, tier 1, an empty slot for the between producers and senders, tier 1, an empty slot for the where the senders draw from, tier 2 Connections: Producers calls between producers and senders — notifications between producers and senders calls what sets the rate — handed over what sets the rate calls The push vendor — outstanding The push vendor publishes to Devices — delivered what sets the rate calls where the senders draw from — runs on The messaging path calls where the senders draw from — also needs it

The messaging pathmust not suffer
Devicessomebody's phone
Producersmentions, calls
The push vendorone connection