wtf( )unctionsystem design, drawn
← all problemsResilienceHard

The retries that kept it down

A downstream service slows under load. Every caller retries three times immediately, so a service already struggling now receives four times the traffic. It stops responding entirely, and it cannot recover while the retries keep coming — because the moment it serves anything, it's buried again.

The outage now outlives its cause. Removing load is the only thing that ends it, and every client is doing the opposite.

Insert the thing that stops a client from calling a service that's already failing.
Components — tap one, then tap a slot on the diagram
!Payments is buried and cannot recover while every caller keeps retrying.

Outside every boundary: Caller B, Payments (degraded; FAILED: overloaded), Caller C, Caller A, an empty slot for the in front of the call Connections: Caller A calls in front of the call (step 1) Caller B calls in front of the call Caller C calls in front of the call in front of the call calls Payments — open: no call at all (step 2)

Caller B
Paymentsdegradedoverloaded
Caller C
Caller A