wtf( )unctionsystem design, drawn
← all problemsMicroservicesHard

Three services, four retries each, sixty-four requests at the leaf

Inventory got slow. Not down — slow. Within ninety seconds it was taking sixty-four times its normal traffic, and nobody had deployed anything. Three services sit between the browser and inventory, each configured to retry a failed call up to four times, and four times four times four is sixty-four.

The fix that shipped that week was exponential backoff with jitter at every layer. The next incident had the same peak, spread over a longer window. Backoff changes when the attempts arrive. It does not change how many of them there are.

  1. R1One user request must have one bound on the total time it may consume, and every hop must be able to read how much of it is left and decline to begin an attempt that cannot finish inside it. A timeout configured separately at each hop is not this: three hops of two seconds each is six seconds of work performed for a caller who left after two.
  2. R2When inventory is shedding load it must say so on the call being shed, so the caller learns it from the response it is already waiting for. Anything the caller has to go and ask about separately reports a state that has already changed — during an overload the interesting state changes in hundreds of milliseconds.
  3. R3Once inventory's recent calls say it is not going to serve this one either, calls must stop being made at all — first attempts included, not only retries. Capping retries leaves the entire first-attempt volume arriving at a service that has already said it cannot take it.
Compose the call path. Tier 1 is what bounds the work one user request may cause and how the overloaded service says stop, tier 2 is what actually stops.
Components — tap one, then tap a slot on the diagram
?Every layer here is individually reasonable. Retrying a failed call is correct at one hop and is a multiplier at three, and no layer can see the product it is a factor of.

Outside every boundary: Cart service (retries four times), Edge service (retries four times), Browser (waits two seconds), Inventory (slow, not down; FAILED: 64× traffic), an empty slot for the bounds the whole request, not one hop, tier 1, an empty slot for the how inventory says it cannot take more, tier 1, an empty slot for the stops the calls being made at all, tier 2 Connections: Browser calls bounds the whole request, not one hop bounds the whole request, not one hop calls Edge service — the time left Edge service calls Cart service Cart service calls stops the calls being made at all stops the calls being made at all calls Inventory — if it is worth trying Inventory calls how inventory says it cannot take more how inventory says it cannot take more controls stops the calls being made at all — stop calling

Cart serviceretries four times
Edge serviceretries four times
Browserwaits two seconds
Inventoryslow, not down64× traffic