Retrying politely into a wall
Retries are jittered and budgeted, so the failing dependency is no longer being hammered. It is still being asked, by every single request, and every request pays the full timeout before failing.
The dependency has been down for four minutes, and the service has spent four minutes of every thread it owns finding that out again.
Outside every boundary: Caller (threads waiting; FAILED: blocked), Last known good, Dependency (down 4 minutes; FAILED: down), an empty slot for the while it is healthy, an empty slot for the once the threshold trips, an empty slot for the after the cooldown Connections: Caller calls while it is healthy (step 1) while it is healthy calls Dependency (step 2) while it is healthy controls once the threshold trips — threshold (step 3) once the threshold trips calls Caller — fail at once once the threshold trips controls after the cooldown — cooldown (step 4) after the cooldown calls Dependency — a few trials after the cooldown controls while it is healthy — on success Last known good calls Caller — fallback