wtf( )unctionsystem design, drawn
← all problemsResilienceHard

The runbook needed the thing that was down

The failover plan is written, rehearsed and correct. Executing it needs the deploy pipeline, which authenticates through the identity service, which reads its configuration from the store that is down.

Every individual dependency here is reasonable. Together they mean the recovery path runs through the failure, and the plan cannot be executed during the only outage it was written for.

Tap the parts of the recovery path that the outage has already taken away.
!Recovery was blocked for forty minutes because every tool needed to fix it depended on the thing that was broken.

Outside every boundary: Break-glass keys (offline custody), Deploy pipeline (reads the store), Runbook (behind the identity), Configuration store (the outage itself; FAILED: down), Identity service (reads the store), Paging (separate provider), DNS failover (pre-staged elsewhere) Connections: Deploy pipeline controls Configuration store — for its config Identity service controls Configuration store Runbook controls Identity service — to be read at all

Tap every component that is wrong.