A forty-second blip that restarted everything
The primary database failed over for forty seconds. Every API pod's health endpoint checks the database, and that endpoint is wired to the restart probe — so the platform killed the entire fleet at once. The cold pods then stampeded the recovering database.
A forty-second blip became a twenty-minute outage, and nothing was wrong with the pods.
Boundaries, outermost first: Pod: API container Outside every boundary: Database (briefly failing over; FAILED: failing over), Platform (runs the probes), an empty slot for the restarts the container, an empty slot for the removes it from the load balancer Connections: Platform controls restarts the container (step 1) Platform controls removes it from the load balancer (step 2) restarts the container controls API container — restart removes it from the load balancer controls API container — in or out of rotation