wtf( )unctionsystem design, drawn
← all problemsAWS SA ProHard

One zone went, and three zones lost the internet

A three-zone VPC that passed its resilience review. Every tier runs in all three Availability Zones, the database is Multi-AZ, and the load balancer is zonal-redundant.

There is one NAT gateway. It was created in us-east-1a during the first sprint and every private route table still points at it, because it works and nothing has ever made anybody look.

us-east-1a is impaired. Tap everything that loses outbound internet access with it.
!One Availability Zone degrades. Two thirds of the fleet is in healthy zones and still serving reads — and every outbound call they make, to payment providers, to package registries, to the licensing API, times out.Not everything in a healthy zone survives, and not everything that talks outward goes through the NAT. Both halves are on the diagram.

Boundaries, outermost first: us-east-1a: NAT gateway (the only one; FAILED: its zone is impaired), Orders API (zone a) us-east-1b: Orders API (zone b), Nightly batch (zone b) us-east-1c: Orders DB (Multi-AZ), Orders API (zone c) Outside every boundary: S3 endpoint (gateway type), Public ALB (all three zones) Connections: Orders API sends outbound traffic to NAT gateway — egress, same zone Orders API sends outbound traffic to NAT gateway — cross-zone egress Nightly batch sends outbound traffic to NAT gateway — and the batch job too Orders API sends outbound traffic to NAT gateway — from the third zone Nightly batch calls S3 endpoint — S3, never via the NAT Public ALB calls Orders API — inbound Orders API calls Orders DB — reads

The NAT gateway in us-east-1a has failed. Tap everything that stops working with it — not the failure itself.