Where the rate limiter has to sit
One client is sending forty times more traffic than anyone else and everyone's latency is suffering. A rate limiter is going in — and where it goes decides whether it works at all.
Each app instance keeping its own counter means the real limit is your limit times the number of instances, and it changes every time you autoscale.
Boundaries, outermost first: App tier: Instance 1 (FAILED: saturated), Instance 3, Instance 2 (FAILED: saturated) Outside every boundary: Clients, an empty slot for the before the app tier, an empty slot for the where counters live Connections: Clients calls before the app tier (step 1) before the app tier calls where counters live — one counter (step 2) before the app tier calls Instance 1 — what's allowed (step 3) before the app tier calls Instance 2 before the app tier calls Instance 3