The limit was a hundred, and they got four hundred
A public API with a per-customer limit. It is enforced on every instance, the numbers look right in every dashboard, and one customer reliably gets several times their allowance.
There are four instances. Each keeps its own count.
Moving the count into a shared store fixes that and introduces two problems that are less obvious: one about which store, and one about what happens when it is unreachable.
- R1Every request for one customer must reach the same counter. State for one customer split across several counters is not merely less accurate — each part sees a fraction of the traffic, so each independently concludes the customer is well within their allowance.
- R2Checking the count and changing it must happen as one uninterruptible step. Two requests that each read the current value and then write one more than it will both be allowed, and under load that is the normal case rather than a rare race.
- R3When the shared counter cannot be reached, requests must still be decided locally rather than waiting or failing. That local decision is per instance and therefore approximate, and it must be deliberately more generous than exact — a limiter whose own outage stops all traffic has turned a degraded dependency into a total one.
Outside every boundary: The service (what is being protected), API instance (one of four), One customer (many requests), an empty slot for the sends a customer to their counter, tier 1, an empty slot for the checks and decrements in one step, tier 1, an empty slot for the decides when nothing can be reached, tier 2 Connections: One customer calls API instance API instance calls sends a customer to their counter sends a customer to their counter calls checks and decrements in one step — same customer, same counter API instance controls decides when nothing can be reached — when the counter is gone API instance calls The service — if allowed