One label, and the metrics store fell over
Somebody wanted to know which customers were hitting the slow endpoint, so they added the account id as a label on the request counter. One line, reviewed in a minute.
A metric with a label becomes a separate stored series for every distinct value of that label. There are two hundred thousand accounts. The store is now holding one series per account per endpoint per status code, all of them in memory at once.
Outside every boundary: Service, Metrics store (holds every one; FAILED: out of memory), requests_total (+ account label; FAILED: 200k series), Which account? (the real question), an empty slot for the where an account id can live, an empty slot for the how you get from one to the other Connections: Service calls requests_total (step 1) requests_total calls Metrics store — a series per value (step 2) Which account? controls where an account id can live — ask it here Which account? must NOT reach requests_total — not as a label requests_total controls how you get from one to the other (step 3) how you get from one to the other calls where an account id can live — jump to one request (step 4)