wtf( )unctionsystem design, drawn
← all problemsObservabilityHard

Paged for a blip, silent for a slow bleed

The alert is "error rate above one percent for five minutes". It pages for every blip that clears itself, and it stayed quiet through a fortnight at nine tenths of a percent that spent the entire quarter's error budget.

A threshold on a rate answers "is it bad right now". The question the team actually has is "at this rate, will we run out of the budget we agreed", and those are not the same question.

Build the alert out of two windows rather than one threshold, and give the slow burn its own severity.
Components — tap one, then tap a slot on the diagram
?An alert that fires on any error is noise. An alert that fires when the monthly budget is gone is too late to be an alert.

Outside every boundary: Success ratio (the SLI), 0.9% for a fortnight (spent the quarter; FAILED: never alerted), A 90-second blip (cleared itself), Page the on-call, Open a ticket, an empty slot for the the window that detects, an empty slot for the confirms it is still happening, an empty slot for the the burn that never spikes Connections: Success ratio calls the window that detects (step 1) the window that detects calls confirms it is still happening — and (step 2) confirms it is still happening calls Page the on-call (step 3) Success ratio calls the burn that never spikes (step 4) the burn that never spikes calls Open a ticket (step 5) A 90-second blip must NOT reach confirms it is still happening — gone before it confirms 0.9% for a fortnight calls the burn that never spikes — burn rate 1

Success ratiothe SLI
0.9% for a fortnightspent the quarternever alerted
A 90-second blipcleared itself
Page the on-call
Open a ticket