wtf( )unctionsystem design, drawn
← all problemsAPI DesignHard

The retry that charged them twice, and the one that charged them wrong

A public payments API. A customer's mobile network drops the response to a charge, their app retries exactly as designed, and they are charged twice — the first request had succeeded and nobody could tell.

The fix everyone reaches for is a key on the request that says "this is the same call". It closes the double charge and opens a worse hole, because the caller supplies that key and callers reuse them: a retry that arrives with the same key and a DIFFERENT amount must not quietly return the first result as though it had done the second.

  1. R1Two requests carrying the same caller-supplied key must produce one charge, and the second must receive what the first returned. That promise has to hold across a restart of whatever is keeping it — a crash between the two is exactly when a retry arrives.
  2. R2A request that reuses a key with different content is a caller mistake and must be refused rather than answered. Returning the first result for it would confirm a charge that was never made, so what is stored alongside the key has to be enough to tell the two apart.
  3. R3Two copies of the same request arriving at the same instant must not both proceed. Exactly one may do the work while the other waits or is refused, and that exclusivity has to be granted from outside the request handlers, which know nothing about each other.
Compose the guarantee. Tier 1 is what recognises a repeat and what proves it is really the same request, tier 2 is what stops two of them at once.
Components — tap one, then tap a slot on the diagram
?A retry is not a mistake to be prevented. It is what a correct client does when it does not know what happened, and it will happen whatever you build.

Outside every boundary: Payments API (public), Customer's app (retries on timeout), Acquiring bank (charges once, or twice), an empty slot for the what this key already returned, tier 1, an empty slot for the what the original request said, tier 1, an empty slot for the lets exactly one copy proceed, tier 2 Connections: Customer's app calls Payments API Payments API calls what this key already returned — seen this key? what this key already returned calls what the original request said — same request? Payments API controls lets exactly one copy proceed — may I proceed lets exactly one copy proceed sends outbound traffic to Acquiring bank

Payments APIpublic
Customer's appretries on timeout
Acquiring bankcharges once, or twice