wtf( )unctionsystem design, drawn
← all problemsPaymentsHard

The bank had a slow afternoon and the shop went down

Checkout calls the acquiring bank inline and waits for it. That works until the acquirer has a slow afternoon — twenty seconds a call instead of two hundred milliseconds — and then every checkout request is holding a worker open, the pool fills, and the shop stops serving anybody at all.

Nobody's card was charged wrongly. The bank was not down. The shop was down because it had agreed to be exactly as fast as the slowest thing it talks to.

  1. R1Checkout must answer in a few hundred milliseconds while the acquirer takes up to twenty seconds, so the capture leaves the request path and is handed to something that holds pending work. Handing it to something that copies it to every listener charges the card once per listener, which is not parallelism.
  2. R2If the process dies between accepting the order and capturing the money, the capture must still happen. Nothing about a pending payment may exist only in memory — this is money that a shopper believes has been taken.
  3. R3The main line must run without a person in it, and a capture that has exhausted its retries must land somewhere a person is expected to look — without holding up the captures behind it. Those are two different places precisely because one is walked by hand and the other must never be.
  4. R4The shopper's page polls for the outcome, which the worker writes once it knows. The capture itself is the record that matters; this is a copy of its result, kept so a page can ask "is it done yet" without touching the ledger. Lose it and the page asks again. Make it durable and you have bought a second copy of the truth, plus the obligation to keep the two agreeing forever — a disagreement you can only have if you built the thing capable of it.
Take the acquirer off the checkout path. Tier 1 is what checkout hands the capture to before answering and where a capture goes once its retries are spent, tier 2 is what the shopper's page polls.
Components — tap one, then tap a slot on the diagram
?A synchronous call makes their latency your latency and their outage your outage. Everything here follows from refusing that one agreement.

Outside every boundary: Checkout API (answers in 300ms), Capture worker (takes one at a time), Acquiring bank (up to 20 seconds), Shopper (waiting on a page), Payments ops (works the parked ones), an empty slot for the holds the capture, one taker, tier 1, an empty slot for the where a spent capture waits, tier 1, an empty slot for the lose it and the page asks again, tier 2 Connections: Shopper calls Checkout API Checkout API calls lose it and the page asks again — what the page polls Checkout API publishes to holds the capture, one taker — before answering holds the capture, one taker publishes to Capture worker Capture worker sends outbound traffic to Acquiring bank Capture worker publishes to lose it and the page asks again — the outcome Capture worker publishes to where a spent capture waits — retries spent where a spent capture waits calls Payments ops Checkout API must NOT reach Acquiring bank

Checkout APIanswers in 300ms
Capture workertakes one at a time
Acquiring bankup to 20 seconds
Shopperwaiting on a page
Payments opsworks the parked ones