One partner is down, and every partner's data stops
The product accepts events from its customers and forwards each one to a third-party address that customer configured. Those addresses fail constantly, for hours at a time, and there is nothing anyone here can do to make them better.
The last design used one queue per destination. It survived a single partner going down and then fell over anyway, for a reason the requirements below state exactly.
- R1Accepting an event must succeed and be acknowledged even while every downstream address is failing, so acceptance is separated from delivery by something that keeps the event until it is delivered.
- R2Within a single third-party address, one very large customer sending a flood of failing traffic must not delay a small customer's traffic to that same address. Volume per customer is extremely uneven.
- R3Operators must be able to change delivery priority while the incident is happening, without moving or copying the pending work.
- R4Each store of pending work has exactly one writer at a time, and that exclusivity is granted from outside the workers rather than agreed between them.
- R5Nothing may be lost. Work still undelivered when its window closes must be moved somewhere cheap and durable that can be inspected and replayed later, so the pending store stays small.
Boundaries, outermost first: OUR SYSTEM: Ingest gateway (acks fast), Delivery worker (retries with backoff), an empty slot for the holds the event once it is accepted, tier 1, an empty slot for the grants one owner per pending store, tier 2, an empty slot for the work waiting, re-orderable in place, tier 2, an empty slot for the where work goes when the window shuts, tier 3 Outside every boundary: Partner endpoint (not ours, often down), Customer apps (emit events) Connections: Customer apps calls Ingest gateway Ingest gateway calls holds the event once it is accepted holds the event once it is accepted calls Delivery worker Delivery worker controls grants one owner per pending store — lock session Delivery worker calls work waiting, re-orderable in place work waiting, re-orderable in place publishes to where work goes when the window shuts — on expiry Delivery worker sends outbound traffic to Partner endpoint