wtf( )unctionsystem design, drawn
← all problemsMicroservicesHard

The search service issued a refund

Search was compromised through a dependency, and eleven minutes later it issued a refund. It forged nothing and escalated nothing. It opened a connection to the payments service, presented the workload certificate its own pod had been issued that morning, and called the refund endpoint.

Every hop was mutually authenticated and every certificate was valid. The postmortem found no misconfiguration, because there was no configuration: payments had never been told which services were allowed to call it, and a service that has not been told which callers to refuse accepts all of them.

  1. R1Payments must refuse any caller no rule names. Workload identity is already deployed — every pod holds a short-lived certificate, every hop is mutually authenticated — so WHO is calling is settled. What is missing is whether that caller may make this call, and for a caller nobody has written a rule about the answer has to be no.
  2. R2A rule must name the operation, not only the service. Search legitimately reads prices from payments; it has never issued a refund and never will. A permission of the form "search may talk to payments" is one a compromised search spends on every endpoint payments exposes.
  3. R3Payments appends ledger entries and never rewrites them. Whatever credential it presents to the ledger must be able to do only that, so a compromise there produces entries reconciliation can find rather than a history that can be edited into agreeing with itself.
  4. R4The call graph is not written down anywhere. Before anything is refused, every call the rules do not name has to be recorded while still being let through — because the calls nobody documented include the ones the business runs on.
Compose the authorization path. Tier 1 is what decides whether a call is allowed at all and what payments itself may do to the ledger, tier 2 is what has to be running before any of it starts refusing anything.
Components — tap one, then tap a slot on the diagram
?Mutual TLS proved which pod was on the other end of the connection and answered no question about what that pod was allowed to do. Those are two questions and only one of them was being asked.

Boundaries, outermost first: PAYMENTS OWNS THIS: Payments (the callee), Ledger (what a refund moves), an empty slot for the what payments presents to the ledger, tier 1 Outside every boundary: Workload identity (already on every pod), Checkout (does issue refunds), Search (assume it is breached; FAILED: compromised), an empty slot for the says whether this caller may make this call, tier 1, an empty slot for the learns the call graph without breaking it, tier 2 Connections: Checkout calls says whether this caller may make this call Search calls says whether this caller may make this call — the same door Workload identity controls says whether this caller may make this call says whether this caller may make this call calls Payments — only if a rule names it says whether this caller may make this call controls learns the call graph without breaking it — each verdict Payments calls what payments presents to the ledger what payments presents to the ledger calls Ledger

Paymentsthe callee
Ledgerwhat a refund moves
Workload identityalready on every pod
Checkoutdoes issue refunds
Searchassume it is breachedcompromised