wtf( )unctionsystem design, drawn
← all problemsResilienceHard

The limit that counted the wrong thing

The front door negotiates a maximum of a hundred concurrent streams per connection, which everybody read as a hundred concurrent requests.

A client opens a stream, the server dispatches the request to a handler, and the client cancels the stream immediately. The stream closes, the counter drops, a new stream takes its place — and the handler keeps running. The counter was accurate throughout. It was counting protocol state, and the resource is consumed by work.

Bound the thing that actually consumes the resource, and close the second route to the same end state.
Components — tap one, then tap a slot on the diagram
!One connection saturated every worker. The concurrency limit was set to 100 and was never exceeded.

Boundaries, outermost first: One connection: Streams: 100 (never exceeded), Handlers (still running; FAILED: unbounded), an empty slot for the what bounds the work, an empty slot for the the second route in Outside every boundary: One client (open, then cancel), Backend pool (exhausted) Connections: One client calls Streams: 100 — open, then cancel (step 1) Streams: 100 calls Handlers — already dispatched (step 2) Handlers calls Backend pool (step 3) what bounds the work controls Handlers (step 4) the second route in controls Streams: 100 — in and out

Streams: 100never exceeded
One clientopen, then cancel
Backend poolexhausted
Handlersstill runningunbounded