They added servers and the backplane got slower
Twenty socket servers, each holding its own connections, and a broker between them so a message published on one reaches the members held by the others. It worked at four servers. At twenty it is slower than it was at twelve, and every deploy loses a handful of messages that nobody can account for.
Both symptoms come from properties of the broker that were fine at four servers and are not properties anyone chose.
- R1A message published while one server is briefly not subscribed — restarting during a deploy, say — must still reach it. Publication cannot be fire-and-forget, where anything sent while nobody is listening is simply gone.
- R2A server must receive the channels it actually holds subscribers for, and not every channel in the system. Otherwise each server's inbound volume is the total message rate rather than its share of it, and adding servers increases total load without reducing anyone's.
- R3Because delivery is retried rather than assumed, the same message can arrive more than once. Servers must recognise one they have already handled, using something that remembers recent identifiers and forgets old ones by itself — the memory must be bounded without anyone pruning it.
Outside every boundary: Server holding members, Publishing server (holds the sender), Server holding none (of this channel), an empty slot for the survives a subscriber being briefly away, tier 1, an empty slot for the decides which servers see this channel, tier 1, an empty slot for the was this one handled already, tier 2 Connections: Publishing server publishes to survives a subscriber being briefly away survives a subscriber being briefly away publishes to decides which servers see this channel decides which servers see this channel publishes to Server holding members — holds members Server holding members calls was this one handled already — seen it?