Eight replicas were slower than two
A contract-review product. Every request is the same long system prompt, then one of five thousand contracts at roughly thirty thousand tokens, then a short question. Thousands of people ask about the same contract, and one person asks about many.
On two replicas, time-to-first-token sat around four hundred milliseconds — the engine recognised the contract from the last request and reused the attention state it had already computed for it. The team scaled to eight replicas for a launch.
Outside every boundary: Replica 2 (holds contract 118), Replica 1 (holds contract 4021), Questions (many users, 5k contracts), Inference router (round-robin today; FAILED: cache-blind), Replica 3 (nothing cached yet), an empty slot for the what it routes on, an empty slot for the when that one is busy Connections: Questions calls Inference router — contract 118 (step 1) what it routes on controls Inference router (step 2) when that one is busy controls Inference router (step 3) Inference router calls Replica 2 (step 4) Inference router calls Replica 1 Inference router calls Replica 3