The backfill that ate the assistant
One model, one deployment, one endpoint. The interactive assistant calls it all day. At two in the morning a job calls the same endpoint to summarise every document uploaded that day — twelve thousand of them, submitted as fast as the client can open connections.
The engine mixes requests together as it goes, so neither workload waits politely behind the other. What they do share is a fixed pool of GPU memory holding the attention state of everything currently in flight, and when it fills, the scheduler evicts in-flight work and starts it over.
Boundaries, outermost first: Interactive path: GPU replicas (one pool of memory; FAILED: evicting live work), Answers (some never finish), One endpoint (both workloads land here; FAILED: nothing separates them), Assistant users (all day, one at a time) Outside every boundary: Nightly backfill (12,000 documents), Summaries (due by morning), an empty slot for the what it submits to, an empty slot for the where that work runs Connections: Assistant users calls One endpoint (step 1) One endpoint calls GPU replicas (step 2) GPU replicas calls Answers (step 3) Nightly backfill publishes to what it submits to — 12,000 jobs (step 4) what it submits to publishes to where that work runs — a few at a time (step 5) where that work runs publishes to Summaries (step 6) Nightly backfill must NOT reach One endpoint — not this one