Throttled on an idle machine
A latency-sensitive service is configured with a CPU request of 200m and a CPU limit of 500m. It is being throttled in about a fifth of every scheduling period, and its p99 has tripled. The node it runs on is thirty percent busy, with cores sitting idle beside it.
Five things are being said in the incident channel. Two describe the mechanism correctly. Three are wrong, and each of the three is about to justify a change that will not help.
Boundaries, outermost first: Node — 30% busy: Spare cores (nobody using them), The service (throttled 18%; FAILED: throttled) In the incident channel: Raise the request to 500m, The old kernel quota bug, The node must be busy, 8 threads, 8× the quota, A quota per 100 ms Connections: The service must NOT reach Spare cores — quota says no