wtf( )unctionsystem design, drawn
← all problemsNetworkingHard

The policy that protected the wrong namespace

The platform team wrote a default-deny network policy when the production namespace was created, and it has worked ever since: nothing reaches those pods unless a rule says so.

A new namespace was set up for payments by copying the deployment manifests across. During an incident review someone notices a build runner in the shared tooling namespace can open a connection straight to the payments database. Nothing was misconfigured — the policy simply is not there, because a network policy is a namespaced object and applies to nothing outside its own namespace.

Put the isolation in the namespace where the traffic lands.
Components — tap one, then tap a slot on the diagram
!A build runner can reach the payments database directly. It has been able to since the namespace was created.

Boundaries, outermost first: prod (namespace): Orders API, Default-deny (written last year) payments (namespace): Payments database (no policy selects it; FAILED: open to the cluster), an empty slot for the isolate these pods tools (namespace): Build runner (shared tooling) Connections: isolate these pods controls Payments database — selects every pod here (step 1) Orders API calls Payments database — 5432 · allowed by rule (step 2) Build runner must NOT reach Payments database — reached it Default-deny controls Orders API — applies here only

Orders API
Payments databaseno policy selects itopen to the cluster
Build runnershared tooling
Default-denywritten last year