Gardener: Deploy network policy in the shoot allowing network traffic for the system components

[vpnachev]

Currently, we assume that the network traffic to and from our system component pods installed inside the shoot cluster are allowed. Unfortunately the shoot owner could define NetworkPolicy disallowing the network traffic to our pods, which affects the gardener ability to operate the shoot cluster. So we have to deploy proper NetworkPolicy whitelisting the traffic to and from our pods (garden.sapcloud.io/role: system-component).

[marwinski]

It appears that our stakeholders want better isolation of the system components. Gardener cannot parse the network policies to detect whether they are ok (well it could but this would mean we build an engine here). I would therefore propose to deploy a standard set of policies and have them checked by the reconciler. If stakeholders are not happy with that policy they need to discuss their concerns with the Gardener team.

An open question are network policies introduced by calico for example. Should we check those, too?

After some discussions and investigation I'd propose the following policies:

• dns: allow ingress port 53 (or was it 10053), egress only to api server ?
• kube-proxy: disallow all ingress, egress to api server
• calico: disallow all ingress, egress to api server
• dashboard: ingress only from vpn-shoot, egress to api server ?
• metrics-server ??
• node-exporter: disallow egress, allow ingress from vpn-shoot
• vpn-shoot: ingress from LB only, egress to the whole cluster
• ingress controller: ? possibly a good time to move to another namespace

[marwinski]

@vpnachev let me re-iterate my point from last Friday: the control plane is currently quite unprotected. Therefore we must add network policies and possibly do do more such as not running those containers with high privileges (not part of this ticket). Some stakeholders will however assume this not to be sure enough and start "optimising", possibly breaking the cluster. Imagine the number of tickets when you are operating 10000+ clusters. It is therefore essential that the reconciler corrects misconfigurations that may break a cluster and will lead to customer tickets.

[marwinski]

There are several issues that need to be addressed with network policies described above in the shoot cluster:

Zone Transfer query possible within the Kubernetes Cluster (Information Disclosure)

Zone Transfer protocol is used to keep redundant DNS servers in sync. The service is reachable on the following DNS servers within the Kubernetes cluster:

100.96.11.2 (no dns, or at least no reverse lookup entry)
100.64.0.10 (kube-dns.kube-system.svc.cluster.local)
Impact: an attacker from within the Kubernetes cluster, can extract the list of all hosts within the cluster, with a single query.

$ dig -tAXFR 100.64.0.10

; <<>> DiG 9.11.5-1-Debian <<>> -tAXFR cluster.local
;; global options: +cmd
[...]

It appears that we need to block TCP port 53 for all pods except the kube-dns pods.

Potential MitM through BGP hijacking

Port 179 is used by Calico for routing between the Pods using the Border Gateway Protocol. The protocol is prone to manipulation and potentially enables Man-in-the-Middle scenarios within Kubernetes network, in the same manner it happens regularly on the Internet. See https://en.wikipedia.org/wiki/BGP_hijacking for details.

[mwennrich]

We also need a ingress network policy for the vpn-shoot, like:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  annotations:
    gardener.cloud/description: |
      Allows Ingress to the vpn-shoot pod 
  name: allow-to-vpn-shoot
  namespace: kube-system
  labels:
    origin: gardener
spec:
  ingress:
  - from:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports:
    - port: 1194
      protocol: TCP
  podSelector:
    matchLabels:
      app: vpn-shoot
      garden.sapcloud.io/role: system-component
      origin: gardener
  policyTypes:
  - Ingress

[gardener-robot]

@dguendisch, @schrodit, @vpnachev, @timuthy, @zanetworker, @rfranzke, @tim-ebert, @ialidzhikov, @mvladev, @timebertt, @danielfoehrkn This issue was referenced by someone in a duplicate issue. Details redacted in public. Check backend logs.