Zero-to-jupyterhub-k8s: Support for NetworkPolicy ingress/egress

[manics]

I'm interested in using a singleuser pod NetworkPolicy to limit egress. The use-case I have in mind is providing a public but very locked-down jupyterhub deployment to give people a taster with minimal barriers, and to provide full access behind a second authenticated deployment.

I found a related discussion on https://github.com/jupyterhub/mybinder.org-deploy/issues/146 and can see pros and cons to including it in this chart:

• Configuring everything in zero-to-jupyterhub-k8s is very convenient
• A NetworkPolicy is very specific to a deployment, and can be applied outside the chart since the pod labels are known

If you think it's worthwhile here's a proposal:

singleuser:
  networkPolicy:
    ingress:
      - List of ingress rules from https://kubernetes.io/docs/concepts/services-networking/network-policies/
    egress:
      - List of egress rules from https://kubernetes.io/docs/concepts/services-networking/network-policies/

applied to

  podSelector:
    matchLabels:
      app: jupyterhub
      component: singleuser-server

[minrk]

I 100% think we should apply network policies here.

The one thing I was bitten by in jupyterhub/mybinder.org-deploy#146 is that kubernetes doesn't check at all for using unsupported features, so you can happily load an egress network policy that is completely ignored without any warnings or errors. I haven't checked in a bit, but I assume gke-1.9 will have sufficiently recent calico (>= 2.6.1) to enable egress policies.

We should start by sketching out a network policy of who needs to talk to whom, and what the default should be. To get started:

ingress:

• single-user servers need only accept ingress directly from the hub and proxy (everybody else should communicate with them via proxy)
• hub needs only accept ingress from proxy, services, and single-user server
• proxy should accept ingress from the world, or maybe only ingress-controller if enabled?

egress:

• proxy only needs egress to hub, services, single-user servers
• hub needs egress to proxy-api, services, single-user servers, and whatever may be needed by the Authenticator (maybe world for OAuthenticators?) and Spawner (kube itself for kubespawner)
• single-user servers need egress to the hub and the outside world, but no other in-cluster addresses

Figuring out the right labeling scheme / default policies for that would be great!

[manics]

Thanks, I'll try and come up with something. A barrier I've already run into is that the DNS server is in the kube-system namespace so the obvious thing is to allow egress to port 53 in kube-system, however namespaceSelector only works on labels, and at least on my cluster namespaces are unlabelled. Best I've come up with is to allow UDP/53 egress to anywhere.

[manics]

This is what I've come up with so far based on @minrk's outline: https://gist.github.com/manics/ccc96169e4ce22f4cf434c7c7f9b9630

I've been testing on openstack (deployed with kubespray, canal network plugin)

proxy

• Ingress from world 80,443, hub 8000,8001,8080
• Egress to hub 8081, singleuser-servers 8888, world 80,443 (for lets-encrypt, not sure about this)

hub

• Ingress from proxy 8081, singleuser-servers 8081
• Egress to proxy 8001, singleuser-servers 8888, k8s-api (I can't find a way to specifically whitelist this), optionally world for auth

singleuser-servers

• Ingress from hub 8888, proxy 8888
• Egress to hub 8081, proxy 8000,8001 (do users need access to all these APIs?), optionally world

[yuvipanda]

This is great, since I need something like this for a course starting in a month :D

I agree this belongs in z2jh. I think the only knob we should turn to begin with is 'who can single-user server talk to?', since you might have plenty of other in-cluster services that you might wanna expose to the singleuser server.

What you have sounds like a good start, @manics!

[minrk]

@manics awesome! Feel free to make a PR with what you have, and we can iron it out.

My only question based on your gist is about best practices in terms of how to organize the information: when giving pods access to the proxy, should we be putting those pods in the proxy's network policy explicitly, or should we be using a single label like 'proxy-api-access' in the network policy and applying that label to all of the pods that need access?

i.e. when granting a new pod access to the proxy, do I modify the pod's labels, or the proxy's network policy? It seems like the former makes the most sense for ingress. I'm not sure if egress is best treated the same or not.

[yuvipanda]

I think ingress is fairly easy policy:

1. Proxy allows ingress from everywhere
2. Hub allows ingress from proxy and singleuser servers only
3. singleuser servers allow ingress from proxy and hub only.

Egress is definitely going to be more complicated, so I suggest we tackle ingress first and then go from there?

[manics]

Thanks for the feedback, I'll open a PR today/tomorrow.

[manics]

Implemented in https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/546

[consideRatio]

@manics @minrk @yuvipanda awesome!!! I learned a lot by reading this thread!

@manics I think the pod will talk to the k8s-api through kube-proxy pods (one per node), so if we manage to target that we are good I think. Also I think they communicate with HTTP by default.

Name:         kube-proxy-gke-jupyter-se-cpu1-core-pool-08ec0c9f-ps6x
Namespace:    kube-system
Node:         gke-jupyter-se-cpu1-core-pool-08ec0c9f-ps6x/10.156.0.2
Start Time:   Tue, 03 Apr 2018 03:55:00 +0200
Labels:       component=kube-proxy
              tier=node
Annotations:  kubernetes.io/config.hash=52dadc5e78a1031dc12599fa181e4548
              kubernetes.io/config.mirror=52dadc5e78a1031dc12599fa181e4548
              kubernetes.io/config.seen=2018-04-03T01:54:55.691951702Z
              kubernetes.io/config.source=file
              scheduler.alpha.kubernetes.io/critical-pod=
Status:       Running
IP:           10.156.0.2
Containers:
  kube-proxy:
    Container ID:  docker://f8c76cadebfa4c5a4bf588a4ea875624c80cce6b04dbab2fabd6a74f8e095e3c
    Image:         gcr.io/google_containers/kube-proxy:v1.9.4-gke.1
    Image ID:      docker://sha256:6c8f7ef27e17f23e30691157b7b713d4032acc6bc04a8657082a26979559995b
    Port:          <none>
    Command:
      /bin/sh
      -c
      exec kube-proxy --master=https://35.198.66.16 --kubeconfig=/var/lib/kube-proxy/kubeconfig --cluster-cidr=10.16.0.0/14 --resource-container="" --oom-score-adj=-998 --v=2 --feature-gates=ExperimentalCriticalPodAnnotation=true --iptables-sync-period=1m --iptables-min-sync-period=10s --ipvs-sync-period=1m --ipvs-min-sync-period=10s 1>>/var/log/kube-proxy.log 2>&1