Zero-to-jupyterhub-k8s: Attempt to use Istio to secure connections between components of jupyterhub

[summerswallow-whi]

I opened this issue as a place to discuss an attempt to implement istio's side car injection to supply encryption in transit for jupyterhub on a kubernetes cluster.

So far I've followed Istio's quick start at https://istio.io/docs/setup/kubernetes/quick-start.html#aws-wkops for Kubernetes with KOps and automatic sidecar injection at https://istio.io/docs/setup/kubernetes/sidecar-injection.html. However, when I do this a helm install of jupyter hub fails.

I realize this is uncharted territory and I will continue to update this as I learn more.

[consideRatio]

@summerswallow-whi excellent! I'm learning about Envoy and Istio as well. Very new to this.

Related from KubeCon 2018 Europe - Envoy, Istio...

• [x] [Envoy Project Intro](https://www.youtube.com/watch?v=RUPgiHqUAZg&t=502s&list=PL5BHGvBbuEcyWw24eT_Tl4xQuP60shCrS&index=7)
• [x] [Envoy Deep Dive](https://www.youtube.com/watch?v=odRR94jhNbU&t=0s&index=270&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)
  
  
  
  • Consul, keeps track of all discovered services?
  
  
  • Gimbal, a heptio project to function as a multi-cluster ingress load balancer
  
  
  • Contour, ingress controller based on envoy
  
  
  • SPIRE, a SPIFFE runtime environment... TLS for envoy communication?
  
  
• [x] [Envoy Internals Deep Dive](https://www.youtube.com/watch?v=gQF23Vw0keg&t=0s&list=PL5BHGvBbuEcyWw24eT_Tl4xQuP60shCrS&index=13)
  
  
  
  • TLS _Thread Local Storage_
  
  
  • RCU _Read Copy Update_
  
  
  • xDS Envoy discovers its various dynamic resources via the filesystem or by querying one or more management servers.
  
  
  • C10k - Supports massive amounts of connections by handling multiple connections per thread (one worker thread per core) with an event loop.
  
  
  • _Hot Restart_ of Envoy supported (shared memory region, scale up Envoy to 2 -> scale down to 1)
  
  
• [ ] [From Kubelet to Istio: Kubernetes Network Security Demystified](https://www.youtube.com/watch?v=Uocf67aD5QQ&t=0s&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo&index=235)
• [ ] [Introduction to Istio Configuration](https://www.youtube.com/watch?v=x1Tyw8dFKjI&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo&index=81)
• [ ] [Extending Istio Service Mesh w/Envoy v2 APIs](https://www.youtube.com/watch?v=yq-c3H1Cd1A&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo&index=127)
• [ ] [Building Hybrid Clouds with Istio](https://www.youtube.com/watch?v=bLJL53UIcqI&t=0s&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo&index=53)
• [ ] [GitOps for Istio - Manage Istio Config like Code](https://www.youtube.com/watch?v=VkKMf23ZokY&index=189&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)
• [ ] [Replacing NGINX with Envoy in a Traffic Control System](https://www.youtube.com/watch?v=MO_0bwcNEZY&t=0s&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo&index=54)
• [ ] [Canary Deploys with Kubernetes, Istio and Envoy](https://www.youtube.com/watch?v=MGLDRKiXNf0&index=227&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)
• [ ] [Accelerating Envoy with the Linux Kernel](https://www.youtube.com/watch?v=ER9eIXL2_14&index=224&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)
• [ ] [Practical and Useful Latency Analysis using Istio and OpenCenus](https://www.youtube.com/watch?v=U8CfaR_t6jk&index=87&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)
• [ ] [Observing and Troubleshooting your Microservices with Istio](https://www.youtube.com/watch?v=TSEqxsctXVc&index=223&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)
• [ ] [Istio Tells me my Service has Slow Response Time, Now What?](https://www.youtube.com/watch?v=ZF-_LN9XYiQ&index=225&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)
• [ ] [Istio - The Weather Company's Journey](https://www.youtube.com/watch?v=0fKi3NeCsSE&index=229&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)
• [ ] [From Kubelet to Istio: Kubernetes Network Security Demystified](https://www.youtube.com/watch?v=Uocf67aD5QQ&index=234&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)
• [ ] [Istio Multi-Cluster Mesh Expansion BOF](https://www.youtube.com/watch?v=_seY39zvgA4&index=272&list=PLj6h78yzYM2N8GdbjmhVU65KYm_68qBmo)

[summerswallow-whi]

Have you made any progress getting it work with jupyterhub.

Right now I am finding I'm injecting the sidecars but nothing can communicate with each other, but I have tagged the entire kubernetes jupyterhub namespace for automatic sidecar injection. I don't think automatic injection will work as it is stated that pods should only have one service attached and I believe the proxy pod has two. I'm going to start looking at manual injection. Though I think automatic injection needs to be in place for notebooks as hub spawns it and it would be ideal if we don't have to mess with the spawner.

[consideRatio]

@summerswallow-whi no I haven't tried setting it up, but I mean to in a month or so.

[summerswallow-whi]

For the moment I've gone the direction of trying to encrypt through weave see issue https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/699

[aleicher]

@summerswallow @consideRatio any update on this? Is there a recommended way to get jupyterhub with the spawner up and running, when using istio?
I disabled istio sidecar injection to make the jobs succeed, but hub and proxy fail to come up properly with istio sidecars. disabling the istio injection for those (sidecar.istio.io/inject: "false") allows to start them properly, but then I cannot expose jupyterhub using the istio gateway.
Would love to hear your results/approach.

[consideRatio]

@aleicher I didn't focus effort into getting istio to setup mTLS between pods etc or routing traffic with istio-gateway. I have little insights to provide about this at the moment =/

I'm curious what stopped the hub / proxy pods to become ready though, if you open a thread on discourse.jupyter.org in the z2jh category where you describe what istio-usage end goal you have, and where you get stuck, i'd be happy to try help you navigate past the issues you run into

[harsimranmaan]

Open-sourced https://github.com/splunk/jupyterhub-istio-proxy to manage traffic routing via istio-gateway. More to come with examples, documentation and setup guide

[shenghu]

@aleicher have you been able to setup Istio with JupyterHub?

[shenghu]

@harsimranmaan can jupyterhub-istio-proxy be used to enforce ssl communication between JupyterHub's services?

[harsimranmaan]

@shenghu To use mTLS with JupyterHub services, you'd need to enable istio sidecar injection.I've had to patch kubespawner to get the mTLS going. Once you do that, the default configurable-http-proxy may not work for you. jupyterhub-istio-proxy can be used instead if you are using the istio service mesh in your cluster. In my setup I have the sidecar injection enabled by default. I can try to help you with your setup if you'd like and send any required patches upstream.

[shenghu]

@harsimranmaan have you run istio w/ jupyterhub in production env? Is there issue, e.g. reliability or performance? It would be appreciated that you can send me how to replace CHP w/ jupyterhub-istio-proxy and the needed patches. Thanks!

Right now I simply enable internal_ssl for jupyterhub and hub is not able to be detected as ready. Not sure if it is caused by CHP. I see these error
[root@rwedevtest1 ~]# kubectl logs -l component=hub -n rwe [W 2020-08-28 16:00:36.387 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 40964): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852) [W 2020-08-28 16:00:46.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 40978): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852) [W 2020-08-28 16:00:56.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 40992): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852) [D 2020-08-28 16:01:00.934 JupyterHub proxy:766] Proxy: Fetching GET http://10.111.56.85:8001/api/routes [I 2020-08-28 16:01:00.938 JupyterHub proxy:320] Checking routes [W 2020-08-28 16:01:06.380 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41008): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852) [W 2020-08-28 16:01:16.379 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41022): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852) [W 2020-08-28 16:01:26.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41036): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852) [W 2020-08-28 16:01:36.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41050): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852) [W 2020-08-28 16:01:46.378 JupyterHub iostream:1433] SSL Error on 10 ('10.61.3.80', 41064): [SSL: PEER_DID_NOT_RETURN_A_CERTIFICATE] peer did not return a certificate (_ssl.c:852)

[shenghu]

I see this is still WIP, https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1520. But we really need enforce ssl within JupyterHub services. Could istio+jupyterhub-istio-proxy be a reasonable solution?

[harsimranmaan]

@shenghu I'll prepare the patches for upstream this week. Thanks for your patience.

[harsimranmaan]

Added some background to running JupyterHub with istio https://medium.com/@harsimran.maan/running-jupyterhub-with-istio-service-mesh-on-kubernetes-a-troubleshooting-journey-707039f36a7b

[consideRatio]

I'm closing this issue now as it does not provide a concrete action point to take. Thank you so much @harsimranmaan for sharing your work on this!

My summary of the Istio state is that z2jh/kubespawner doesn't support this yet, but @harsimranmaan has demonstrated a set of changes and tooling can make it work!

[zachwe]

An alternative which requires less configuration than @harsimranmaan's proposal is to run configurable-http-proxy with --change-origin (documented here). This works out of the box with Istio. @consideRatio would you approve a PR that enables optionally running the proxy with this setting?

[consideRatio]

@zachwe I don't fully understand the change you suggest, could you open an issue where you describe the change in a way that doesn't require me to know so much more than Istio want to add sidecards where the network traffic become routed through?

Not understanding the change better, I cannot evaluate if we can implement it without breaking changes so well, or how much knowledge is required to maintain the feature in the future etc.