Zero-to-jupyterhub-k8s: Document / prototype using Magnum to deploy k8s on openstack

[choldgraf]

We recently tried deploying JupyterHub on k8s using openstack, and run into several issues in setting up the underlying resources. A few folks mentioned that Magnum was a potential solution to deploying a k8s cluster that would be a bit less of a hassle.

Does anybody have any experience deploying things with Magnum? We'd love to hear about your experiences / thoughts on its feasibility.

cc @enolfc who I think we chatted with about this

[ablekh]

@choldgraf
While I don't (yet) have direct experience in Magnum-related deployments, earlier I have spent some time researching relevant technologies. My understanding is that, for deployments based on OpenStack (as opposed to ones on public cloud), Magnum-based Kubernetes deployments are both feasible and rather straightforward (the latter does not apply to OpenStack itself :-). You just need to create relevant Magnum template(s) and launch cluster(s), based on the template(s) - essentially, 1-2 commands. For some details, examples and CERN case studies, please review the following resources, which, I hope, you will find useful.

• https://github.com/mjura/kubernetes-demo
• https://cloudbase.it/easily-deploy-a-kubernetes-cluster-on-openstack
• http://openstack-in-production.blogspot.com/2016/06/scaling-magnum-and-kubernetes-2-million.html
• http://openstack-in-production.blogspot.com/2017/01/containers-on-cern-cloud.html (follow-up post)

Off-topic, but related -- Consider Juju as yet another alternative for cloud-neutral K8s deployment:

• https://jujucharms.com/canonical-kubernetes (seems to be Ubuntu ecosystem-focused)

[manics]

I haven't used (and don't have access to) Magnum, but looking at the installation docs it seems the bulk of the work is done by an OpenStack server side component, which may not be installed by default. The video does look pretty neat though.

[rochaporto]

Jumping in late, but here goes some info on CERN / Magnum and jupyterhub.

With a recent Magnum release (Queens), which brings RBAC and Ingress it is quite easy to get jupyterhub and binderhub running. Note we cherry-pick a couple of patches, namely ingress tls with traefik:
https://review.openstack.org/#/c/575373/

but it should not make a difference to get something up and running, and it's quite easy to patch a deployment to add tls based on the review link above.

I'm happy to add a section with an OpenStack/Magnum specific set of instructions.

[ablekh]

@rochaporto Adding a section with self-sufficient relevant instructions will be much helpful & appreciated.

[betatim]

It would be great to see some more documentation on getting a k8s cluster on other providers. I think the relevant place to add it would be https://zero-to-jupyterhub.readthedocs.io/en/latest/#getting-to-step-zero-your-kubernetes-cluster. OpenStack is popular so I think we should have that in the list.

Did you also try having persistent storage with your JupyterHub? Getting that to work seems to be one of the trickier things and BidnerHub doesn't use it.

[enolfc]

@betatim, I have persistent storage working with JupyterHub and make it available at $HOME/persistent (not sure if that's the best approach as it's a bit confusing to users). I'm using NFS for PV so need to add the storageClass. Check the following:

singleuser:
  lifecycleHooks:
    postStart:
      exec: { "command": ["/bin/sh", "-c", "ln -s /persistent $HOME/persistent"]}
  storage:
    capacity: 1Gi
    homeMountPath: /persistent
    dynamic:
      storageClass: "managed-nfs-storage"
      storageAccessModes: ["ReadWriteMany"]

[craig-willis]

@rochaporto Have you started working on this? I'd be interested in contributing, reviewing or testing recommendations.

We've successfully deployed Z2JH in several OpenStack environments (SDSC's Cloud, NCSA's Nebula, XSEDE's Jetstream) without Magnum (via terraform) and I was considering whether to contribute our patterns to the Z2JH documentation. At the PEARC18 conference, I was introduced to several related initiatives, including one based on Magnum:

• Deploying Jupyter Notebooks at scale on XSEDE resources for Science Gateways and workshops: https://arxiv.org/abs/1805.04781
• Scaling JupyterHub on Jetstream https://github.com/sarajlic10/xsede-jetstream
• https://github.com/etiennedub/terraform-binderhub
• https://github.com/nds-org/kubeadm-terraform
• https://github.com/nds-org/esiphub

Perhaps there's opportunity for some collaboration around best practices on OpenStack?

[choldgraf]

@craig-willis I think this sounds fantastic. We've seen a few patterns now of K8S + JHub on OpenStack, and I think there'd be value in condensing this down into something actionable and clear. @etiennedub may have ideas on this too.

I am +100 on a more structured / tutorial / guide-like document that we can all start iterating on for deploying K8S and/or JupyterHub on OpenStack.

[craig-willis]

Great. Thanks, @choldgraf. Assuming there's nothing already in the works, we've started drafting some words.

https://docs.google.com/document/d/15FQW_miNWgFBlQazlQAJ7S1-EX0Z62t8z0FhAe_DoIw/

I'm optimistic that we can come to a consensus around a Magnum and non-Magnum approach, with the obvious caveat that no two OpenStacks are created equal.

Hoping @BenGalewsky @zonca @sarajlic10 @julienchastang (and any others) will also join in.

[aculich]

@craig-willis thanks for starting the document! It looks good and I also looped in @cmd-ntrf and @etiennedub on this github issue and the document that you started.

[choldgraf]

Thanks @craig-willis :-)

with the obvious caveat that no two OpenStacks are created equal.

ain't that the truth ;-)

It's important to remember that the Z2JH guide is also just one of many ways you could deploy JupyterHub on Kubernetes, so it's more important to provide a path, than to provide the path

[verdurin]

Has there been any more recent work on this?

[craig-willis]

I was holding off on updating the document until I had the chance to work through @zonca's Kubespray approach. @julienchastang may have more recent experience with Kubespray. I still primarily use kubeadm-terraform on the OpenStack systems we deploy on, where Magnum is not an option.

[julienchastang]

Yes I can confirm that @zonca's kubespray instructions (#1, #2, #3) work as advertised.

[craig-willis]

Thanks, @julienchastang. Is this the approach you will use going forward -- i.e., should it be the recommended approach in the docs for Jetstream?

[julienchastang]

That's the plan but I am very far from being a kubernetes expert and have not experimented with this solution a lot at this point. I am currently going through an experimentation phase. Nevertheless I am hopeful I can run with this approach.

[verdurin]

Thanks. We are hosting a workshop on Friday and would prefer to run on local OpenStack, if possible. Will look at the kubespray approach.

[choldgraf]

I believe that @cmd-ntrf recently updated their binderhub / jupyterhub deployment (which I think bootstraps their own k8s cluster on openstack). Maybe he can give an idea of what worked well etc.

[cmd-ntrf]

As @choldgraf said, I have updated last week the Terraform repo we presented at PEARC18 that bootstraps a k8s cluster and install BinderHub.

The updated repo can be found here: https://github.com/cmd-ntrf/terraform-binderhub. It installs binderhub v0.2.0-58e8ae9.

It uses Terraform and an updated version of Data 8 kubeadm-bootstrap repo. The data8 repo currently install kubernetes 1.9.2 which is vulnerable to Kubernetes CVE-2018-1002105. So I updated to Kubernetes 1.13.0.

I am currently at KubeCon, but I'll try to wrap a PR at least for data8 repo.

[verdurin]

In the end I followed the procedure outlined at: https://blog.jupyter.org/how-to-deploy-jupyterhub-with-kubernetes-on-openstack-f8f6120d4b1 which is a combination of Terraform with Kubespray.

Workshop was very successful.

[manics]

I'm closing this as it's quite old and it doesn't sound like there are any concrete actions for Z2JH.