Zero-to-jupyterhub-k8s: Automatically testing singleuser servers

[manics]

I'm setting up continuous deployment with GitLab for our JupyterHub platform (I should be able to share the config for anyone interested), and one of the things I'd like to do is add in automatic testing of a staging deployment:

1. Deploy the JupyterHub Helm chart and config to a staging environment
2. Test a singleuser server can be spawned
3. Test notebook(s) are successfully run in the singleuser server
4. If tests pass automatically update the production deployment

It sounds like (2) and (3) are the sorts of thing someone might have done elsewhere in the Jupyterhub (or other external) projects, can you give me any pointers on where to look?

An added benefit of this issue would be adding the tests to https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/ci/test.sh

[consideRatio]

I'd love to see something like that setup for this repo! I don't have pointers for (2) and (3), but in #625 I've added static validation of the chart and the kubernetes objects (helm lint, helm template into yamllint, then kubeval).

Is the work you do on that platform publicly accessible? I'd love to learn from it.

[manics]

It'll be public, just need to decide where to put it. I'll let you know when it's done.

[choldgraf]

Correct me if I'm wrong, but it sounds like you're mostly interested in figuring out the API for spawning servers etc w/o the UI?

[manics]

Yes, I think that would do it. I was hoping someone had an example somewhere I could copy.

[betatim]

We do some automated testing as part of the binderhub CI as well as mybinder.org deployment which could serve as a starting point for your work.

[choldgraf]

note that this is related to https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/172, which will probably be closed whenever this issue is closed!

[jgerardsimcock]

@manics We'd like to do continuous deployment for our jupyterhub, can you please share your config?

[manics]

@ jgerardsimcock I haven't had time to add ci-testing, so at the moment I've only created an automatic deployment using GitLab: https://gitlab.com/openmicroscopy/mirrors/k8s-analysis-deploy

Unfortunately this is currently broken due to issues with the z2jh prepuller, so I'm back to manual deployments at the moment.

[betatim]

@jgerardsimcock have you taken a look at the tests we do for mybinder.org? Links in https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/639#issuecomment-380049228

[consideRatio]

I'd love to see your GitLab setup @manics!

I worked some on #846 and got plenty of excellent help from @minrk but there is a far way to go still.

[manics]

@consideRatio Our GitLab setup for deploying z2jh: https://gitlab.com/openmicroscopy/mirrors/k8s-analysis-deploy
There are several deficiencies at the moment, including lack of tests to verify JupyterHub is running after the deploy, and a proper staging environment.

Or did you mean the underlying infrastructure? We've got a Kubernetes cluster with a gitlab-runner that has cluster-admin permissions, hooked up to the above repo.

[consideRatio]

@manics oh awesome thanks for sharing! I'm very happy about learning about GitLab integration in general. It is very relevant to us to figure out a good way to test the z2jh project, I find it quite hard to setup. Switching from using minikube's localkube to kubeadm caused a big headache especially.

[manics]

Covered by https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/1053