Kubeflow: Notebooks connection error

[SpontaneousDuck]

/kind bug

What steps did you take and what happened:
Creating notebooks looks like it works but any connection returns a 404 error on the jupyter server. tried a bunch of different provided servers and was only able to get the tensorflow 1.5 w/gpu image working. All other images return a webpage with a 404 error and a jupyter header. Custom images also do the same thing (tensorflow/tensorflow). I just upgraded from kubeflow v0.5 and all images I tried worked on that version.

What did you expect to happen:
Notebook launches correctly from the dashboard

Environment:

• Kubeflow version: v0.6.2
• kfctl version: v0.6.2-0-g47a0e4c7
• Kubernetes platform: Bare-metal
• Kubernetes version: 1.14.3
• OS: Ubuntu 18.04

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.99. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[jlewi]

Can you provide the docker image that you are trying, the logs of your jupyter server, and the URL you are trying to hit?

[SpontaneousDuck]

Tested with the provided tensorflow-1.13.1-notebook-gpu:v0.5.0 and tensorflow-2.0.0a-notebook-gpu:v0.5.0 images.

So I have a bare-metal Kubernetes cluster with 1 management node and multiple gpu nodes. It looks like the first problem I had was I deployed with kfctl generate all && kfctl apply all instead of kfctl generate k8s && kfctl apply k8s. After I did that my first notebook worked. Upon creating more notebooks the logs for the first notebook showed that connections for notebook #2 and #3 I created all were routed to the first notebook and thus returned a 404 since they were to wrong path. After deleting the original notebook, requests were routed properly to the second and third notebook.

[jlewi]

Thanks @SpontaneousDuck this was a bug in the notebook controller that was fixed in kubeflow/kubeflow#4402

The fix should be in 0.7.0 which we released this morning.