Zero-to-jupyterhub-k8s: 404 error on redirect to singleuser notebook

[rafalrojek]

Hello,
I'm creating jupyterhub on AKS (Azure Kubernetes Service), with AAD (Azure Active Directory). Everything works properly until redirect to user notebook. I get 404 error. There ins't any error on login and singleuser pod initialization. As auth plugin I use oauthenticator.azuread.AzureAdOAuthenticator. I extended hub and singleuser images with PyJWT.
On hub pod I only see a redirect to notebook:

[I 2019-11-28 11:27:08.779 JupyterHub log:158] 302 GET /hub/user/rojek,%20rafa%C5%82/ -> /user/rojek,%20rafa%C5%82/?redirects=1 (rojek, rafał@89.171.167.46) 24.36ms

On notebook pod I get 404 errors:

[W 2019-11-28 11:48:46.505 SingleUserLabApp configurable:168] Config option `open_browser` not recognized
 by `SingleUserLabApp`.  Did you mean `browser`?
[I 2019-11-28 11:48:46.831 SingleUserLabApp extension:59] JupyterLab extension loaded from /opt/conda/lib/python3.6/site-packages/jupyterlab
[I 2019-11-28 11:48:46.831 SingleUserLabApp extension:60] JupyterLab application directory is /opt/conda/share/jupyter/lab
[W 2019-11-28 11:48:46.839 SingleUserLabApp labapp:215] JupyterLab server extension not enabled, manually loading...
[I 2019-11-28 11:48:46.840 SingleUserLabApp extension:59] JupyterLab extension loaded from /opt/conda/lib/python3.6/site-packages/jupyterlab
[I 2019-11-28 11:48:46.840 SingleUserLabApp extension:60] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2019-11-28 11:48:46.844 SingleUserLabApp singleuser:405] Starting jupyterhub-singleuser server version 0.9.2
[I 2019-11-28 11:48:46.852 SingleUserLabApp notebookapp:1685] Serving notebooks from local directory: /home/jovyan
[I 2019-11-28 11:48:46.852 SingleUserLabApp notebookapp:1685] The Jupyter Notebook is running at:
[I 2019-11-28 11:48:46.852 SingleUserLabApp notebookapp:1685] http://(jupyter-rojek-2c-20rafa-c5-82 or 127.0.0.1):8888/user/rojek%2C%20rafa%C5%82/?token=********
[I 2019-11-28 11:48:46.852 SingleUserLabApp notebookapp:1686] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2019-11-28 11:48:46.859 SingleUserLabApp notebookapp:1726] 

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://(jupyter-rojek-2c-20rafa-c5-82 or 127.0.0.1):8888/user/rojek%2C%20rafa%C5%82/?token=********
[I 2019-11-28 11:48:49.377 SingleUserLabApp log:158] 302 GET /user/rojek%2C%20rafa%C5%82/ -> /user/rojek%2C%20rafa%C5%82/lab? (@10.244.1.107) 1.16ms
[W 2019-11-28 11:48:49.574 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/?redirects=1 (@89.171.167.46) 41.37ms
[W 2019-11-28 11:48:49.651 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=9b2c8d3489227115310662a343fce11c (@89.171.167.46) 2.00ms
[W 2019-11-28 11:48:49.654 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=7afb461de36accb1aa133a1710f5bc56 (@89.171.167.46) 1.47ms
[W 2019-11-28 11:48:49.702 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/style/style.min.css?v=c8939f92700460241fdda8650d977cae (@89.171.167.46) 1.37ms
[W 2019-11-28 11:48:49.705 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/custom/custom.css (@89.171.167.46) 1.62ms
[W 2019-11-28 11:48:49.726 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe (@89.171.167.46) 1.57ms
[W 2019-11-28 11:48:49.729 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/proptypes/index.js?v=c40890eb04df9811fcc4d47e53a29604 (@89.171.167.46) 1.54ms
[W 2019-11-28 11:48:49.731 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/preact-compat/index.js?v=f865e990e65ad27e3a2601d8adb48db1 (@89.171.167.46) 1.64ms
[W 2019-11-28 11:48:49.733 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/preact/index.js?v=00a2fac73c670ce39ac53d26640eb542 (@89.171.167.46) 1.58ms
[W 2019-11-28 11:48:49.746 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce (@89.171.167.46) 2.22ms
[W 2019-11-28 11:48:49.785 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/proptypes/index.js?v=c40890eb04df9811fcc4d47e53a29604 (@89.171.167.46) 2.83ms
[W 2019-11-28 11:48:49.832 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/preact-compat/index.js?v=f865e990e65ad27e3a2601d8adb48db1 (@89.171.167.46) 2.36ms
[W 2019-11-28 11:48:49.880 SingleUserLabApp log:158] 404 GET /user/rojek,%20rafa%C5%82/static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce (@89.171.167.46) 2.80ms

[consideRatio]

I'm unsure whats wrong, but i suspect using a username in JupyterHub like name, surname can be causing the issue.

[rafalrojek]

Probably yes, but when we have Jupyterhub with dockerspawner we don't have this issue. Also we can't change this because we are using this AAD for many other things

[consideRatio]

Are you using the same Docker image?

[rafalrojek]

No, because dockerspawner is working on jubyterhub 1.0, but this I must use jupyterhub 0.9.6 and I use your docker image for that

[consideRatio]

Can you check if this is an issue on the latest version of this helm chart? We have jupyterhub something later than 1.0.0, i wonder if this is already fixed

[rafalrojek]

Alpha version of chart don't start on my cluster so I can't start higher version of chart

[consideRatio]

What makes it not start? Is that an issue already reported somewhere? I'm working towards a release, so this is an important question for me!

[rafalrojek]

At this moment I'm not at work but I will try check taht on monday

[consideRatio]

@rafalrojek thanks! There has been a lot of fixes since alpha.1 so the perfect insight would be to get feedback if it works on master or if there is something stopping your ability still to use this helm chart.

[rafalrojek]

Today I tried to install the latest version: jupyterhub-0.9.0-alpha.1.073.53374ee. Hub and singleuser started but the login problem still exist:

➜  k8s git:(k8s) ✗ k --namespace=jhub-test logs jupyter-rojek-2c-20rafa-c5-82 
[W 2019-12-02 07:46:31.243 SingleUserLabApp configurable:168] Config option `open_browser` not recognized by `SingleUserLabApp`.  Did you mean `browser`?
[I 2019-12-02 07:46:32.887 SingleUserLabApp extension:158] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 2019-12-02 07:46:32.887 SingleUserLabApp extension:159] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2019-12-02 07:46:34.268 SingleUserLabApp singleuser:561] Starting jupyterhub-singleuser server version 1.0.0
[I 2019-12-02 07:46:34.272 SingleUserLabApp log:174] 302 GET /user/rojek%2C%20rafa%C5%82/ -> /user/rojek%2C%20rafa%C5%82/lab? (@10.244.0.143) 1.55ms
[I 2019-12-02 07:46:34.277 SingleUserLabApp notebookapp:1825] Serving notebooks from local directory: /home/jovyan
[I 2019-12-02 07:46:34.277 SingleUserLabApp notebookapp:1825] The Jupyter Notebook is running at:
[I 2019-12-02 07:46:34.277 SingleUserLabApp notebookapp:1825] http://jupyter-rojek-2c-20rafa-c5-82:8888/user/rojek%2C%20rafa%C5%82/
[I 2019-12-02 07:46:34.277 SingleUserLabApp notebookapp:1826] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2019-12-02 07:46:34.281 SingleUserLabApp singleuser:542] Updating Hub with activity every 300 seconds
[E 2019-12-02 07:46:37.456 SingleUserLabApp web:2991] Could not open static file ''
[W 2019-12-02 07:46:37.458 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/ (@89.64.29.102) 31.56ms
[W 2019-12-02 07:46:37.533 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=3c2a865c832a1322285c55c6ed99abb2 (@89.64.29.102) 2.27ms
[W 2019-12-02 07:46:37.536 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=7afb461de36accb1aa133a1710f5bc56 (@89.64.29.102) 1.62ms
[W 2019-12-02 07:46:37.603 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/custom/custom.css (@89.64.29.102) 1.49ms
[W 2019-12-02 07:46:37.606 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/style/style.min.css?v=e91a43337d7c294cc9fab2938fa723b3 (@89.64.29.102) 1.92ms
[W 2019-12-02 07:46:37.639 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 (@89.64.29.102) 1.38ms
[W 2019-12-02 07:46:37.645 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/create-react-class/index.js?v=94feb9971ce6d26211729abc43f96cd2 (@89.64.29.102) 1.30ms
[W 2019-12-02 07:46:37.650 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe (@89.64.29.102) 1.47ms
[W 2019-12-02 07:46:37.652 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/react/react-dom.production.min.js (@89.64.29.102) 1.81ms
[W 2019-12-02 07:46:37.656 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/requirejs/require.js?v=951f856e81496aaeec2e71a1c2c0d51f (@89.64.29.102) 1.32ms
[W 2019-12-02 07:46:37.710 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 (@89.64.29.102) 1.39ms
[W 2019-12-02 07:46:37.765 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/react/react-dom.production.min.js (@89.64.29.102) 1.72ms
[W 2019-12-02 07:46:37.831 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/create-react-class/index.js?v=94feb9971ce6d26211729abc43f96cd2 (@89.64.29.102) 1.45ms
[W 2019-12-02 07:46:37.886 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/requirejs/require.js?v=951f856e81496aaeec2e71a1c2c0d51f (@89.64.29.102) 1.54ms
[W 2019-12-02 07:46:38.002 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/base/images/favicon.ico?v=97c6417ed01bdc0ae3ef32ae4894fd03 (@89.64.29.102) 2.15ms
[W 2019-12-02 07:46:55.631 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82 (@89.64.29.102) 1.70ms
[W 2019-12-02 07:46:55.703 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=3c2a865c832a1322285c55c6ed99abb2 (@89.64.29.102) 1.55ms
[W 2019-12-02 07:46:55.709 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/style/style.min.css?v=e91a43337d7c294cc9fab2938fa723b3 (@89.64.29.102) 1.50ms
[W 2019-12-02 07:46:55.712 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=7afb461de36accb1aa133a1710f5bc56 (@89.64.29.102) 1.56ms
[W 2019-12-02 07:46:55.759 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/custom/custom.css (@89.64.29.102) 1.59ms
[W 2019-12-02 07:46:55.774 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe (@89.64.29.102) 1.30ms
[W 2019-12-02 07:46:55.810 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/react/react.production.min.js?v=34f96ffc962a7deecc83037ccb582b58 (@89.64.29.102) 1.48ms
[W 2019-12-02 07:46:55.815 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/react/react-dom.production.min.js (@89.64.29.102) 1.36ms
[W 2019-12-02 07:46:55.822 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/create-react-class/index.js?v=94feb9971ce6d26211729abc43f96cd2 (@89.64.29.102) 1.62ms
[W 2019-12-02 07:46:55.824 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/requirejs/require.js?v=951f856e81496aaeec2e71a1c2c0d51f (@89.64.29.102) 1.36ms
[W 2019-12-02 07:46:55.880 SingleUserLabApp log:174] 404 GET /user/rojek,%20rafa%C5%82/static/components/requirejs/require.js?v=951f856e81496aaeec2e71a1c2c0d51f (@89.64.29.102) 2.37ms

[consideRatio]

[E 2019-12-02 07:46:37.456 SingleUserLabApp web:2991] Could not open static file ''

This is the real question i figure, what is going on there and why.

My guess is an error in the singleuserlabapp caused by the username with a space and , in it.

[consideRatio]

Note that you can use the AzureAD users etc, but mapping them to amother names if needed, or stripping away spaces etc

[consideRatio]

Id personally use their email or something, that is unique while their name isnt, so thats problematic.

[consideRatio]

I dont remember in detail how to go about it though and it may differ between authenticators, but it is doable and certainly somehong id suggest. I may not find time to resolve this edge case even though i do consider it a bug somewhere.

[konradsitarz]

This is linked with the issue https://github.com/jupyterhub/oauthenticator/issues/213.

With the AzureAD authenticator hub config setting c.AzureAdOAuthenticator.username_claim
can be set to e.g. 'unique_name'.

[rafalrojek]

That right! You can repair this issue with:

hub:
  extraConfig:
    oauthenticator: |
      c.AzureAdOAuthenticator.username_claim="unique_name"