Zero-to-jupyterhub-k8s: Connection to notebook server could not be established

[ablekh]

Hi all,

Currently, I am trying to use the _z2jh_ guide to implement and deploy a pilot JupyterHub cluster on Azure cloud (for South Big Data Hub). In certain places I had to deviate from the guide in order to move forward (I'm documenting the issues and hope to eventually update the guide accordingly, when ready). In the meantime, I am stuck at the situation, where attempts to create a new notebook result in the connection error message (see title). Below is the (hopefully) relevant output from the log. You advice / help will be appreciated. cc: @yuvipanda @choldgraf

[I 2018-03-24 07:06:22.540 SingleUserNotebookApp handlers:153] Creating new notebook in
[I 2018-03-24 07:06:22.591 SingleUserNotebookApp log:122] 201 POST /user/ablekh/api/contents ([email protected]) 52.16ms
[I 2018-03-24 07:06:23.829 SingleUserNotebookApp log:122] 200 GET /user/ablekh/api/config/notebook?_=1521875164144 ([email protected]) 2.83ms
[I 2018-03-24 07:06:23.866 SingleUserNotebookApp log:122] 200 GET /user/ablekh/api/config/common?_=1521875164145 ([email protected]) 2.24ms
[I 2018-03-24 07:06:23.929 SingleUserNotebookApp log:122] 200 GET /user/ablekh/api/kernelspecs ([email protected]) 2.22ms
[I 2018-03-24 07:06:23.978 SingleUserNotebookApp log:122] 200 GET /user/ablekh/api/contents/Untitled.ipynb?type=notebook&_=1521875164146 ([email protected]) 22.99ms
[W 2018-03-24 07:06:24.348 SingleUserNotebookApp log:122] 404 GET /user/ablekh/nbextensions/widgets/notebook/js/extension.js?v=20180324065916 ([email protected]) 2.35ms
[I 2018-03-24 07:06:24.352 SingleUserNotebookApp log:122] 200 GET /user/ablekh/api/contents/Untitled.ipynb/checkpoints?_=1521875164147 ([email protected]) 1.94ms
[I 2018-03-24 07:06:24.375 SingleUserNotebookApp kernelmanager:145] Kernel started: c5370cb8-111a-44dd-99e7-f582f8e8c6db
[I 2018-03-24 07:06:24.377 SingleUserNotebookApp log:122] 201 POST /user/ablekh/api/sessions ([email protected]) 15.50ms
[I 2018-03-24 07:06:25.108 SingleUserNotebookApp handlers:189] Adapting to protocol v5.1 for kernel c5370cb8-111a-44dd-99e7-f582f8e8c6db
[W 2018-03-24 07:06:25.110 SingleUserNotebookApp log:122] 400 GET /user/ablekh/api/kernels/c5370cb8-111a-44dd-99e7-f582f8e8c6db/channels?session_id=47102B1CD42446CD8F2379E6363504D7 ([email protected]) 386.68ms
[W 2018-03-24 07:06:26.232 SingleUserNotebookApp handlers:253] Replacing stale connection: c5370cb8-111a-44dd-99e7-f582f8e8c6db:47102B1CD42446CD8F2379E6363504D7

Snippet from config.yaml (to emphasize what single-user image I am using):

single-user:
  image:
    name: jupyter/datascience-notebook
    tag: c19283de5a6f

If it matters, I am using JupyterHub Helm chart v0.6.

[consideRatio]

@ablekh I'm not confident about interpreting the logs, but I your config.yaml should say singleuser instead of single-user. Perhaps that might solve your issue.

If not I don't know how to help you get past this issue, but I'd recommend inspecting the logs of the spawned pod (kubectl logs <podname, jupyter-erik or similar>). By the way, to make sure I get it right, is this an issue that arise when you create a new notebook within a running singleuser server, or does it occur when you are creating a new singleuser server?

[choldgraf]

hey @ablekh - more generally, posting a link to your repository / your deployment would be helpful in debugging as well.

[ablekh]

@consideRatio Thank you for your advice. I have replaced the configuration key with singleuser and it seems to improve some other things (e.g., now I properly see all available kernels - Python 3, R and Julia), however, unfortunately, it hasn't fixed the original issue. As for inspecting logs, I definitely have done that: the log snippet I shared above is from the output of the command kubectl --namespace=NAMESPACE logs jupyter-ablekh. Not sure I understand the question in your last sentence, though ... I'm not creating a single-user server manually (via CLI or such), but AFAIK rather JupyterHub is supposed to spawn one, when a user (me, in this case) creates a new notebook (via GUI).

[ablekh]

@choldgraf I understand. Currently, I don't maintain a public repo for this pilot deployment and prefer not to share links to the deployment at this time. Having said that, the following is full content of the relevant config.yaml file. Let me know what other information, if any, could be helpful for investigating this issue.

hub:
  cookieSecret: "<SECRET1>"

proxy:
  secretToken: "<SECRET2>"

prePuller:
  enabled: false

rbac:
  enabled: false

auth:
  type: github
  github:
    clientId: "<ID>"
    clientSecret: "<SECRET>"
    callbackUrl: "<URL>"
    org_whitelist:
      - "<ORG_NAME>"
  scopes:
      - "read:org"

singleuser:
  image:
    name: jupyter/datascience-notebook
    tag: c19283de5a6f

[consideRatio]

@ablekh ah now I grasp things better I think!

I'll attempt to summarize

1. You have a JupyterHub setup using z2jh
2. You login and press "start a server", you end up watching your file area
3. Then you create a new notebook, and this happens

My question was meant to figure out if you actually ended up with a successful server start before the notebook creation failed.

I don't have any ideas on how to proceed though =/

[ablekh]

@consideRatio Understood. Yep, your summary is correct. Re: _"I don't have any ideas on how to proceed though"_ -- No problem. Thank you for support, anyway. :-)

P.S. Hopefully, @choldgraf and/or @yuvipanda will share their wisdom on the matter soon.

[choldgraf]

hey @ablekh could you do the following:

1. Post the logs from your user pod again (you mention that some stuff has changed when you changed to singleuser so let's confirm that the error message is the same)
2. Post the logs from your hub pod? Error messages often flow through here as well.
3. List the main deviations that you had to take from the base guide, so we can make sure this wouldn't have messed something up.

To confirm, your jupyterhub starts up fine, as does your user session. You can see the list of files in your home directory, and when you click the button to create a new notebook, what happens next from the user's perspective?

[consideRatio]

@ablekh =D You are welcome!

[ablekh]

Hey @choldgraf - I appreciate you getting in touch so fast. Will gather relevant info and reply soon ...

[ablekh]

@choldgraf Here is the requested info:

1. See attached file SBDH_Azure_JH_Singleuser_Server_Output_032918.txt.
   (output generated by kubectl --namespace=sbdh-jh-v1-v081 logs jupyter-ablekh)
2. See attached file SBDH_Azure_JH_Hub_Server_Output_032918.txt.
   (output generated by kubectl --namespace=sbdh-jh-v1-v081 logs hub-2186441756-6zdlx)
3. Main deviations from Azure-specific guidelines were as follows:
   
   
   
   • created resource group in eastus (vs. centralus)
   
   
   • per recommendations from Microsoft people, used Azure ACS instead of Azure AKS for cluster creation
     
     (az acs create --orchestrator-type Kubernetes --name sbdh-jh-v1 --resource-group sbdh_jh_v1 --ssh-key-value ssh-key-sbdh_jh_v1.pub --output table)
   
   
   • upgraded Helm to satisfy requirements (helm init --upgrade) -- Helm Server was out-of-date
   
   
   • updated config.yaml to customize the setup (in particular, add GitHub auth; see the contents above)
   
   
   • then updated the cluster, based on the above-mentioned configuration changes
     
     (helm upgrade sbdh-jh-v1 jupyterhub/jupyterhub --version=v0.6 -f config.yaml)
   
   
   • added DNS prefix to the cluster URL for easier / more flexible access
   
   

Here is the list of pods currently existing on the cluster (kubectl --namespace=sbdh-jh-v1-v081 get pod):

NAME                                           READY     STATUS    RESTARTS   AGE
hub-2186441756-6zdlx                           1/1       Running   0          1d
jupyter-ablekh                                 1/1       Running   0          41m
proxy-2742675326-644lc                         2/2       Running   0          6d
pull-all-nodes-1521849673-sbdh-jh-v1-1-tp667   1/1       Running   0          1d

SBDH_JH_Azure_Singleuser_Server_Output_032918.txt
SBDH_JH_Azure_Hub_Server_Output_032918.txt

[consideRatio]

Writing quickly from mobile: got any jupyter extensions are installed? I
recollect breaking my notebooks like this once while fiddling with an
extension.

Also, what kernel are you starting? Tried different kernels? Same result?

On Fri, Mar 30, 2018, 04:29 Aleksandr Blekh [e-mail]
wrote:

@choldgraf https://github.com/choldgraf Here is the requested info:

1. See attached file SBDH_Azure_JH_Singleuser_Server_Output_032918.txt.
   (output generated by kubectl --namespace=sbdh-jh-v1-v081 logs
   jupyter-ablekh)
2. See attached file SBDH_Azure_JH_Hub_Server_Output_032918.txt.
   (output generated by kubectl --namespace=sbdh-jh-v1-v081 logs
   hub-2186441756-6zdlx)
3. Main deviations from Azure-specific guidelines were as follows:
   
   
   
   • created resource group in eastus (vs. centralus)
   
   
   • per recommendations from Microsoft people, used Azure ACS instead
     
     of Azure AKS for cluster creation
     
     (az acs create --orchestrator-type Kubernetes --name sbdh-jh-v1
     
     --resource-group sbdh_jh_v1 --ssh-key-value ssh-key-sbdh_jh_v1.pub --output
     
     table)
   
   
   • upgraded Helm to satisfy requirements (helm init --upgrade) --
     
     Helm Server was out-of-date
   
   
   • updated config.yaml to customize the setup (in particular, add
     
     GitHub auth; see the contents above)
   
   
   • then updated the cluster, based on the above-mentioned
     
     configuration changes
     
     (helm upgrade sbdh-jh-v1 jupyterhub/jupyterhub --version=v0.6 -f
     
     config.yaml)
   
   
   • added DNS prefix to the cluster URL for easier / more flexible
     
     access
   
   

Here is the list of pods currently existing on the cluster (kubectl
--namespace=sbdh-jh-v1-v081 get pod):

NAME READY STATUS RESTARTS AGE
hub-2186441756-6zdlx 1/1 Running 0 1d
jupyter-ablekh 1/1 Running 0 41m
proxy-2742675326-644lc 2/2 Running 0 6d
pull-all-nodes-1521849673-sbdh-jh-v1-1-tp667 1/1 Running 0 1d

SBDH_JH_Azure_Singleuser_Server_Output_032918.txt
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/files/1862499/SBDH_JH_Azure_Singleuser_Server_Output_032918.txt
SBDH_JH_Azure_Hub_Server_Output_032918.txt
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/files/1862508/SBDH_JH_Azure_Hub_Server_Output_032918.txt

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/600#issuecomment-377428255,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADqMuoZPCXEZcKS_ZKyjoKJ5-7vWRPYxks5tjZiTgaJpZM4S5oxr
.

[ablekh]

@consideRatio Thank you. No, I haven't installed any extensions (yet) - this is vanilla JupyterHub install. I tried all three available kernels (Python 3, R and Julia) with the same result ... Which leads me to believe that this issue is not kernel-related, but setup-related.

[yuvipanda]

(in the throes of a big deadline next week, but quickly jumping in)

From the logs, these line jumps out to me:

[W 2018-03-24 07:06:25.110 SingleUserNotebookApp log:122] 400 GET /user/ablekh/api/kernels/c5370cb8-111a-44dd-99e7-f582f8e8c6db/channels?session_id=47102B1CD42446CD8F2379E6363504D7 ([email protected]) 386.68ms

These make me think that the problem is somewhere in somewhere between the user and the single user server. All other requests seem to succeed except this particular one. This particular one also looks like a websocket connection to me (@minrk can correct me if I'm wrong). So I feel this is a websocket proxying / connecting issue, somewhere between the user and the singleuser server. So if we look at the things that are between the user and the singleuser server:

1. The user's browser. What browser are you using? Can you try a different one? If that doesn't work, can you look in the browser developer console to see what kinda request is being sent that is causing the 400 error? This is my current favorite culprit.
2. Some sort of firewall that blocks / mangles Websocket connections? We've run into these in the past at berkeley.
3. Does Azure's load balancer need special config for websockets? It didn't the last time I tried, and I hope that hasn't changed!
4. Configurable HTTP Proxy causing problems? It hasn't for us anywhere else, but perhaps it is here? Unsure!

I'm not sure where exactly the problem is, but I'd recommend starting looking at (1), and then going to (2).

[ablekh]

@yuvipanda Appreciate your help. Will review the questions / try recommended steps and let all know.

P.S. Do you think that my use of Azure ACS instead of AKS could result in such issue?

[yuvipanda]

That's possible! I never got Azure ACS to work, always ran into random
undebuggable issues with it.

On Fri, Mar 30, 2018 at 12:15 AM, Aleksandr Blekh [e-mail]
wrote:

@yuvipanda https://github.com/yuvipanda Appreciate your help. Will
review the questions / try recommended steps and let all know.

P.S. Do you think that my use of Azure ACS instead of AKS could result in
such issue?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/600#issuecomment-377463684,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23v0GIkuxMi9mkfft1FPLEga2Zjwyks5tjduFgaJpZM4S5oxr
.

--
Yuvi Panda T
http://yuvi.in/blog

[ablekh]

Hey, folks, I have some (_good_ and _bad_) news. After further trying various things, I recalled that I had the same/similar issue, when working on SBDH's previous pilot deployment (GKE-based). So, I decided to review my earlier related discussions and I ran across this comment of mine. Well, I tried reproducing the issue on my mobile (now Android instead of Windows Mobile) and ... no luck (reproducing) - all worked beautifully! That's, obviously, good news. The bad news is that I/we still don't understand the exact reason for the issue. I tried removing relevant cookies on my PC as well as using three different browsers (all with no success) and can't think of any potential reasons for why my PC would be so special and act in such a weird way. :confused:

[yuvipanda]

That's great and sad to hear, @ablekh!

Can you check if there's any errors in the javascript console when you try connecting with your notebook? That might shed some more light.

[ablekh]

@yuvipanda Thank you! Here is the output from JavaScript console (Firefox -> Web Developer -> Web Console):

(after a single-user notebook server is spawn)

Source map error: TypeError: NetworkError when attempting to fetch resource.
Resource URL: moz-extension://c6ce0434-6293-43b0-ba89-22640b8debe4/browser-polyfill.js
Source Map URL: browser-polyfill.js.map[Learn More]

Source map error: request failed with status 404
Resource URL: http://<APP_FQDN>/user/ablekh/static/components/preact/index.js?v=00a2fac73c670ce39ac53d26640eb542
Source Map URL: preact.min.js.map[Learn More]

Welcome to Project Jupyter! Explore the various tools available and their corresponding documentation. If you are interested in contributing to the platform, please visit the community resources section at http://jupyter.org/community.html.
main.js:67
Loaded moment locale en
bidi.js:19

Source map error: request failed with status 404
Resource URL: http://<APP_FQDN>/user/ablekh/static/components/proptypes/index.js?v=c40890eb04df9811fcc4d47e53a29604
Source Map URL: index.js.map[Learn More]

Source map error: request failed with status 404
Resource URL: http://<APP_FQDN>/user/ablekh/static/components/preact-compat/index.js?v=f865e990e65ad27e3a2601d8adb48db1
Source Map URL: preact-compat.min.js.map[Learn More] 

(upon a new notebook creation request)

Use of Mutation Events is deprecated. Use MutationObserver instead.
jquery.min.js:3
actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later...
menubar.js:304
accessing "actions" on the global IPython/Jupyter is not recommended. Pass it to your objects contructors at creation time
main.js:208
Loaded moment locale en
bidi.js:19
load_extensions 
Arguments { 0: "jupyter-js-widgets/extension", … }
utils.js:60
Session: kernel_created (a16cbaf3-382b-442b-ad3c-fa537ea7dce4)
session.js:54
Starting WebSockets: ws://<APP_FQDN>/user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22
kernel.js:459
Loading extension: jupyter-js-widgets/extension
utils.js:37
Firefox can’t establish a connection to the server at ws://<APP_FQDN>/user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22/channels?session_id=abbefe5bf03b4e4f82ae837c101fbc86.
28ae1224-b119-4fdf-acce-edcdde11c372:164:43
Kernel: kernel_disconnected (50773b2f-6359-473d-b224-c6a47a5bfc22)
kernel.js:103
WebSocket connection failed:  ws://<APP_FQDN>/user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22 true
kernel.js:544
Connection lost, reconnecting in 1 seconds.
kernel.js:562
Kernel: kernel_reconnecting (50773b2f-6359-473d-b224-c6a47a5bfc22)
kernel.js:103
Starting WebSockets: ws://<APP_FQDN>/user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22
kernel.js:459
Firefox can’t establish a connection to the server at ws://<APP_FQDN>/user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22/channels?session_id=abbefe5bf03b4e4f82ae837c101fbc86.
28ae1224-b119-4fdf-acce-edcdde11c372:164:43
Kernel: kernel_disconnected (50773b2f-6359-473d-b224-c6a47a5bfc22)
kernel.js:103
WebSocket connection failed:  ws://<APP_FQDN>/user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22 true
kernel.js:544
Connection lost, reconnecting in 2 seconds.
kernel.js:562

[yuvipanda]

ok, so that's clearly a websocket issue. I'm unsure where it is coming from
tho...

Can you try on a different computer? On a different network?

On Sun, Apr 1, 2018 at 3:35 PM, Aleksandr Blekh [e-mail]
wrote:

@yuvipanda https://github.com/yuvipanda Thank you! Here is the output
from JavaScript console (Firefox -> Web Developer -> Web Console):

(after a single-user notebook server is spawn)

Source map error: TypeError: NetworkError when attempting to fetch resource.
Resource URL: moz-extension://c6ce0434-6293-43b0-ba89-22640b8debe4/browser-polyfill.js
Source Map URL: browser-polyfill.js.map[Learn More]

Source map error: request failed with status 404
Resource URL: http:///user/ablekh/static/components/preact/index.js?v=00a2fac73c670ce39ac53d26640eb542
Source Map URL: preact.min.js.map[Learn More]

Welcome to Project Jupyter! Explore the various tools available and their corresponding documentation. If you are interested in contributing to the platform, please visit the community resources section at http://jupyter.org/community.html.
main.js:67
Loaded moment locale en
bidi.js:19

Source map error: request failed with status 404
Resource URL: http:///user/ablekh/static/components/proptypes/index.js?v=c40890eb04df9811fcc4d47e53a29604
Source Map URL: index.js.map[Learn More]

Source map error: request failed with status 404
Resource URL: http:///user/ablekh/static/components/preact-compat/index.js?v=f865e990e65ad27e3a2601d8adb48db1
Source Map URL: preact-compat.min.js.map[Learn More]

(upon a new notebook creation request)

Use of Mutation Events is deprecated. Use MutationObserver instead.
jquery.min.js:3
actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later...
menubar.js:304
accessing "actions" on the global IPython/Jupyter is not recommended. Pass it to your objects contructors at creation time
main.js:208
Loaded moment locale en
bidi.js:19
load_extensions
Arguments { 0: "jupyter-js-widgets/extension", … }
utils.js:60
Session: kernel_created (a16cbaf3-382b-442b-ad3c-fa537ea7dce4)
session.js:54
Starting WebSockets: ws:///user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22
kernel.js:459
Loading extension: jupyter-js-widgets/extension
utils.js:37
Firefox can’t establish a connection to the server at ws:///user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22/channels?session_id=abbefe5bf03b4e4f82ae837c101fbc86.
28ae1224-b119-4fdf-acce-edcdde11c372:164:43
Kernel: kernel_disconnected (50773b2f-6359-473d-b224-c6a47a5bfc22)
kernel.js:103
WebSocket connection failed: ws:///user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22 true
kernel.js:544
Connection lost, reconnecting in 1 seconds.
kernel.js:562
Kernel: kernel_reconnecting (50773b2f-6359-473d-b224-c6a47a5bfc22)
kernel.js:103
Starting WebSockets: ws:///user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22
kernel.js:459
Firefox can’t establish a connection to the server at ws:///user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22/channels?session_id=abbefe5bf03b4e4f82ae837c101fbc86.
28ae1224-b119-4fdf-acce-edcdde11c372:164:43
Kernel: kernel_disconnected (50773b2f-6359-473d-b224-c6a47a5bfc22)
kernel.js:103
WebSocket connection failed: ws:///user/ablekh/api/kernels/50773b2f-6359-473d-b224-c6a47a5bfc22 true
kernel.js:544
Connection lost, reconnecting in 2 seconds.
kernel.js:562

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/600#issuecomment-377822139,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB23ojFGZLkG06XGK_8hDJzUeaEtByLks5tkVY7gaJpZM4S5oxr
.

--
Yuvi Panda T
http://yuvi.in/blog

[ablekh]

@yuvipanda Upon seeing the output, I agree. As for trying on different computer/network, currently I don't have access to different equipment, except for my tablet and smartphone, where the latter, I believe, can be considered a different computer on a different network ...

[ablekh]

Additional issue (unrelated, mentioning here for convenience) is that spawning a single-user notebook server on our current (bare-bones) Azure configuration takes IMO way too long time (over a minute). If I remember correctly, the same action has been quite fast on GKE. Is there something relevant that is specific to Azure?

[yuvipanda]

I think:

prePuller:
  enabled: false

Might be part of the reason? The image pre-puller pulls the images on all nodes to make user startup faster.

Can you open a different issue for it? If you can provide the output of kubectl --namespace=<namespace> describe pod <user-pod-name> in it that would help.

[ablekh]

@yuvipanda Sure, will open a separate issue. Thank you for the preliminary advice.

[consideRatio]

Note that the values.yaml for prepulling was updated from v0.5 to v0.6!

# correct for v0.6
prePuller:
  hook:
    enabled: true

[ablekh]

@consideRatio Appreciate the clarification.

[ablekh]

Hey people, good news! I was improving our pilot Azure-based JupyterHub cluster tonight and decided to secure the connectivity by enabling HTTPS. I have chosen to follow the z2jh guide and updated the cluster to use Let's Encrypt (BTW, do I need to do anything to setup certificate auto-renewal?). Anyway, it went pretty well and after the cluster upgrade, I have soon enjoyed seeing the green lock, as expected. :+1:

So, what is the good news, you ask. Well, apparently, enabling TLS security on the cluster resolved the problem initiated this issue / discussion thread. Yes, _Web sockets_ now work fine on my PC and I tested the setup by creating notebooks for all three currently available kernels. Yay! :-) I plan to close the issue soon, however, before doing that, I have a couple of quick questions for everyone willing to help.

1. Running kubectl --namespace=sbdh-jh-v1-v081 get pod results in the following output. What do all those pull-all-nodes-... lines with the Error status mean? What are the reasons and relevant fixes?

NAME                                           READY     STATUS      RESTARTS   AGE
hub-2186441756-6zdlx                           1/1       Running     0          19d
pre-puller-1522220422-sbdh-jh-v1-6-pq3p0       0/1       Completed   0          19d
proxy-677095668-qdtbf                          3/3       Running     0          10m
pull-all-nodes-1521849673-sbdh-jh-v1-1-29hv7   0/1       Error       0          23d
pull-all-nodes-1521849673-sbdh-jh-v1-1-344nq   0/1       Error       0          22d
pull-all-nodes-1521849673-sbdh-jh-v1-1-4g2lw   0/1       Error       0          22d
pull-all-nodes-1521849673-sbdh-jh-v1-1-53vr3   0/1       Error       0          21d
pull-all-nodes-1521849673-sbdh-jh-v1-1-6gwv7   0/1       Error       0          21d
pull-all-nodes-1521849673-sbdh-jh-v1-1-6tq3b   0/1       Error       0          12d
pull-all-nodes-1521849673-sbdh-jh-v1-1-8slht   0/1       Error       0          15d
pull-all-nodes-1521849673-sbdh-jh-v1-1-b6zx1   0/1       Error       0          12d
pull-all-nodes-1521849673-sbdh-jh-v1-1-c27xn   0/1       Error       0          22d
pull-all-nodes-1521849673-sbdh-jh-v1-1-d5vrj   0/1       Error       0          16d
pull-all-nodes-1521849673-sbdh-jh-v1-1-ff3nf   0/1       Error       0          22d
pull-all-nodes-1521849673-sbdh-jh-v1-1-gn0qt   0/1       Error       0          22d
pull-all-nodes-1521849673-sbdh-jh-v1-1-hh44c   0/1       Error       0          20d
pull-all-nodes-1521849673-sbdh-jh-v1-1-hmw9z   0/1       Error       0          12d
pull-all-nodes-1521849673-sbdh-jh-v1-1-mjvzc   0/1       Error       0          12d
pull-all-nodes-1521849673-sbdh-jh-v1-1-s9r76   0/1       Error       0          12d
pull-all-nodes-1521849673-sbdh-jh-v1-1-smrvz   0/1       Error       0          14d
pull-all-nodes-1521849673-sbdh-jh-v1-1-tp667   0/1       Error       0          19d
pull-all-nodes-1521849673-sbdh-jh-v1-1-v6cnz   0/1       Error       0          15d
pull-all-nodes-1521849673-sbdh-jh-v1-1-vbtbk   0/1       Error       0          11d
pull-all-nodes-1521849673-sbdh-jh-v1-1-vmp24   0/1       Error       0          12d
pull-all-nodes-1521849673-sbdh-jh-v1-1-zjds7   1/1       Running     0          4d

1. When I start a new terminal session on my JupyterHub instance, the window opens in a new tab with the following message in the first line: groups: cannot find name for group ID 1000 (the second line is the expected prompt jovyan@jupyter-ablekh:~$). Based on the message, it seems that currently used JupyterHub Docker image references a group with ID 1000 (or a user belonging to such group), which apparently doesn't have a name on the system. Any thoughts? Related info: https://github.com/jupyter/docker-stacks/issues/188

[consideRatio]

@ablekh nice! You don't need to setup anything else for automatic renewals - it will be handled! I think we should update the docs to make that clear. Regarding 1. and 2. - writing quickly before going to work!

1. Woooh, I don't know about the error, can you see what kubectl decsribe pod pull... and kubectl logs pull...? I think this issue, whatever it is, would go away with Helm 2.8.2 + a recent version of z2jh.

2. Yepp! This is also fixed this as well! You can update your chart values under singleuser, there is a uid entry i recall.

[ablekh]

@consideRatio Much appreciate your prompt reply and clarifications. Have a great week! :-)

[ablekh]

@consideRatio _"You can update your chart values under singleuser, there is a uid entry i recall."_ Could you clarify this sentence or point in the right direction?

[ablekh]

@minrk Do you have any thoughts on why enabling HTTPS might fix a Web sockets issue? Just curious ...

[minrk]

Were you using HTTPS without trusted certificates? If so, there's a known issue where websockets on untrusted SSL connections will be refused.

If not, there might be a bug in the ingress somewhere not proxying websockets correctly, and it's not setup right with http, but it does the right thing with the https ingress enabled, which uses a different ingress setup.

[ablekh]

@minrk No, at the time I was not using HTTPS at all (simply hasn't had a chance to enable it at that point). Appreciate your explanation and quick reply. :-)

[consideRatio]

@ablekh we made sure to use groupId of 100 instead of 1000 quite recently. That resolved the bothering message when opening terminals.

You can either use a 0.7 prerelease version of the helm chart or update your config.yaml to set the gid (under singleuser, not under the hub).

I dont fully remember the name of the value, but perhaps it was "singleuser.fsGid"

/ from mobile, vacation in france, hope that helps

[ablekh]

@consideRatio Appreciate the reply - it certainly helps. Will update my config.yaml. Have a nice vacation! :-)

[bebosudo]

I was having issues with z2jh after a datacenter migration, and the reason was the lack of a proxy for websocket, which I discovered by investigating the web console.
Thanks for your comments!