Jupyterlab: JupyterHub+JupyterLab+JupyterEnterpriseGateway Restart Kernel hangs

Created on 9 Mar 2020 · 49Comments · Source: jupyterlab/jupyterlab

Description

Happens while using JupyterHub + Jupyterlab + JupyterEnterpriseGateway (remote kernels)

After kernel restart; connection to jupyterlab frontend is lost,
cannot see any output and the cell status gets stuck as [*].

The issue only happens with remote kernels
I'm opening this issue here because the problem seems to be only with jupyterlab. Restarting works fine on jupyter notebooks (/tree).

Logs

The connection to kernel doesn't look to like the problem (after restart i can send interrupt and shutdown)
I don't see any errors on the jupyterhub logs it looks like everything is okay... (below)

[D 2020-03-09 12:53:25.945 SingleUserNotebookApp managers:403] Kernel retrieved: {'id': 'f11f3af0-b332-4f2d-b30f-075229913f46', 'name': 'python3_docker', 'last_activity': '2020-03-09T03:53:25.893675Z', 'execution_state': 'starting', 'connections': 1}
....
Request kernel at: http://127.0.0.1:8888/api/kernels/f11f3af0-b332-4f2d-b30f-075229913f46
[D 2020-03-09 12:53:35.989 SingleUserNotebookApp managers:403] Kernel retrieved: {'id': 'f11f3af0-b332-4f2d-b30f-075229913f46', 'name': 'python3_docker', 'last_activity': '2020-03-09T03:53:26.185178Z', 'execution_state': 'idle', 'connections': 1}

and here is JupyterEnterpriseGateway log showing connection established

My guess on what the problem is;

when restarting execution_state goes from starting to idle (because in remote kernels when restarting kernel is killed then started again ) but @jupyterlab/services implementation expects execution_state restarting ?

Related Issues:

I saw that the enterprise gateway related implementations are in @jupyterlab/services. Did some digging and these caught my eye as related:
This comment
Same issue was reported before on JupyterEnterpriseGateway This issue

Reproduce

I'm using JupyterHub + Jupyterlab like ;

jupyterhub-singleuser
c.Spawner.default_url = '/lab'

Launch EnterpriseGateway python docker kernel (or any other remote kernel)
start notebook -> it works
restart notebook -> wait -> gets stuck as busy

Versions:
Ubuntu 18.04
Jupyterhub 1.1.0
JupyterLab 1.2.7 (also fails with 2.0.0)
EnterpriseGateway (2.0.0)

services Bug

Source

cceyda

👍1

Most helpful comment

FYI, #9388 is now released in jlab 3.0.0rc11

Thank you Jason!

jupyter_server 1.0.8, containing the fix from https://github.com/jupyter-server/jupyter_server/pull/350 (thank you @ricklamers!), is also available.

Thanks to everyone who worked on this issue 😄

Yes, definitely - thank you!

kevin-bates on 3 Dec 2020

👍2

All 49 comments

Disclaimer: I don't have a lot of experience with JupyterHub and enterprise gateway.

@jupyterlab/services implementation expects execution_state restarting ?

IIRC, the notebook server sends a restarting message when the notebook itself decides to restart the kernel. I don't think services is waiting for a restart message if it has initiated the restart.

Do you know the easiest way to get access to a setup like this to try to debug it better?

jasongrout on 9 Mar 2020

JupyterhHub and enterprise gateway both provide docker installations but you would still need to do basic configuration. I unfortunately don't have another simpler way to setup a debug environment.
PS. yes I have tested with jupyter 2.0 but the issue persists

cceyda on 10 Mar 2020

@snu-ceyda please check you jupyterlab log, i have run with the three project long time, and it works well.

SolarisYan on 13 Mar 2020

There are no errors on the jupyterhub log, which is running jupyterlab as SingleUserNotebookApp
(as i mention in issue description)

cceyda on 15 Mar 2020

Give a couple days to re-create my hub test env and I will take a look at this.

lresende on 27 Mar 2020

Hi, we've been getting some visits to the corresponding EG issue: https://github.com/jupyter/enterprise_gateway/issues/756

@snu-ceyda - are you in a position to launch jupyter notebook from hub, try the restart using '/tree', then retry using '/lab'?

I don't see this issue with jupyter notebook but do with jupyter lab.

I think your hunch about execution_state has merit and I'll take a look - thanks for pointing that out. My understanding was that restarts were implemented using shutdown/startup and, aside from the very initial starting state introduced in the server, I don't recall seeing anywhere that the server also sets a restarting state similarly, but I'll keep an eye open for that.

kevin-bates on 16 Apr 2020

I suspect the difference here (from local and remote kernels) is that Lab may be assuming the kernel's ports remain constant across restarts and, as such, there's no need to re-establish the websocket that handles the kernel's messaging.

In looking at the code bases for Notebook and Lab front-ends relative to restarts, it appears that Notebook's restart, stops the channels, then, on successful restart, calls _kernel_created() which starts the channels thereby handling the case where the ports may have changed.

Whereas there's no such attempt in the Lab restart logic and this is why an explicit reconnect then triggers the expected interaction with the (remote) kernel.

Unfortunately, I don't know this code model well enough to try things out, but would be happy to verify any changes should someone take this on. It seems like a call to the private method _reconnect() following the successful restart RPC is what needs to happen.

@lresende, @blink1073, @jasongrout - would any of you be able to confirm this and perhaps provide a prototype? I'm able to build lab, so providing the file(s) should be sufficient (and greatly appreciated).

kevin-bates on 18 Apr 2020

I'm afk today, but we have logic to detect whether we're running on jhub elsewhere, we could use that to choose a proper restart method.

blink1073 on 18 Apr 2020

The old restart logic is here. Jhub detection logic example is here

blink1073 on 18 Apr 2020

Thanks for the responses Steve. However, this issue is completely independent of JupyterHub and really a function of a kernel's restart recreating the underlying 0MQ ports. So, in that sense, we could technically eliminate EG from the title as well.

I'm not sure what providing the previous Lab restart logic is for - it doesn't appear to reconnect following a restart either. I suspect this has been an issue with these kinds of kernels in Lab in general.

Since restart == shutdown + startup and Lab establishes a connection following a successful startup, why doesn't Lab also establish a connection following a successful restart (similar to Notebook)?

kevin-bates on 19 Apr 2020

Ah, you are correct. Yes, I think it makes sense in general to reconnect during a restart. We should call the public reconnect, since it handles the connection state and reconnect attempts. Basically, add await this.reconnect() to the end of restart().

blink1073 on 21 Apr 2020

There are two connections we are talking about here. There is the lab frontend connecting to the server over websocket, and there is the server connecting to the kernel over 0mq. During a restart, I'm thinking what should happen is that the server maintains the websocket connection while restarting the 0mq connection, thus the jlab frontend does not need to reconnect its websocket.

Where does that logic go wrong, even if the 0mq ports change?

jasongrout on 21 Apr 2020

Thank you both for your responses.

To Jason's question, I'm not certain that logic exists in the server. This is purely based on observations and a lot of time looking around (although I'll admit this portion of things isn't a bailiwick of mine). What also seems to support this is that if I force new ports on local kernels using only notebook (i.e., not coupled with EG), I can reproduce the issue. When EG is configured, there's an extra layer of "connection" going on due to the forwarding that takes place to EG, but it does include some WebSocket management that was contributed.

My understanding is that it's the ZMQChannelsHandler that establishes the connection with the ZMQ ports (see create_stream()), but there's nothing that originates within the server that does the same. Also, since the Notebook UI is clearly issuing a reconnect on restarts, and the server needs to support both clients, I'm not sure adding this kind of thing into the server (while it might make sense) is what should happen.

kevin-bates on 21 Apr 2020

Looking into the notebook code, my logic may be wrong in that it appears that the zmq channel is connected to at the start of the websocket handler connection, and is not refreshed when the kernel is restarted.
create_stream is only called from open:
https://github.com/jupyter/notebook/blob/7de8df4dd80f4da03066ddb498d1541182ad19a8/notebook/services/kernels/handlers.py#L123
https://github.com/jupyter/notebook/blob/7de8df4dd80f4da03066ddb498d1541182ad19a8/notebook/services/kernels/handlers.py#L256

Edit: sounds like I came to basically the same conclusion as @kevin-bates above

jasongrout on 21 Apr 2020

There still is a problem in the server, though. It's making the assumption that the zmq channels don't change. For example, when the last connection is dropped, buffering is enabled and it stores the zmq channels that should be buffered. However, opening a new connection with the same session (e.g., the client tries to reconnect) restores the same zmq channels. Instead, it should not assume that the zmq channels are the same, and it should replay the messages it has, then connect to the channels afresh.

jasongrout on 21 Apr 2020

👍1

Diving a bit more deeply, restarting a kernel definitely should not assume that the same connection information is valid. Note the newports option in the jupyter_client kernel manager implementation: https://github.com/jupyter/jupyter_client/blob/ea823fd84b4f5db0a050457b9b2cd7619e9a2d53/jupyter_client/manager.py#L381

jasongrout on 21 Apr 2020

Instead, it should not assume that the zmq channels are the same, and it should replay the messages it has, then connect to the channels afresh.

I agree. So currently buffered messages are likely not getting replayed between kernel shutdown and its restart (on new ports).

Yes, I believe the newports parameter only comes into play on automatic-restarts (when the server detects that the process associated with the kernel has stopped responding to poll requests), but not on manually invoked restarts at the application level. I had to change its default to True, among other things, to jury-rig this issue's reproduction using local kernels.

kevin-bates on 21 Apr 2020

I was able to reproduce this by setting the newports to default to True and also setting the jupyter_client manager.py cache_ports to default to False. This made sure that restarting a kernel came up with new port numbers.
It's a bit tricky how we should handle things in jlab, but I think a start is:

diff --git a/packages/services/src/kernel/default.ts b/packages/services/src/kernel/default.ts
index 6454c9686..b35a70559 100644
--- a/packages/services/src/kernel/default.ts
+++ b/packages/services/src/kernel/default.ts
@@ -982,10 +982,15 @@ export class KernelConnection implements Kernel.IKernelConnection {
     this._clearKernelState();
     this._updateStatus('restarting');

-    // Kick off an async kernel request to eventually reset the kernel status.
-    // We do this with a setTimeout so that it comes after the microtask
-    // logic in _handleMessage for restarting/autostarting status updates.
+    // Reconnect to a new websocket and kick off an async kernel request to
+    // eventually reset the kernel status. We do this with a setTimeout so
+    // that it comes after the microtask logic in _handleMessage for
+    // restarting/autostarting status updates.
     setTimeout(() => {
+      // We must reconnect since the kernel connection information may have
+      // changed, and the server only refreshes its zmq connection when a new
+      // websocket is opened.
+      void this.reconnect(); 
       void this.requestKernelInfo();
     }, 0);
   }

This runs into the problem I noted above, though, namely that a reconnect with the same client session id pulls up the old channels, not the new ones, so I think that may need to be fixed as well.

jasongrout on 22 Apr 2020

Thanks for looking at this so carefully Jason!

In looking at this closer, I suspect the server-side portion of this is one of those dumb-luck scenarios. When the kernel management requests are forwarded to a gateway server, the self.session.session value is recreated (IIRC). This is why we don't see message replays on gateway servers. As a result, when the get_buffer() call is made during the open of the "reconnect", the stop_buffering() branch of that method is taken because the current session key (prefixed by the same kernel_id) doesn't match the session key in the buffer. This results in the channels getting recreated via create_stream() in the else-portion of the buffer replay. This is even the case for local kernels spawned from a gateway server.

I'll have to figure out how best to change this, both relative to gateway and relative to underlying port changes across restarts. That said, I think the server and lab issues can be handled independently and a fix from lab will restore expected operations to gateway users - of which more and more are starting to move to Lab - need this. Is there an ETA for Lab's next patch release?

Regarding replay of buffered messages across kernel process restarts - what is the use case for that? I ask that because I wonder if restart should behave similar to shutdown and stop buffering. Shutdown stops buffering via the MappingKernelManager, while MappingKernelManager's restart method is implemented as KernelManager.shutdown + KernelManager.start and I wonder if MappingKernelManager restart shouldn't just stop buffering anyway.

kevin-bates on 22 Apr 2020

Thanks for engaging at a deep level yourself.

The session in question is IIRC the client session id, which is controlled by the client (unless gateway is doing some proxying here or something, and acting as the client?).
The patch can't be deployed yet because of the problem with the server. Roughly, what happens is that jlab requests a restart, and that restart happens, and then jlab disconnects and creates a new websocket connection, but with the same client id. The bad code on the server means that we never get connected to the new ports. I have another patch that seems to take care of the problem on the server, for notebook/services/kernels/handler.py:

    def open(self, kernel_id):
        super(ZMQChannelsHandler, self).open()
        km = self.kernel_manager
        km.notify_connect(kernel_id)

        # on new connections, flush the message buffer
        buffer_info = km.get_buffer(kernel_id, self.session_key)
        if buffer_info and buffer_info['session_key'] == self.session_key:
            self.log.info("Restoring connection for %s", self.session_key)
            # self.channels = buffer_info['channels']
            replay_buffer = buffer_info['buffer']
            if replay_buffer:
                self.log.info("Replaying %s buffered messages", len(replay_buffer))
                for channel, msg_list in replay_buffer:
                    stream = buffer_info['channels']
                    # stream = self.channels[channel]
                    self._on_zmq_reply(stream, msg_list)

        try:
            # The channel addresses may have changed (the kernel may have
            # restarted), so always reconnect.
            self.create_stream()
        except web.HTTPError as e:
            self.log.error("Error opening stream: %s", e)
            # WebSockets don't response to traditional error codes so we
            # close the connection.
            for channel, stream in self.channels.items():
                if not stream.closed():
                    stream.close()
            self.close()
            return

        km.add_restart_callback(self.kernel_id, self.on_kernel_restarted)
        km.add_restart_callback(self.kernel_id, self.on_restart_failed, 'dead')

        for channel, stream in self.channels.items():
            stream.on_recv_stream(self._on_zmq_reply)

jasongrout on 22 Apr 2020

Regarding replay of buffered messages across kernel process restarts - what is the use case for that? I ask that because I wonder if restart should behave similar to shutdown and stop buffering. Shutdown stops buffering via the MappingKernelManager, while MappingKernelManager's restart method is implemented as KernelManager.shutdown + KernelManager.start and I wonder if MappingKernelManager restart shouldn't just stop buffering anyway.

The message buffering is just a kludgy way of getting some sort of continuity of messages across disconnects. I can imagine having a long-running process that is printing output, I close my laptop and go home, and at some point it gets restarted (perhaps it runs out of memory and is auto-restarted, or perhaps I somehow connect some other way and restart)? It's kind of nice if I can still have access to all of the messages stored in the buffer to see where the error happened, or how far I got. I think the message buffering is independent of the restarts.

jasongrout on 22 Apr 2020

Hi Jason, I probably can get to this until tomorrow (due to scheduled commitments).

I can imagine having a long-running process that is printing output, I close my laptop and go home, and at some point it gets restarted (perhaps it runs out of memory and is auto-restarted, or perhaps I somehow connect some other way and restart)? It's kind of nice if I can still have access to all of the messages stored in the buffer to see where the error happened, or how far I got.

I thought that a new session key would be generated if even the tab of the browser (associated with the notebook) was closed and this buffer replay logic was purely for network interruptions between the browser and server (and perhaps the server and kernel I suppose - since the replay probably knows where things are going).

kevin-bates on 22 Apr 2020

@jasongrout - I apologize for not getting back to this sooner. (I finally managed to paddle out past the waves.)

I've tried out both of the changes you propose and they appear to work fine. The notebook changes make a lot of sense to me, as does the lab change.

Did you want to create PRs for both?

kevin-bates on 29 Apr 2020

@kevin-bates - could you create the notebook PR and I'll do the Lab PR?

jasongrout on 15 May 2020

PR in lab at #8432.

jasongrout on 15 May 2020

Fantastic - will do - thank you Jason!

kevin-bates on 15 May 2020

@Zsailer - ideally the notebook server would keep the same websocket connection even if the zmq ports change for the kernel (on a kernel restart, for example). We're working around this for now because this architecture of the websocket bridge is pretty hardcoded in the server.

Just CCing you as a heads-up since you are working a lot on the next-generation server.

jasongrout on 16 May 2020

Hi,
We are observing the same issue even after taking jupyter/notebook#5450 and #8432.. we are using jupyterlab 2.2.2 and notebook 6.0.3 versions. Pls suggest.

suryag10 on 19 Aug 2020

If you're expecting to have buffer replays work with notebook:5450 and lab:8432 and EG - it's more involved than what these PRs address since the session key doesn't propagate to EG (iirc). However, taking 8432 should enable the websocket reconnection to take place on restarts - providing functional parity with Notebook UI (which wasn't the case previously). That functionality should also work with EG configured.

kevin-bates on 20 Aug 2020

Hi kevin,
Thx for the response and support. I had taken only #8432, but its the same behaviour. Once the kernel is restarted, we will not be able to execute any cells, unless we refresh the webpage. We are using EG 2.1.1, jupyterlab 2.2.2, notebook 6.0.3, jupyterhub 1.1.0.

suryag10 on 20 Aug 2020

ok - I'm unable to look at this until next week as I'm trying to take some time off, but will try to spend some time with it then. You should see a websocket connection established following the restart - similar to what you see using Notebook - if using Lab with 8432. I would run EG (and Notebook) with --debug to ensure that logging occurs (it should appear in the EG log).

kevin-bates on 20 Aug 2020

Thx Kevin, Will try to debug more and update the status.

suryag10 on 20 Aug 2020

More info,
when restart kernel is done on native notebook, there is connection close happening and a new connection being established. See below:

But this is not happening when done with jupyterlab:

{"time":"2020-08-23T20:10:17.216Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:17.217Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request list kernels: https://eg.cjup-surya:8888/api/kernels"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:17.260Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/kernels?1598213417068 ([email protected]) 44.89ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:18.078Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:18.078Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request kernel at: https://eg.cjup-surya:8888/api/kernels/481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:18.101Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Kernel retrieved: {'id': '481dabf9-c4a8-4211-875f-555d6806a37a', 'name': 'python_kubernetes', 'last_activity': '2020-08-23T20:10:13.090691Z', 'execution_state': 'idle', 'connections': 2}"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:18.101Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"RemoteKernelManager.kernel_model: 481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:18.101Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request kernel at: https://eg.cjup-surya:8888/api/kernels/481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:18.121Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Kernel retrieved: {'id': '481dabf9-c4a8-4211-875f-555d6806a37a', 'name': 'python_kubernetes', 'last_activity': '2020-08-23T20:10:13.090691Z', 'execution_state': 'idle', 'connections': 2}"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:18.122Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/sessions?1598213417931 ([email protected]) 44.81ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:20.187Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:20.188Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request restart kernel at: https://eg.cjup-surya:8888/api/kernels/481dabf9-c4a8-4211-875f-555d6806a37a/restart"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:21.465Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:21.479Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/metrics?1598213421321 ([email protected]) 14.26ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:23.420Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:23.441Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/contents/?content=1&1598213423279 ([email protected]) 22.09ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:24.870Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:24.871Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/terminals?1598213424715 ([email protected]) 1.77ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:27.503Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:27.504Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request list kernels: https://eg.cjup-surya:8888/api/kernels"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:27.529Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Restart kernel response: 200 OK"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:27.530Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"RemoteKernelManager.kernel_model: 481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:27.530Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request kernel at: https://eg.cjup-surya:8888/api/kernels/481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:27.540Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/kernels?1598213427360 ([email protected]) 37.39ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:27.550Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Kernel retrieved: {'id': '481dabf9-c4a8-4211-875f-555d6806a37a', 'name': 'python_kubernetes', 'last_activity': '2020-08-23T20:10:27.526619Z', 'execution_state': 'idle', 'connections': 2}"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:27.551Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 POST /jupyter/user/cjup/api/kernels/481dabf9-c4a8-4211-875f-555d6806a37a/restart?1598213420044 ([email protected]) 7364.40ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:27.824Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:27.826Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/static/base/images/favicon-busy-1.ico ([email protected]) 2.13ms"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:28.438Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:28.439Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request kernel at: https://eg.cjup-surya:8888/api/kernels/481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:28.473Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Kernel retrieved: {'id': '481dabf9-c4a8-4211-875f-555d6806a37a', 'name': 'python_kubernetes', 'last_activity': '2020-08-23T20:10:27.526619Z', 'execution_state': 'idle', 'connections': 2}"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:28.473Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"RemoteKernelManager.kernel_model: 481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:28.474Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request kernel at: https://eg.cjup-surya:8888/api/kernels/481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:28.491Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Kernel retrieved: {'id': '481dabf9-c4a8-4211-875f-555d6806a37a', 'name': 'python_kubernetes', 'last_activity': '2020-08-23T20:10:27.526619Z', 'execution_state': 'idle', 'connections': 2}"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:28.492Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/sessions?1598213428293 ([email protected]) 54.93ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:33.944Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:33.960Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/contents/?content=1&1598213433803 ([email protected]) 16.09ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:35.115Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:35.116Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/terminals?1598213434974 ([email protected]) 1.79ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:37.831Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:37.831Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request list kernels: https://eg.cjup-surya:8888/api/kernels"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:37.862Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/kernels?1598213437688 ([email protected]) 32.13ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:38.730Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:38.731Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request kernel at: https://eg.cjup-surya:8888/api/kernels/481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:38.752Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Kernel retrieved: {'id': '481dabf9-c4a8-4211-875f-555d6806a37a', 'name': 'python_kubernetes', 'last_activity': '2020-08-23T20:10:27.526619Z', 'execution_state': 'idle', 'connections': 2}"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:38.752Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"RemoteKernelManager.kernel_model: 481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:38.752Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Request kernel at: https://eg.cjup-surya:8888/api/kernels/481dabf9-c4a8-4211-875f-555d6806a37a"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:38.770Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Kernel retrieved: {'id': '481dabf9-c4a8-4211-875f-555d6806a37a', 'name': 'python_kubernetes', 'last_activity': '2020-08-23T20:10:27.526619Z', 'execution_state': 'idle', 'connections': 2}"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:38.771Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/sessions?1598213438588 ([email protected]) 41.69ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:44.217Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:44.225Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/contents/?content=1&1598213444070 ([email protected]) 9.21ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:45.368Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:45.369Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/terminals?1598213445218 ([email protected]) 1.80ms"},"type":"log","level":"INFO"}
{"time":"2020-08-23T20:10:45.563Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"Allowing whitelisted Hub user cjup"},"type":"log","level":"DEBUG"}
{"time":"2020-08-23T20:10:45.571Z","timezone":"UTC","host":"jupyterhub-cjup","systemid":"2106a117733f42d697284fbc54927928","log":{"message":"200 GET /jupyter/user/cjup/api/contents/?content=1&1598213445421 ([email protected]) 8.

suryag10 on 23 Aug 2020

Hi @suryag10 - I just built #8432 and the kernel is completely functional across a restart (unlike prior to this change). You might not be building the change completely. Here's my history (with non-applicable commands removed):

  516  git checkout -b reconnectrestart jasongrout/reconnectrestart 
  518  conda create -n jupyterlab-dev python=3.8 jupyter
  519  conda activate jupyterlab-dev
  520  pip install -e .
  521  jlpm install
  522  jlpm run build
  523  jlpm run build:core
  524  jupyter lab build
  526  jupyter lab --dev-mode --debug --notebook-dir=~/notebooks --gateway-url=http://yarn-eg-node-1.fyre.ibm.com:8888

Here is the output from the Lab side:

[D 11:33:25.978 LabApp] Restart kernel response: 200 OK
[D 11:33:25.978 LabApp] RemoteKernelManager.kernel_model: c3b43bd3-0311-4927-80a2-ea12a1fc9877
[D 11:33:25.978 LabApp] Request kernel at: http://yarn-eg-node-1.fyre.ibm.com:8888/api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877
[D 11:33:26.124 LabApp] Kernel retrieved: {'id': 'c3b43bd3-0311-4927-80a2-ea12a1fc9877', 'name': 'spark_python_yarn_cluster', 'last_activity': '2020-08-24T18:33:25.931571Z', 'execution_state': 'starting', 'connections': 1}
[D 11:33:26.125 LabApp] 200 POST /api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877/restart?1598293993424 (::1) 12697.73ms
[D 11:33:26.171 LabApp] Closing websocket connection /api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877/channels
[D 11:33:26.175 LabApp] Initializing websocket connection /api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877/channels
[D 11:33:26.176 LabApp] Accepting token-authenticated connection from ::1
[D 11:33:26.177 LabApp] 101 GET /api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877/channels?session_id=e39bcb3e-e32d-41f6-88f4-2520a3b2b830&token=8e7519f8c8ac1d5df502b21722fd9282918da9b9b3665471 (::1) 2.32ms
[I 11:33:26.177 LabApp] Connecting to ws://yarn-eg-node-1.fyre.ibm.com:8888/api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877/channels
[D 11:33:26.331 LabApp] Connection is ready: ws: <tornado.websocket.WebSocketClientConnection object at 0x7fd95ec69100>
[D 11:33:31.111 LabApp] Accepting token-authenticated connection from ::1
[D 11:33:31.111 LabApp] Request list kernels: http://yarn-eg-node-1.fyre.ibm.com:8888/api/kernels
[D 11:33:31.263 LabApp] 200 GET /api/kernels?1598294011107 (::1) 152.54ms
[D 11:33:31.956 LabApp] Accepting token-authenticated connection from ::1
[D 11:33:31.956 LabApp] Request kernel at: http://yarn-eg-node-1.fyre.ibm.com:8888/api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877
[D 11:33:32.038 LabApp] Accepting token-authenticated connection from ::1
[D 11:33:32.043 LabApp] 200 GET /api/contents/alice?content=1&1598294012035 (::1) 5.65ms
[D 11:33:32.098 LabApp] Kernel retrieved: {'id': 'c3b43bd3-0311-4927-80a2-ea12a1fc9877', 'name': 'spark_python_yarn_cluster', 'last_activity': '2020-08-24T18:33:31.903982Z', 'execution_state': 'busy', 'connections': 1}
[D 11:33:32.098 LabApp] RemoteKernelManager.kernel_model: c3b43bd3-0311-4927-80a2-ea12a1fc9877
[D 11:33:32.098 LabApp] Request kernel at: http://yarn-eg-node-1.fyre.ibm.com:8888/api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877
[D 11:33:32.246 LabApp] Kernel retrieved: {'id': 'c3b43bd3-0311-4927-80a2-ea12a1fc9877', 'name': 'spark_python_yarn_cluster', 'last_activity': '2020-08-24T18:33:31.903982Z', 'execution_state': 'busy', 'connections': 1}

and the EG side:

[I 2020-08-24 11:33:25.934 EnterpriseGatewayApp] Kernel restarted: c3b43bd3-0311-4927-80a2-ea12a1fc9877
[D 2020-08-24 11:33:25.934 EnterpriseGatewayApp] Connecting to: tcp://172.16.18.84:35077
[I 200824 11:33:25 web:2162] 200 POST /api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877/restart (9.211.42.74) 12404.72ms
[I 200824 11:33:26 web:2162] 200 GET /api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877 (9.211.42.74) 1.22ms
[D 2020-08-24 11:33:26.204 EnterpriseGatewayApp] Websocket closed c3b43bd3-0311-4927-80a2-ea12a1fc9877:8e541af3-d79a4d109ca8a1feb6562e38
[I 2020-08-24 11:33:26.204 EnterpriseGatewayApp] Starting buffering for c3b43bd3-0311-4927-80a2-ea12a1fc9877:8e541af3-d79a4d109ca8a1feb6562e38
[D 2020-08-24 11:33:26.204 EnterpriseGatewayApp] Clearing buffer for c3b43bd3-0311-4927-80a2-ea12a1fc9877
[D 2020-08-24 11:33:26.285 EnterpriseGatewayApp] Initializing websocket connection /api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877/channels
[W 2020-08-24 11:33:26.287 EnterpriseGatewayApp] No session ID specified
[I 2020-08-24 11:33:26.288 EnterpriseGatewayApp] Adapting from protocol version 5.1 (kernel c3b43bd3-0311-4927-80a2-ea12a1fc9877) to 5.3 (client).
[I 200824 11:33:26 web:2162] 101 GET /api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877/channels (9.211.42.74) 4.08ms
[D 2020-08-24 11:33:26.289 EnterpriseGatewayApp] Opening websocket /api/kernels/c3b43bd3-0311-4927-80a2-ea12a1fc9877/channels
[D 2020-08-24 11:33:26.289 EnterpriseGatewayApp] Getting buffer for c3b43bd3-0311-4927-80a2-ea12a1fc9877
[D 2020-08-24 11:33:26.289 EnterpriseGatewayApp] Clearing buffer for c3b43bd3-0311-4927-80a2-ea12a1fc9877
[D 2020-08-24 11:33:26.290 EnterpriseGatewayApp] Connecting to: tcp://172.16.18.84:35077
[D 2020-08-24 11:33:26.290 EnterpriseGatewayApp] Connecting to: tcp://172.16.18.84:55782
[D 2020-08-24 11:33:26.291 EnterpriseGatewayApp] Connecting to: tcp://172.16.18.84:42963
[D 2020-08-24 11:33:26.291 EnterpriseGatewayApp] Connecting to: tcp://172.16.18.84:52514
[D 2020-08-24 11:33:27.911 EnterpriseGatewayApp] activity on c3b43bd3-0311-4927-80a2-ea12a1fc9877: status (starting)
[D 2020-08-24 11:33:27.914 EnterpriseGatewayApp] activity on c3b43bd3-0311-4927-80a2-ea12a1fc9877: status (busy)
[D 2020-08-24 11:33:27.915 EnterpriseGatewayApp] Kernel info reply received: c3b43bd3-0311-4927-80a2-ea12a1fc9877

kevin-bates on 24 Aug 2020

hi kevin,
Thx for the update. Looks like my builds are conflicting with the code changes for this fix and not able to see the logs you are getting as above. Would it be possible for you to share the wheel file for jupyterlab, such that i can use the same and test it?

suryag10 on 25 Aug 2020

I don't think Lab is packaged as a wheel, nor would I know how best to share a build as I'm not a Lab dev. Have you tried following the build steps I provided above? These were taken from the contributing docs and should be reproducible.

@blink1073, @afshin, @jasongrout - could one of you (at your convenience) please review my build steps and/or provide some tips for packaging/sharing builds, if possible? Thanks.

What are the plans to merge this and get it into a release carrier?

kevin-bates on 25 Aug 2020

I don't think Lab is packaged as a wheel, nor would I know how best to share a build as I'm not a Lab dev. Have you tried following the build steps I provided above? These were taken from the contributing docs and should be reproducible.

[suryag10] Thx kevin for the response. Jupyterlab is released as a tar ball or a wheel file as well. See https://pypi.org/project/jupyterlab/#files. Yes i had followed the same steps as mentioned above.

@blink1073, @afshin, @jasongrout - could one of you (at your convenience) please review my build steps and/or provide some tips for packaging/sharing builds, if possible? Thanks.

[suryag10] This would help a lot.

What are the plans to merge this and get it into a release carrier?

suryag10 on 25 Aug 2020

By the way, i am running this setup on K8S cluster. Would this make the difference?

suryag10 on 25 Aug 2020

By the way, i am running this setup on K8S cluster. Would this make the difference?

Other than ensuring the appropriate image is updated and properly deployed, no, it shouldn't.

I'm assuming you're using Hub and EG as well - correct?

Not sure if this would be helpful, but, if possible, you _could_ run Lab outside K8s and hit EG's service (using the service port if you don't have an ingress setup). This would avoid any deployment SNAFUs that might be interfering with things. Or, is there a way to determine you are indeed running the Lab you built?

kevin-bates on 25 Aug 2020

@blink1073, @afshin, @jasongrout - could one of you (at your convenience) please review my build steps and/or provide some tips for packaging/sharing builds, if possible? Thanks.

See https://github.com/jupyterlab/jupyterlab/issues/8900, the issue @suryag10 opened in jlab.

jasongrout on 25 Aug 2020

❤1 👍1

By the way, i am running this setup on K8S cluster. Would this make the difference?

Other than ensuring the appropriate image is updated and properly deployed, no, it shouldn't.

I'm assuming you're using Hub and EG as well - correct?

[Surya] Yes correct.
Not sure if this would be helpful, but, if possible, you _could_ run Lab outside K8s and hit EG's service (using the service port if you don't have an ingress setup). This would avoid any deployment SNAFUs that might be interfering with things. Or, is there a way to determine you are indeed running the Lab you built?
[Surya] i tried both ways running lab outside and inside cluster. Still facing the same issue. Now following the steps specified by jason to build the same. Will update the next steps.

suryag10 on 27 Aug 2020

My understanding is that https://github.com/jupyterlab/jupyterlab/pull/8432 fixed the JupyterLab portion of this issue. This has been merged in master now. @suryag10 opened a PR for a backport, but just to their fork of the jlab repo. @suryag10 - can you open up your backport to the main jupyterlab repo so we can merge it?

Closing as fixed in master. Thanks everyone for diving in and taking apart this tricky combination of things across different projects!

jasongrout on 22 Oct 2020

Follow-up: it looks like https://github.com/jupyterlab/jupyterlab/pull/9388 is also needed for JupyterLab to handle the manual restart case.

jasongrout on 21 Nov 2020

👍1

I have updated jupyterlab from master after https://github.com/jupyterlab/jupyterlab/pull/9388 merge, rebuilt, and did a little test. but I still see the same problem. do I also need to update EG?

Jupyterhub 1.2.2
JupyterLab 3.0.0rc10~ (from master after pr merge)

cceyda on 2 Dec 2020

Hi @cceyda - I apologize for the inconvenience. I've found getting "dev" builds running to be quite tricky and that's w/o any hub configuration to deal with.

Here's what I've had to do...

Checkout jupyterlab master and build lab.
Run lab with --dev-mode. For whatever reason, just building lab and running w/o dev-mode doesn't produce the correct code at runtime. You can confirm you have the correct code in place by inspecting the elements and pulling up the packages/services/src/kernel/default.ts file and searching for the restart() method:

You should see the additional call to this.reconnect().
Lab 3.0 now uses jupyter_server rather than notebook as its server. However, to talk to EG, you also need to build jupyter_server master to pick up the change in https://github.com/jupyter-server/jupyter_server/pull/350. We will need to produce a jupyter_server 1.0.8 release with this commit in it.
(The symptom that you do not have this patch in place is that no cells will run - even prior to the manual restart! If you're not seeing this and haven't rebuilt jupyter_server, then I suspect you may even be using lab 2.x against notebook server.)

Once those 3 items are in place, you should be able to manually restart the kernel. You should also see a 'Reconnect to Kernel' option in lab's kernel menu as well.

Here's a snapshot of an expected cell error after restart since I purposely ran print(i) having not defined i following the restart:
Screen Shot 2020-12-02 at 12 32 54 PM

kevin-bates on 2 Dec 2020

@kevin-bates Thank you for the detailed explanation~
as you suspected in (2) the changes weren't picked up because it wasn't --dev-mode, which is why I could still run cells despite not having jupyter-server latest.

We will need to produce a jupyter_server 1.0.8 release

I installed 'jupyter_server' from master, ~but couldn't test further because a lot of packages have version dependencies that conflict:
pkg_resources.ContextualVersionConflict: (jupyter-server 1.1.0.dev0 (/root/jupyter_server), Requirement.parse('jupyter-server~=1.0.1'), {'jupyterlab-server', 'nbclassic'})~

~To resolve I would need to install all packages from source and 😅...I think I will just wait for the official release.~

🥳 Okay,I just faked the version number and now can confirm the restarts work! 🎉

jupyterlab --dev-mode --config jupyter_notebook_config.py -> config has EG configs

Thanks to everyone who worked on this issue 😄

cceyda on 2 Dec 2020

Thank you for the update and I'm happy we could share this "experience". :smile:

We will be working to get a 1.0.8 jupyter_server release out today or tomorrow. I'll ping back here once that is available.

kevin-bates on 3 Dec 2020

FYI, #9388 is now released in jlab 3.0.0rc11

jasongrout on 3 Dec 2020

FYI, #9388 is now released in jlab 3.0.0rc11

Thank you Jason!

jupyter_server 1.0.8, containing the fix from https://github.com/jupyter-server/jupyter_server/pull/350 (thank you @ricklamers!), is also available.

Thanks to everyone who worked on this issue 😄

Yes, definitely - thank you!

kevin-bates on 3 Dec 2020

👍2

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Real Time Collaboration

blink1073 · 92Comments

Add execution time to notebook/console cells

jasongrout · 45Comments

File Save Error

dclong · 61Comments

Toolbar Buttons are not visible

andyhegedus · 55Comments

Kernel does not reconnect after disconnection

tom-programming · 48Comments