Hydrogen: Multiple independent kernels

[TKCen]

It's useful to have separate kernels in different files/projects if they are not related. It seems it's only possible to use one shared kernel of the same type currently.

Maybe give each running kernel a unique identifier (hash, PID, ...), this makes them distinguishable and would also give the user the opportunity to switch between them (if necessary).

I'd like to use this feature to run/debug several micro services for example, once #25 is implemented.

[malarinv]

This would be an essential feature, as a long running process shouldn't hinder running other unrelated files. Currently, i have to wait till the kernel to become idle in order to see the output of executing even a completely unrelated file.

It might be useful to add an option to start in a new kernel.

[nikitakit]

It's been a year and a half since this issue was opened, and I want to point out that since then remote kernel support has been added to hydrogen. It's now possible to use kernel gateway to manage multiple running kernels, and switch between them at will. Additionally, it was always the case that each atom window keeps its own set of kernels.

It should now be possible to use these features day-to-day, and to reflect on whether there's a better UI for presenting kernel-switching. (There are limitations to the current approach, such as the fact that manual switching takes time, and clears results boxes.) Both kernel-per-language and kernel-per-file are reasonable usecases, as are many things in between. I personally enjoy how hydrogen works very cleanly with minimal configuration, and it would be ideal if kernel switching didn't sacrifice those qualities. One of the big reasons I haven't proposed code that modifies kernel switching is that I don't have a clear idea of an interaction setup that better balances ease-of-use and kernel-switching than what's currently in Hydrogen.

[lgeiger]

Thanks for bringing up this discussion 🎉

I think this could be a very powerful feature. I haven't thought long about this so take my following suggestions with a grain of salt. It's just a first draft 😄

For this to be transparent to the user I propose the introduction of sessions:

• Each session holds one instance of a running kernel (remote or local)
• Multiple files can be "connected" to the same session
• Hydrogen should be able to keep reference to which file belongs to which session (which may be be tricky if the file is renamed).
• If a user want's to run commands and no kernel is running for the language, a new session and kernel are started
• If there's a running kernel for the desired language (started from a other file) the user get's prompted if he wants to start a new session or wants connect to a running session.
• I would save the case of multiple kernels for the same file for later.

This approach would mix remote kernels and ZMQ kernels but I think that's a good thing.
Basically what I suggest is having a high level way of managing multiple different kernels with different interfaces and clients. Maybe we could build something on top of https://github.com/nteract/enchannel.

[ngr-t]

Emacs's package ESS manages multiple processes in almost the same way as what @lgeiger propose. Adding to that,

• ESS has "switch-process" command so a user can change the process for each buffer anytime (s)he wants to (maybe it corresponds to "the case of multiple kernels for the same file").
• We can rename processes (to be precise, rename the buffers on which the results are shown)

I have used ESS and thought the way as sufficiently simple and powerful, though sometimes I'm confused when multiple processes are running. I wish Hydrogen had a similar feature.

[rgbkrk]

Oops, autoclosed since I merged the PR.

[maximsch2]

So this used to work perfectly well until the last update - different files/views will connect to their separate kernels (through the remote kernel gateway), but now this workflow was broken. Was this an intential update?

[BenRussert]

@maximsch2 remote kernels should still be working, we did deprecate the connection.json method of connecting existing kernels

If you arent talking about the connection.json method this would be a new issue.

[maximsch2]

@BenRussert , sounds good. I've posted my issue as a separate bug: https://github.com/nteract/hydrogen/issues/884

[lgeiger]

This is one of our oldest open issues and I think it's finally time to tackle it.
Here're my ideas on how we could handle this in way that hopefully is very flexible yet doesn't sacrifice ease-of-use. The proposed workflow mixes both local and remote kernels in order to be less confusing and easier to use:

Every time the user switches to a new tab and wants to execute some code, he gets prompted if he likes to connect to a running/existing kernel or a remote kernel or if he likes to start a new local kernel.

Internally the kernels are stored together with the file paths related to them, so they can be reused. Multiple files can be connect to the same kernel but one file is connected to at most one kernel.

I've roughly sketched how the UI (mainly the popup panels) may look like to get a idea on how this would feel:

Kernels are never killed if the user closes a editor tab so they can be reused later. Therefore we either need a command to shutdown specific kernels or a UI that lists all running kernels and the possibility to shut them down individually, similar to how the notebook does it:

Right now remote kernels also implement a concept of session which for my understanding would become redundant and make things overly complicated for users if we handle multiple kernels internally. I think if we want to go this route we should remove the concept of additional remote sessions. @nikitakit I could really need some help and feedback for this since you're the expert for remote kernels 😄

What do you think about this workflow? Would this be a welcome improvement or would it make things just more complicated?

[rgbkrk]

I definitely would have wished for one kernel per file when I was running through tutorials yesterday. It was frustrating to have to restart the kernel for each file I was going to, at least with a JS kernel (mostly because of const x = require(...)).

[BenRussert]

What if we made the default startup to start a new kernel for the current editor? If you wanted to connect to an existing kernel you would do an additional ui step like @lgeiger's mockup above.

[rgbkrk]

I like that @BenRussert