Distributed: Launch Jupyter notebook server from Scheduler

[mrocklin]

Should we add convenience functionality to co-deploy a notebook server along with the Dask Scheduler?

From a technical perspective, this is orthogonal, and reasonably out of scope. In practice though, this might be convenient for lots of people. We see workflows like this pretty commonly. We even baked it into the Helm chart.

Technically speaking, we could include the Juptyer server in a variety of manners:

1. In the same event loop
2. In a forked process for nice cleanup
3. In a spawned process for resilience

These have different levels of integration, which have different trade offs

More generally, we might think about how we would want to run related processes next to the scheduler, and a general extension point for that. In the simplest possible form, this might just be a preload script or Scheduler Plugin (see https://docs.dask.org/en/latest/setup/custom-startup.html) . I'm curious what else we might want to add on top of that to make this work well.

cc @mmccarty (who asked about this) and @jcrist @ericdill @jacobtomlinson @quasiben who may have thoughts.

[jacobtomlinson]

My initial reaction is that this is out of scope and sounds like feature creep.

I feel like notebooks are personal enough that people will want to do some amount of configuration. Where is the home directory? What plugins should be installed? What does the conda environment look like?

I would also imagine that notebook servers and configuration should have more longevity than dask clusters, which I tend to consider ephemeral.

We also have the dask labextension which is the inverse of this and allows spawning multiple (potentially remote) dask clusters from within a single notebook server.

The more general point around extensions is interesting. Jupyter has something similar with the proxy server.

[mrocklin]

Mike I wonder if the first step here is to try using preload scripts in
your use case and see what UX issues remain. Maybe that helps us to
identify particular improvements (like web proxies for example) that could
improve the situation, and also be sold as general improvements.

On Tue, Sep 10, 2019 at 1:22 AM Jacob Tomlinson [e-mail]
wrote:

My initial reaction is that this is out of scope and sounds like feature
creep.

I feel like notebooks are personal enough that people will want to do some
amount of configuration. Where is the home directory? What plugins should
be installed? What does the conda environment look like?

I would also imagine that notebook servers and configuration should have
more longevity than dask clusters, which I tend to consider ephemeral.

We also have the dask labextension which is the inverse of this and allows
spawning multiple (potentially remote) dask clusters from within a single
notebook server.

The more general point around extensions is interesting. Jupyter has
something similar
https://github.com/jupyterhub/jupyter-server-proxy/blob/master/docs/server-process.rst
with the proxy server.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/distributed/issues/3043?email_source=notifications&email_token=AACKZTBUIMTFU5XXYBGZGEDQI5KNPA5CNFSM4IVANZGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6KIV3Q#issuecomment-529828590,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTBHQ4PSPLTPYRCBQYLQI5KNPANCNFSM4IVANZGA
.

[mmccarty]

FWIW, I agree with the sentiment on scope. Trying this out with a preload script is certainly worth a try. We will report back here with possible improvements.

[mrocklin]

Tiny blogpost here: https://github.com/dask/dask-blog/pull/54