Docker-stacks: [base-notebook] Cannot connect to server after upgrading to Jupyter Lab 3.0.0

[mirekphd]

After upgrading (by unpinning 3 Jupyter packages in my fork of docker-stacks) Jupyter Lab / Server no longer works (only Notebook is still OK).

To reproduce:

1. Clone my fork mirekphd/docker-stacks (or reproduce it by making this change).
2. build mirekphd/base-notebook container (cd base-notebook and docker build --tag mirekphd/base-notebook .)
3. Try to run Jupyter Lab:
   docker run -d --rm --name base-notebook -p 8989:8888 -e JUPYTER_ENABLE_LAB=1 mirekphd/base-notebook:latest && docker logs -f base-notebook
4. Navigate to localhost:8989 and see the "Unable to connect" browser error.
5. Switch to Notebook (e.g. -e JUPYTER_ENABLE_LAB_NOT) to see it works correctly.

Note that earlier today I raised a less reproducible (closed-source) version of this issue in https://github.com/jupyterlab/jupyterlab/issues/9520, so you may find some additional hints / details there.

[romainx]

Hello,

The problem seems to come from the file jupyter_notebook_config.py that needs to be renamed in jupyter_server_config.py in /etc/jupyter/, and change in this file NotebookApp by ServerApp, see this discussion.

c = get_config()  # noqa: F821
-c.NotebookApp.ip = '0.0.0.0'
-c.NotebookApp.port = 8888
-c.NotebookApp.open_browser = False
+c.ServerApp.ip = '0.0.0.0'
+c.ServerApp.port = 8888
+c.ServerApp.open_browser = False

After these changes the problem is fixed and JupyterLab is accessible.
We have not migrated yet to the 3.0 version, so I will make this change along with the bump of version.

Best

[romainx]

Hello,

Here is the documentation Migrating from Notebook Server explaining this change.

[mirekphd]

Thank you so much @romainx !

The docs even warn about the impending doom: _"NBClassic will only work for a limited time. Jupyter Server is likely to evolve beyond a point where Jupyter Notebook frontend will no longer work with the underlying server."_ How unfortunate that the two interfaces cannot continue to co-exist peacefully...

[FuriouslyCurious]

Hello,

The problem seems to come from the file jupyter_notebook_config.py that needs to be renamed in jupyter_server_config.py in /etc/jupyter/, and change in this file NotebookApp by ServerApp, see this discussion.

c = get_config()  # noqa: F821
-c.NotebookApp.ip = '0.0.0.0'
-c.NotebookApp.port = 8888
-c.NotebookApp.open_browser = False
+c.ServerApp.ip = '0.0.0.0'
+c.ServerApp.port = 8888
+c.ServerApp.open_browser = False

After these changes the problem is fixed and JupyterLab is accessible.
We have not migrated yet to the 3.0 version, so I will make this change along with the bump of version.

Best

Thanks for this solution! If someone running as a non-root user, then these config files should be placed in ~/.jupter config directory.

[phish108]

Hi @FuriouslyCurious @romainx

I tried to apply this hotfix ontop of jupyter/base-notebook:latest by adding

RUN  sed -ire "s/^c.NotebookApp/c.ServerApp/g" /etc/jupyter/jupyter_notebook_config.py

before switching to the container user.

The error persists with the present base-notebook container using JL 3.0.0.

I manually added the fix to .jupyter/jupyter_notebook_config.py without effect.

Despite setting c.ServerApp.open_browser = False I continue to get the following warning

[W ... ServerApp] No web browser found: could not locate runnable browser.

My package list is:

$ conda list | grep jup
jupyter-archive           0.7.0                      py_0    conda-forge
jupyter_client            6.1.7                      py_0    conda-forge
jupyter_core              4.7.0            py38h578d9bd_0    conda-forge
jupyter_server            1.1.3            py38h578d9bd_0    conda-forge
jupyter_telemetry         0.1.0              pyhd8ed1ab_1    conda-forge
jupyterhub                1.2.2            py38h578d9bd_0    conda-forge
jupyterhub-base           1.2.2            py38h578d9bd_0    conda-forge
jupyterlab                3.0.0              pyhd8ed1ab_0    conda-forge
jupyterlab-git            0.22.0                     py_0    conda-forge
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.0.0              pyhd8ed1ab_0    conda-forge

[romainx]

@phish108 you need to first copy the file to jupyter_server_config.py and you have to maintain both files if you want to use either classic notebook and JupyterLab.

RUN sed -re "s/c.NotebookApp/c.ServerApp/g" \
    /etc/jupyter/jupyter_notebook_config.py > /etc/jupyter/jupyter_server_config.py

[phish108]

Couldn't this be part of base-notebook? I understand that both configs do not conflict with each other or am I missing something?

[romainx]

@phish108 Maybe but I have not tested the behavior with the current version of JupyterLab 2.2.9.
I've drafted a PR #1206 that bumps the version of JupyterLab to 3.0.0 but it's not ready right now since it conflicts with some extensions installed in scipy-notebook.
Maybe we could release a kind of intermediate version with this code.

[mirekphd]

and you have to maintain both files if you want to use either classic notebook and JupyterLab.

Strictly speaking jupyter_server_config.py is not sufficient to run both interfaces on their own, because Notebook still requires jupyter_notebook_config.py. You can switch from Lab to Notebook (e.g. using Help->Launch classic Notebook) without jupyter_notebook_config.py, but you cannot start Notebook standalone without it.

[phish108]

@romainx Thanks for the infos. Eventually I have my container running and noticed that all of the community extensions I use are currently outdated, so this will take a while.

@mirekphd I got that part. But apart from the server configuration they do not differ too much, so the two files could exist side by side in the base-notebook.

I tried this side-by-side setup for JL 2.2.9 and it causes no new conflicts or warnings for me using a stripped down r-notebook.

[romainx]

@phish108 thanks for the information I'm currently modifying my update PR #1206 to rollback to JupyterLab 2.2.9 waiting for extension compatibility. I've included this change to prepare the upgrade. So there is a chance to have it included soon 😄

[mirekphd]

I think @romainx the initial answer here could be improved by incorporating the two-files setup or at least a warning that renaming (instead of copying) the config will break standalone Notebook.

[mirekphd]

I tried this side-by-side setup for JL 2.2.9 and it causes no new conflicts or warnings for me using a stripped down r-notebook.

I tried the dual config file setup in jupyterlab==3.0.0 and while superficially both interfaces seemed to work OK, after some testing it turned out that new Jupyter Notebook extensions and their configurator refused to work (only previously-loaded extensions work correctly). The warnings (HTTP 404 errors more like) I'm getting in Notebook log are as follows (while the /nbextensions/ page stays blank):

[W 16:08:39.192 NotebookApp] 404 GET /static/notebook/js/mathjaxutils.js?v=20201230160654 (<ip>) 0.900000ms referer=http://<url>:8989/nbextensions/
[W 16:09:05.254 NotebookApp] 404 GET /static/notebook/js/mathjaxutils.js?v=20201230160654 (<ip>) 1.030000ms referer=http://<url>:8989/nbextensions/

So for production containers, I too had to roll back to jupyterlab==2.2.9, notebook=6.1.5, jupyterhub=1.2.2, and nbconvert==5.6.1 (the last one as a workaround reported here for an unrelated API change resulting in multiple _'template_path not recognized'_ errors).

BTW, I do hope it's not the last usable version of Notebook...

[romainx]

@mirekphd In my last build I've used the following versions that are slightly different than your versions. Have you tried this setup ? If so, have you identified issues?

jupyterlab=2.2.9
-notebook=6.1.5
+notebook=6.1.6
-jupyterhub=1.2.2
+jupyterhub=1.3.0
-nbconvert==5.6.1
+nbconvert==6.0.7

Thanks

[mirekphd]

@romainx from the list of your rolled-back Jupyter packages I think that only nbconvert>=6 is a dog, I had to downgrade it to nbconvert==5.6.1 to work around https://github.com/ipython-contrib/jupyter_contrib_nbextensions/issues/1529

To reproduce that issue, switch from Lab to Notebook (Help -> Launch Classic Notebook) and then try to run its extensions configurator (Edit -> nbextensions config) and you should see in server log multiple 'template_path not recognized' warnings like these, while the /nbextensions/ page that normally contains Jupyter Notebook Extensions Configurator would remain unresponsive (blank):

[W 15:32:31.406 LabApp] Config option `template_path` not recognized by `ExporterCollapsibleHeadings`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.414 LabApp] Config option `template_path` not recognized by `ExporterCollapsibleHeadings`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.437 LabApp] Config option `template_path` not recognized by `TocExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.442 LabApp] Config option `template_path` not recognized by `TocExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.452 LabApp] Config option `template_path` not recognized by `LenvsHTMLExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.461 LabApp] Config option `template_path` not recognized by `LenvsHTMLExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.474 LabApp] Config option `template_path` not recognized by `LenvsTocHTMLExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.482 LabApp] Config option `template_path` not recognized by `LenvsTocHTMLExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.523 LabApp] Config option `template_path` not recognized by `LenvsLatexExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.528 LabApp] Config option `template_path` not recognized by `LenvsLatexExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.754 LabApp] Config option `template_path` not recognized by `LenvsSlidesExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.760 LabApp] Config option `template_path` not recognized by `LenvsSlidesExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.877 LabApp] Config option `template_path` not recognized by `ExporterCollapsibleHeadings`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.884 LabApp] Config option `template_path` not recognized by `ExporterCollapsibleHeadings`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.907 LabApp] Config option `template_path` not recognized by `TocExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.912 LabApp] Config option `template_path` not recognized by `TocExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.921 LabApp] Config option `template_path` not recognized by `LenvsHTMLExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.929 LabApp] Config option `template_path` not recognized by `LenvsHTMLExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.942 LabApp] Config option `template_path` not recognized by `LenvsTocHTMLExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.951 LabApp] Config option `template_path` not recognized by `LenvsTocHTMLExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.991 LabApp] Config option `template_path` not recognized by `LenvsLatexExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:31.996 LabApp] Config option `template_path` not recognized by `LenvsLatexExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:32.200 LabApp] Config option `template_path` not recognized by `LenvsSlidesExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?
[W 15:32:32.205 LabApp] Config option `template_path` not recognized by `LenvsSlidesExporter`.  Did you mean one of: `extra_template_paths, template_name, template_paths`?

[R7President]

1. I suggest a document on how to move from 2.2.x to 3.0.0 that can include renaming the config file from jupyter_notebook_config.py to jupyter_server_config.py (but see note below there appears to be a different file)
2. in addition to ip, port and open_browser there are more items that moved: I noticed allow_remote_access
3. There is something else wrong with open_browser, it does not seem to be possible to disable at all, with the possible exception that including a jupyter_notebook_config.py file in with that setting appears to work
4. And the password option appears to have changed, copying over the previous line does not work, I had to re-generate the password.
5. jupyter lab --generate-config generates a different file in .jupyter called
   Writing default config to: /home/jupy/.jupyter/jupyter_lab_config.py
   modifying this file appears to work, EXCEPT for the open_browser flag.

[romainx]

@R7President you're right we will do it carefully. What has been exchanged here is just a workaround since the current stack still uses JupyterLab 2.2.x.

So the plan is to

• perform a stack update with JupyterLab 2.2.x -> PR #1206
• study and perform the upgrade to JupyterLab 2.2.x -> #1207