Docker-stacks: Kernel Error with latest images on Win10 Docker for windows

[MikaelUmaN]

What docker image you are using?
latest images as of 2019-11-16:

• scipy
• datascience
• minimal

What complete docker command do you run to launch the container (omitting sensitive values)?

docker run -p 8888:8888 -e JUPYTER_ENABLE_LAB=yes -v ${PWD}:/home/jovyan jupyter/scipy-notebook:latest

What steps do you take once the container is running to reproduce the issue?

1. Go to jupyter lab link
2. Use python launcher to create new notebook.

What do you expect to happen?

New notebook starts OK..

What actually happens?

"Kernel error" with output:

Error Starting Kernel

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/opt/conda/lib/python3.7/site-packages/notebook/services/sessions/handlers.py", line 72, in post
    type=mtype))
  File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/opt/conda/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 88, in create_session
    kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
  File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/opt/conda/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 101, in start_kernel_for_session
    self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
  File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/opt/conda/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/opt/conda/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 168, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
    km.start_kernel(**kwargs)
  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/manager.py", line 240, in start_kernel
    self.write_connection_file()
  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/connect.py", line 547, in write_connection_file
    kernel_name=self.kernel_name
  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/connect.py", line 212, in write_connection_file
    with secure_write(fname) as f:
  File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/connect.py", line 105, in secure_write
    assert '0600' == oct(stat.S_IMODE(os.stat(fname).st_mode)).replace('0o', '0')
AssertionError

Other Info
I have tested the same flow with tag: 1386e2046833 and it works without problems, so something is wrong (at least for me...) with the latest images.

Docker Desktop: 2.1.0.4
Windows 10v: 19023.1

[parente]

There was a recent change in Jupyter to ensure secure files are written with proper permissions. -v ${PWD}:/home/jovyan mounts the current working directory as the home directory within the container. It looks like the new jupyter_client code is unable to set the proper permissions on the volume mounted from Windows 10 by docker.

[Bidek56]

Windows 10 Docker file permission has been a mess for a while. You may need to:

1. Run the older image
2. Wait for WSL2 to come out
3. Sign up for Windows Insider edition.
   Until then just stick with Linux :)

[MikaelUmaN]

I’m using win 10 insider with WSL 2. Though not yet for Docker because WSL 2 seems too unstable right now. Just now I couldn’t open my Ubuntu.

I am running the old image for moment.

I’d much prefer Linux but at work I am kind of tied up unfortunately...

[Bidek56]

If you can run Win 10 insider at work, then you can probably run Virtualbox on Windows, which will allow you to run Linux, from there you can run a working Docker, but if you are going to go thru so much trouble, may as well just do a pip install jupyter in your local env. Good luck.

[MikaelUmaN]

I cant run insider at work. That’s at home.

Regardless, given other images work fine as regards permissions and writing files, the solution is not to install virtual box but to fix this issue in the jupyter client.

I already have a local anaconda and jupyter installed. But I much prefer container based development and think these images are great.

[mattsiler]

You could try adding this to the Docker file:
conda install -c anaconda jupyter_client=5.3.1

I think the issue you're facing might be the same one reported here:
https://stackoverflow.com/questions/58214911/kernel-error-jupyter-client-connect-py-assertionerror

I hit the kernel start error when I re-build my environment last week, and this fix seems to have worked for me.

[gsteele13]

I had the same problem trying to deploy a Jupyterhub where the home directories were on an NFS mount (with squashed permissions):

  File "/opt/conda/lib/python3.7/site-packages/jupyter_client/connect.py", line 105, in secure_write
    assert '0600' == oct(stat.S_IMODE(os.stat(fname).st_mode)).replace('0o', '0')
AssertionError

Rollback to 5.3.1 got rid of the error. It would be good to have a fix for this.

[jaskiratr]

@mattsiler I ran in same issue. Confirming that adding conda install -c anaconda jupyter_client=5.3.1 to Dockerfile fixes it for the moment. Thanks!

[MikaelUmaN]

Any update on this? Tried again with latest everything and still the same error.

[gsteele13]

I'm still suffering from this...

We are running a jupyterhub with the persistent home directories of the docker containers mounted on an NFS mount with root_squash to avoid having to handle username mapping.

Due to this bug, I still have to roll back to 5.3.1 to get a functioning server...

Is there maybe an option to override the secure file check for those that don't need it?

Thanks

[parente]

Is there maybe an option to override the secure file check for those that don't need it?

This repo is essentially packaging for Jupyter projects in Docker. A flag for disabling would have to exist or get added to https://github.com/jupyter/notebook I believe. Have a look at https://jupyter-notebook.readthedocs.io/en/stable/config.html to see if such a flag exists. If it doesn't, you could make a proposal in the notebook issue tracker about adding such a flag. I think it'd be a hard sell: disabling a security check to support the use case of running a Linux container on Windows 10 with a home volume mount.

[gsteele13]

Maybe useful to add:

This problem was solved for me in a recent upgrade to the latest version. (I also could no longer find the offending code in the jupyter-client source.)

To be specific: I no longer have any trouble deploying containers running a jupyter notebook server with home directories for which full POSIX permission changes are not possible.

Here is my version info:

jupyter core     : 4.6.3
jupyter-notebook : 6.0.3
jupyter client   : 6.1.3

It could be that the problem is now also solved for the original windows 10 case.

Cheers,
Gary

[gsteele13]

OK, maybe a minor update:

Curiously, users who had earlier logged in worked fine on my jupyterhub.

However, new users ran into errors that appear to be related to secure write problems. Their server did start up, but they could not start kernels, which we traced back to this related issue:

https://github.com/jupyter/notebook/issues/5058

We are going to try integrating:

ENV JUPYTER_ALLOW_INSECURE_WRITES=true

in the Dockerfile to see if this is a solution.

[gsteele13]

just confirmed that ENV JUPYTER_ALLOW_INSECURE_WRITES=true solves our problems!

[MikaelUmaN]

Yes. Confirmed that this works. Closing. Thank you very much :)!