Notebook: Kernel error: jupyter_client/connect.py AssertionError

[dahjan]

Error message

Traceback (most recent call last):
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
result = await result
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(exc_info) # type: ignore
File "/home/djan/anaconda3/lib/python3.7/site-packages/notebook/services/sessions/handlers.py", line 72, in post
type=mtype))
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(exc_info) # type: ignore
File "/home/djan/anaconda3/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 88, in create_session
kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 742, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File "/home/djan/anaconda3/lib/python3.7/site-packages/notebook/services/sessions/sessionmanager.py", line 101, in start_kernel_for_session
self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
File "/home/djan/anaconda3/lib/python3.7/site-packages/tornado/gen.py", line 209, in wrapper
yielded = next(result)
File "/home/djan/anaconda3/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 168, in start_kernel
super(MappingKernelManager, self).start_kernel(kwargs)
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
km.start_kernel(kwargs)
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/manager.py", line 240, in start_kernel
self.write_connection_file()
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/connect.py", line 547, in write_connection_file
kernel_name=self.kernel_name
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/connect.py", line 212, in write_connection_file
with secure_write(fname) as f:
File "/home/djan/anaconda3/lib/python3.7/contextlib.py", line 112, in __enter__
return next(self.gen)
File "/home/djan/anaconda3/lib/python3.7/site-packages/jupyter_client/connect.py", line 105, in secure_write
assert '0600' == oct(stat.S_IMODE(os.stat(fname).st_mode)).replace('0o', '0')
AssertionError

System information

Running Jupyter (v. 1.0.0) on a Linux server, with anaconda3 (v. 2019.07) 64-bit.

Available Kernels

Available kernels:
python3 /home/djan/.local/share/jupyter/kernels/python3
nn /home/djan/.local/share/jupyter/kernels/nn

Reproducing the issue

1. Install the newest anaconda3 version.
2. Run Jupyter notebook in the base version --> Kernel runs fine
3. Create a new environment with python: conda create -n test python=3.7
4. After activating the environment, Jupyter notebook is not available, thus I must install it via Conda: conda install -c anaconda jupyter
5. Run Jupyter notebook in the test environment --> Kernel breaks down as explained above

What I tried so far

There seems to be an issue with file permissions when running Jupyter on a server inside a Conda environment. I tried to give all files inside my anaconda3 folder the requested permissions (0600), but this did not help.

[dahjan]

Update

It turns out that the newest version of jupyter_client, available under anaconda as version 5.3.3, breaks the Jupyter environment on a server.

Installing the previous version of jupyter_client via following command fixes this issue:

conda install -c anaconda jupyter_client=5.3.1

[mirekphd]

The AssertionError message I get when an assertion fails in line 105 of connect.py of the two latest versions of jupyter_client affected by this bug is the same or rather varies slightly in its string contents:

File "/opt/conda/lib/python3.6/site-packages/jupyter_client/connect.py", line 105, in secure_write
assert '0600' == oct(stat.S_IMODE(os.stat(fname).st_mode)).replace('0o', '0')
AssertionError

The downgrade to 5.3.1 proposed by @dahjan worked for me as well. This means that the bug was introduced in 5.3.3 and remains in 5.3.4.

@dahjan - please reopen the issue - downgrading the pkg is just a temp workaround, and we need a future-proof solution.

Appeal to the developers: please remove BOTH broken versions of jupyter_client (5.3.4 and 5.3.3) from BOTH pypi and Anaconda's conda-forge channel.

More info

I suspect the bug will be a long-lasting one because client-server connections to python 3.6 kernels in Jupyter Notebook and Lab work under some circumstances. For instance, they work correctly under docker run on a rather insecure server (with all ports open) but fail with AssertionError under my corporate (Red Hat) Openshift installation where only ports 8888 and 80 are open between the client and server and there is no routing to the internet (the most relevant differences to this bug I think), so 99% of testers will fail to reproduce it (who in the open source community tests Jupyter kernel connections on on-prem Openshift installations..?:)

My suspicion is confirmed by the fact that it now applies to TWO historical versions of jupyter_client, so this standard downgrade-by-one trick did not work for me when applied to version 5.3.4 (because 5.3.3 is affected too).

[kevin-bates]

Hi @mirekphd - I'm sorry about the inconvenience. The latest release of jupyter_client (5.3.4) did two things.

1. It _moved_ the code for secure_write into the jupyter_core (4.6.0) package
2. It addressed a slippery win32 issue (in the moved code in jupyter_core)

I agree that there is likely a non-win32 issue here relative to the "old" statement you show from jupyter_client. However, that particular statement is no longer an assertion, but a raise with a detailed message containing the current permissions - which would prove extremely helpful in this particular case.

Could you please confirm that you're using the true latest versions of each library and try again so that we can gain access to what the permissions happen to be on this particular platform?

cc: @MSeal

[ketozhang]

Could you please confirm that you're using the true latest versions of each library

Can you clarify again which libraries? I understand jupyter_core is one.

[kevin-bates]

jupyter_client is the other.

[scivm]

I made some comments on the original PR that caused this issue. It is still happening in all the latest docker images in the Jupyter project (build 1 month ago.) I have to add:

RUN conda install --yes jupyter_client==5.3.1

in all my Dockerfiles.

[bollwyvl]

So is an updated jupyter_client bottom pin of jupyter_core >=4.6.1 now appropriate?

[scivm]

Yes, my issue is related to Azurefile cifs filesystem permissions. There has been some work to solve the issue in Jupyter core which resulted in 4.6.1 allowing execute bit on owner. CIF mounts are setting the execute bit on owner which is difficult to prevent. I am now looking at trying to change some mount options related to group and world which came as default with jupyterhub azurefile storage type I am using.

https://github.com/jupyter/jupyter_core/issues/172