Jupyterlab: Kernel does not reconnect after disconnection
Describe the bug
Kernel does not reconnect after disconnection
To Reproduce
Open jupyter lab on server, connect to server on client via 'ssh -L 8888:localhost:8888 server' (I am not sure if you need ssh tunnel to see the bug), open the browser at localhost:8888 and open a python notebook, play with it.
Close ssh tunnel for some seconds.
It will show
Server Connection Error
A connection to the Jupyter server could not be established. JupyterLab will continue trying to reconnect. Check your network connection or Jupyter server configuration.
At this point all is lost. When reconnecting the ssh tunnel it won't detect the connection by itself (as told in the dialog) and reactivate, so I have to click on "Dismiss". Now the kernel will be disconnected from the notebook. The command "Reconnect to kernel" will do nothing.
The only way to restore functionality is by clicking on the "No Kernel!" text, and selecting MANUALLY the kernel corresponding to the notebook.
Expected behavior
- Ideally: the client reconnects automatically to the server when the connection is restored
- At minimum: the "Reconnect To Kernel" command will work to reconnect the client to its kernel. It would be nice to be able to run this command without having to click on "Dismiss" on the dialog
Desktop (please complete the following information):
- OS: Ubuntu 18.06 (server) macOS 10.13.6 (client)
- Browser: Firefox 68.0
- JupyterLab: 1.0.2
Additional context
All the connections between client and server go through an ssh tunnel (see above)
Troubleshoot Output
$PATH:
/home/user/.npm-global/bin
/usr/local/cuda-9.1/bin
/home/user/bin
/home/user/unix/bin
/home/user/anaconda3/bin
/home/user/anaconda3/condabin
/opt/lumerical/fdtd/bin
/opt/lumerical/mode/bin
/home/user/local_storage/bin
/home/user/local_storage/unix/bin
/usr/local/texlive/2017/bin/x86_64-linux
/opt/anaconda/anaconda3/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/games
/usr/local/games
/snap/bin
sys.path:
/home/user/anaconda3/bin
/home/user/anaconda3/lib/python37.zip
/home/user/anaconda3/lib/python3.7
/home/user/anaconda3/lib/python3.7/lib-dynload
/home/user/anaconda3/lib/python3.7/site-packages
sys.executable:
/home/user/anaconda3/bin/python3.7
sys.version:
3.7.3 (default, Mar 27 2019, 22:11:17)
[GCC 7.3.0]
platform.platform():
Linux-4.4.0-150-generic-x86_64-with-debian-stretch-sid
which -a jupyter:
/home/user/anaconda3/bin/jupyter
pip list:
Package Version
---------------------------------- ---------
alabaster 0.7.12
anaconda-client 1.7.2
anaconda-navigator 1.9.7
anaconda-project 0.8.3
asn1crypto 0.24.0
astroid 2.2.5
astropy 3.2.1
atomicwrites 1.3.0
attrs 19.1.0
Babel 2.7.0
backcall 0.1.0
backports.functools-lru-cache 1.5
backports.os 0.1.1
backports.shutil-get-terminal-size 1.0.0
backports.tempfile 1.0
backports.weakref 1.0.post1
beautifulsoup4 4.7.1
bitarray 0.9.3
bkcharts 0.2
bleach 3.1.0
blessings 1.7
bokeh 1.3.0
boto 2.49.0
Bottleneck 1.2.1
certifi 2019.6.16
cffi 1.12.3
chardet 3.0.4
Click 7.0
cloudpickle 1.2.1
clyent 1.2.2
colorama 0.4.1
conda 4.7.10
conda-build 3.18.9
conda-package-handling 1.3.11
conda-verify 3.4.2
contextlib2 0.5.5
cryptography 2.7
cycler 0.10.0
Cython 0.29.12
cytoolz 0.10.0
dask 2.1.0
decorator 4.4.0
defusedxml 0.6.0
distributed 2.1.0
docutils 0.14
entrypoints 0.3
et-xmlfile 1.0.1
fastcache 1.1.0
filelock 3.0.12
Flask 1.1.1
future 0.17.1
gevent 1.4.0
glob2 0.7
gmpy2 2.0.8
gpustat 0.6.0
greenlet 0.4.15
h5py 2.9.0
hdf5storage 0.1.15
heapdict 1.0.0
html5lib 1.0.1
hyperopt 0.1.2
idna 2.8
imageio 2.5.0
imagesize 1.1.0
importlib-metadata 0.17
ipykernel 5.1.1
ipython 7.6.1
ipython-genutils 0.2.0
ipywidgets 7.5.0
isort 4.3.21
itsdangerous 1.1.0
jdcal 1.4.1
jedi 0.13.3
jeepney 0.4
Jinja2 2.10.1
joblib 0.13.2
json5 0.8.4
jsonschema 3.0.1
jupyter 1.0.0
jupyter-client 5.3.1
jupyter-console 6.0.0
jupyter-core 4.5.0
jupyterlab 1.0.2
jupyterlab-server 1.0.0
keyring 18.0.0
kiwisolver 1.1.0
lazy-object-proxy 1.4.1
libarchive-c 2.8
lief 0.9.0
llvmlite 0.29.0
locket 0.2.0
lxml 4.3.4
MarkupSafe 1.1.1
matplotlib 3.1.0
mccabe 0.6.1
mistune 0.8.4
mkl-fft 1.0.12
mkl-random 1.0.2
mkl-service 2.0.2
mock 3.0.5
more-itertools 7.0.0
mpmath 1.1.0
msgpack 0.6.1
multipledispatch 0.6.0
navigator-updater 0.2.1
nbconvert 5.5.0
nbformat 4.4.0
networkx 2.3
nltk 3.4.4
nose 1.3.7
notebook 6.0.0
numba 0.45.0
numexpr 2.6.9
numpy 1.16.4
numpydoc 0.9.1
nvidia-ml-py3 7.352.0
olefile 0.46
openpyxl 2.6.2
packaging 19.0
pandas 0.24.2
pandocfilters 1.4.2
parso 0.5.0
partd 1.0.0
path.py 12.0.1
pathlib2 2.3.4
patsy 0.5.1
pep8 1.7.1
pexpect 4.7.0
pickleshare 0.7.5
Pillow 6.1.0
pip 19.1.1
pkginfo 1.5.0.1
pluggy 0.12.0
ply 3.11
prometheus-client 0.7.1
prompt-toolkit 2.0.9
psutil 5.6.3
ptyprocess 0.6.0
py 1.8.0
pycodestyle 2.5.0
pycosat 0.6.3
pycparser 2.19
pycrypto 2.6.1
pycurl 7.43.0.3
pyflakes 2.1.1
Pygments 2.4.2
pylint 2.3.1
pymongo 3.8.0
pyodbc 4.0.26
pyOpenSSL 19.0.0
pyparsing 2.4.0
pyrsistent 0.14.11
PySocks 1.7.0
pytest 5.0.1
pytest-arraydiff 0.3
pytest-astropy 0.5.0
pytest-doctestplus 0.3.0
pytest-openfiles 0.3.2
pytest-remotedata 0.3.1
python-dateutil 2.8.0
pytz 2019.1
PyWavelets 1.0.3
PyYAML 5.1.1
pyzmq 18.0.0
QtAwesome 0.5.7
qtconsole 4.5.2
QtPy 1.8.0
qutip 4.4.0
requests 2.22.0
rope 0.14.0
ruamel-yaml 0.15.46
scikit-image 0.15.0
scikit-learn 0.21.2
scipy 1.3.0
seaborn 0.9.0
SecretStorage 3.1.1
Send2Trash 1.5.0
setuptools 41.0.1
simplegeneric 0.8.1
singledispatch 3.4.0.3
six 1.12.0
snowballstemmer 1.9.0
sortedcollections 1.1.2
sortedcontainers 2.1.0
soupsieve 1.8
Sphinx 2.1.2
sphinxcontrib-applehelp 1.0.1
sphinxcontrib-devhelp 1.0.1
sphinxcontrib-htmlhelp 1.0.2
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.2
sphinxcontrib-serializinghtml 1.1.3
sphinxcontrib-websupport 1.1.2
spyder 3.3.6
spyder-kernels 0.5.1
SQLAlchemy 1.3.5
statsmodels 0.10.1
sympy 1.4
tables 3.5.2
tblib 1.4.0
terminado 0.8.2
testpath 0.4.2
toolz 0.10.0
torch 1.1.0
torchfile 0.1.0
torchvision 0.3.0
tornado 6.0.3
tqdm 4.32.1
traitlets 4.3.2
umap-learn 0.3.9
unicodecsv 0.14.1
urllib3 1.24.2
visdom 0.1.8.8
wcwidth 0.1.7
webencodings 0.5.1
websocket-client 0.56.0
Werkzeug 0.15.4
wheel 0.33.4
widgetsnbextension 3.5.0
wrapt 1.11.2
wurlitzer 1.0.2
xlrd 1.2.0
XlsxWriter 1.1.8
xlwt 1.3.0
zict 1.0.0
zipp 0.5.1
conda list:
# packages in environment at /home/user/anaconda3:
#
# Name Version Build Channel
_anaconda_depends 2019.03 py37_0
_ipyw_jlab_nb_ext_conf 0.1.0 py37_0
_libgcc_mutex 0.1 main
alabaster 0.7.12 py37_0
anaconda custom py37_1
anaconda-client 1.7.2 py37_0
anaconda-navigator 1.9.7 py37_0
anaconda-project 0.8.3 py_0
asn1crypto 0.24.0 py37_0
astroid 2.2.5 py37_0
astropy 3.2.1 py37h7b6447c_0
atomicwrites 1.3.0 py37_1
attrs 19.1.0 py37_1
babel 2.7.0 py_0
backcall 0.1.0 py37_0
backports 1.0 py_2
backports.functools_lru_cache 1.5 py_2
backports.os 0.1.1 py37_0
backports.shutil_get_terminal_size 1.0.0 py37_2
backports.tempfile 1.0 py_1
backports.weakref 1.0.post1 py_1
beautifulsoup4 4.7.1 py37_1
bitarray 0.9.3 py37h7b6447c_0
bkcharts 0.2 py37_0
blas 1.0 mkl
bleach 3.1.0 py37_0
blessings 1.7 pypi_0 pypi
blosc 1.16.3 hd408876_0
bokeh 1.3.0 py37_0
boto 2.49.0 py37_0
bottleneck 1.2.1 py37h035aef0_1
bzip2 1.0.8 h7b6447c_0
ca-certificates 2019.5.15 0
cairo 1.14.12 h8948797_3
certifi 2019.6.16 py37_0
cffi 1.12.3 py37h2e261b9_0
chardet 3.0.4 py37_1
click 7.0 py37_0
cloudpickle 1.2.1 py_0
clyent 1.2.2 py37_1
colorama 0.4.1 py37_0
conda 4.7.10 py37_0
conda-build 3.18.9 py37_0
conda-env 2.6.0 1
conda-package-handling 1.3.11 py37_0
conda-verify 3.4.2 py_1
contextlib2 0.5.5 py37_0
cryptography 2.7 py37h1ba5d50_0
cudatoolkit 9.0 h13b8566_0
curl 7.65.2 hbc83047_0
cycler 0.10.0 py37_0
cython 0.29.12 py37he6710b0_0
cytoolz 0.10.0 py37h7b6447c_0
dask 2.1.0 py_0
dask-core 2.1.0 py_0
dbus 1.13.6 h746ee38_0
decorator 4.4.0 py37_1
defusedxml 0.6.0 py_0
distributed 2.1.0 py_0
docutils 0.14 py37_0
entrypoints 0.3 py37_0
et_xmlfile 1.0.1 py37_0
expat 2.2.6 he6710b0_0
fastcache 1.1.0 py37h7b6447c_0
filelock 3.0.12 py_0
flask 1.1.1 py_0
fontconfig 2.13.0 h9420a91_0
freetype 2.9.1 h8a8886c_1
fribidi 1.0.5 h7b6447c_0
future 0.17.1 py37_0
get_terminal_size 1.0.0 haa9412d_0
gevent 1.4.0 py37h7b6447c_0
glib 2.56.2 hd408876_0
glob2 0.7 py_0
gmp 6.1.2 h6c8ec71_1
gmpy2 2.0.8 py37h10f8cd9_2
gpustat 0.6.0 pypi_0 pypi
graphite2 1.3.13 h23475e2_0
greenlet 0.4.15 py37h7b6447c_0
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb453b48_1
h5py 2.9.0 py37h7918eee_0
harfbuzz 1.8.8 hffaf4a1_0
hdf5 1.10.4 hb1b8bf9_0
hdf5storage 0.1.15 pypi_0 pypi
heapdict 1.0.0 py37_2
html5lib 1.0.1 py37_0
hyperopt 0.1.2 pypi_0 pypi
icu 58.2 h9c2bf20_1
idna 2.8 py37_0
imageio 2.5.0 py37_0
imagesize 1.1.0 py37_0
importlib_metadata 0.17 py37_1
intel-openmp 2019.4 243
ipykernel 5.1.1 py37h39e3cac_0
ipython 7.6.1 py37h39e3cac_0
ipython_genutils 0.2.0 py37_0
ipywidgets 7.5.0 py_0
isort 4.3.21 py37_0
itsdangerous 1.1.0 py37_0
jbig 2.1 hdba287a_0
jdcal 1.4.1 py_0
jedi 0.13.3 py37_0
jeepney 0.4 py37_0
jinja2 2.10.1 py37_0
joblib 0.13.2 py37_0
jpeg 9b h024ee3a_2
json5 0.8.4 py_0
jsonschema 3.0.1 py37_0
jupyter 1.0.0 py37_7
jupyter_client 5.3.1 py_0
jupyter_console 6.0.0 py37_0
jupyter_core 4.5.0 py_0
jupyterlab 1.0.2 py37hf63ae98_0
jupyterlab_server 1.0.0 py_0
keyring 18.0.0 py37_0
kiwisolver 1.1.0 py37he6710b0_0
krb5 1.16.1 h173b8e3_7
lazy-object-proxy 1.4.1 py37h7b6447c_0
libarchive 3.3.3 h5d8350f_5
libcurl 7.65.2 h20c2e04_0
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
liblief 0.9.0 h7725739_2
libpng 1.6.37 hbc83047_0
libsodium 1.0.16 h1bed415_0
libssh2 1.8.2 h1ba5d50_0
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.0.10 h2733197_2
libtool 2.4.6 h7b6447c_5
libuuid 1.0.3 h1bed415_2
libxcb 1.13 h1bed415_1
libxml2 2.9.9 hea5a465_1
libxslt 1.1.33 h7d1a2b0_0
llvmlite 0.29.0 py37hd408876_0
locket 0.2.0 py37_1
lxml 4.3.4 py37hefd8a0e_0
lz4-c 1.8.1.2 h14c3975_0
lzo 2.10 h49e0be7_2
markupsafe 1.1.1 py37h7b6447c_0
matplotlib 3.1.0 py37h5429711_0
mccabe 0.6.1 py37_1
mistune 0.8.4 py37h7b6447c_0
mkl 2019.4 243
mkl-service 2.0.2 py37h7b6447c_0
mkl_fft 1.0.12 py37ha843d7b_0
mkl_random 1.0.2 py37hd81dba3_0
mock 3.0.5 py37_0
more-itertools 7.0.0 py37_0
mpc 1.1.0 h10f8cd9_1
mpfr 4.0.1 hdf1c602_3
mpmath 1.1.0 py37_0
msgpack-python 0.6.1 py37hfd86e86_1
multipledispatch 0.6.0 py37_0
navigator-updater 0.2.1 py37_0
nbconvert 5.5.0 py_0
nbformat 4.4.0 py37_0
ncurses 6.1 he6710b0_1
networkx 2.3 py_0
ninja 1.9.0 py37hfd86e86_0
nltk 3.4.4 py37_0
nose 1.3.7 py37_2
notebook 6.0.0 py37_0
numba 0.45.0 py37h962f231_0
numexpr 2.6.9 py37h9e4a6bb_0
numpy 1.16.4 py37h7e9f1db_0
numpy-base 1.16.4 py37hde5b4d6_0
numpydoc 0.9.1 py_0
nvidia-ml-py3 7.352.0 pypi_0 pypi
olefile 0.46 py37_0
openpyxl 2.6.2 py_0
openssl 1.1.1c h7b6447c_1
packaging 19.0 py37_0
pandas 0.24.2 py37he6710b0_0
pandoc 2.2.3.2 0
pandocfilters 1.4.2 py37_1
pango 1.42.4 h049681c_0
parso 0.5.0 py_0
partd 1.0.0 py_0
patchelf 0.9 he6710b0_3
path.py 12.0.1 py_0
pathlib2 2.3.4 py37_0
patsy 0.5.1 py37_0
pcre 8.43 he6710b0_0
pep8 1.7.1 py37_0
pexpect 4.7.0 py37_0
pickleshare 0.7.5 py37_0
pillow 6.1.0 py37h34e0f95_0
pip 19.1.1 py37_0
pixman 0.38.0 h7b6447c_0
pkginfo 1.5.0.1 py37_0
pluggy 0.12.0 py_0
ply 3.11 py37_0
prometheus_client 0.7.1 py_0
prompt_toolkit 2.0.9 py37_0
psutil 5.6.3 py37h7b6447c_0
ptyprocess 0.6.0 py37_0
py 1.8.0 py37_0
py-lief 0.9.0 py37h7725739_2
pycodestyle 2.5.0 py37_0
pycosat 0.6.3 py37h14c3975_0
pycparser 2.19 py37_0
pycrypto 2.6.1 py37h14c3975_9
pycurl 7.43.0.3 py37h1ba5d50_0
pyflakes 2.1.1 py37_0
pygments 2.4.2 py_0
pylint 2.3.1 py37_0
pymongo 3.8.0 pypi_0 pypi
pyodbc 4.0.26 py37he6710b0_0
pyopenssl 19.0.0 py37_0
pyparsing 2.4.0 py_0
pyqt 5.9.2 py37h05f1152_2
pyrsistent 0.14.11 py37h7b6447c_0
pysocks 1.7.0 py37_0
pytables 3.5.2 py37h71ec239_1
pytest 5.0.1 py37_0
pytest-arraydiff 0.3 py37h39e3cac_0
pytest-astropy 0.5.0 py37_0
pytest-doctestplus 0.3.0 py37_0
pytest-openfiles 0.3.2 py37_0
pytest-remotedata 0.3.1 py37_0
python 3.7.3 h0371630_0
python-dateutil 2.8.0 py37_0
python-libarchive-c 2.8 py37_11
pytorch 1.1.0 py3.7_cuda9.0.176_cudnn7.5.1_0 pytorch
pytz 2019.1 py_0
pywavelets 1.0.3 py37hdd07704_1
pyyaml 5.1.1 py37h7b6447c_0
pyzmq 18.0.0 py37he6710b0_0
qt 5.9.7 h5867ecd_1
qtawesome 0.5.7 py37_1
qtconsole 4.5.2 py_0
qtpy 1.8.0 py_0
qutip 4.4.0 py37h9de70de_1 conda-forge
readline 7.0 h7b6447c_5
requests 2.22.0 py37_0
rope 0.14.0 py_0
ruamel_yaml 0.15.46 py37h14c3975_0
scikit-image 0.15.0 py37he6710b0_0
scikit-learn 0.21.2 py37hd81dba3_0
scipy 1.3.0 py37h7c811a0_0
seaborn 0.9.0 py37_0
secretstorage 3.1.1 py37_0
send2trash 1.5.0 py37_0
setuptools 41.0.1 py37_0
simplegeneric 0.8.1 py37_2
singledispatch 3.4.0.3 py37_0
sip 4.19.8 py37hf484d3e_0
six 1.12.0 py37_0
snappy 1.1.7 hbae5bb6_3
snowballstemmer 1.9.0 py_0
sortedcollections 1.1.2 py37_0
sortedcontainers 2.1.0 py37_0
soupsieve 1.8 py37_0
sphinx 2.1.2 py_0
sphinxcontrib 1.0 py37_1
sphinxcontrib-applehelp 1.0.1 py_0
sphinxcontrib-devhelp 1.0.1 py_0
sphinxcontrib-htmlhelp 1.0.2 py_0
sphinxcontrib-jsmath 1.0.1 py_0
sphinxcontrib-qthelp 1.0.2 py_0
sphinxcontrib-serializinghtml 1.1.3 py_0
sphinxcontrib-websupport 1.1.2 py_0
spyder 3.3.6 py37_0
spyder-kernels 0.5.1 py37_0
sqlalchemy 1.3.5 py37h7b6447c_0
sqlite 3.29.0 h7b6447c_0
statsmodels 0.10.1 py37hdd07704_0
sympy 1.4 py37_0
tblib 1.4.0 py_0
terminado 0.8.2 py37_0
testpath 0.4.2 py37_0
tk 8.6.8 hbc83047_0
toolz 0.10.0 py_0
torchfile 0.1.0 pypi_0 pypi
torchvision 0.3.0 py37_cu9.0.176_1 pytorch
tornado 6.0.3 py37h7b6447c_0
tqdm 4.32.1 py_0
traitlets 4.3.2 py37_0
umap-learn 0.3.9 py37_0 conda-forge
unicodecsv 0.14.1 py37_0
unixodbc 2.3.7 h14c3975_0
urllib3 1.24.2 py37_0
visdom 0.1.8.8 pypi_0 pypi
wcwidth 0.1.7 py37_0
webencodings 0.5.1 py37_1
websocket-client 0.56.0 pypi_0 pypi
werkzeug 0.15.4 py_0
wheel 0.33.4 py37_0
widgetsnbextension 3.5.0 py37_0
wrapt 1.11.2 py37h7b6447c_0
wurlitzer 1.0.2 py37_0
xlrd 1.2.0 py37_0
xlsxwriter 1.1.8 py_0
xlwt 1.3.0 py37_0
xz 5.2.4 h14c3975_4
yaml 0.1.7 had09818_2
zeromq 4.3.1 he6710b0_3
zict 1.0.0 py_0
zipp 0.5.1 py_0
zlib 1.2.11 h7b6447c_3
zstd 1.3.7 h0b5b093_0
Command Line Output
[D 14:18:39.636 LabApp] Searching ['/home/user', '/home/user/.jupyter', '/home/user/anaconda3/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 14:18:39.637 LabApp] Looking for jupyter_config in /etc/jupyter
[D 14:18:39.637 LabApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 14:18:39.637 LabApp] Looking for jupyter_config in /home/tom/anaconda3/etc/jupyter
[D 14:18:39.637 LabApp] Looking for jupyter_config in /home/tom/.jupyter
[D 14:18:39.637 LabApp] Looking for jupyter_config in /home/tom
[D 14:18:39.638 LabApp] Looking for jupyter_notebook_config in /etc/jupyter
[D 14:18:39.638 LabApp] Looking for jupyter_notebook_config in /usr/local/etc/jupyter
[D 14:18:39.638 LabApp] Looking for jupyter_notebook_config in /home/tom/anaconda3/etc/jupyter
[D 14:18:39.638 LabApp] Loaded config file: /home/tom/anaconda3/etc/jupyter/jupyter_notebook_config.json
[D 14:18:39.638 LabApp] Looking for jupyter_notebook_config in /home/user/.jupyter
[D 14:18:39.638 LabApp] Loaded config file: /home/tom/.jupyter/jupyter_notebook_config.py
[D 14:18:39.638 LabApp] Loaded config file: /home/tom/.jupyter/jupyter_notebook_config.json
[D 14:18:39.638 LabApp] Looking for jupyter_notebook_config in /home/tom
[D 14:18:39.643 LabApp] Paths used for configuration of jupyter_notebook_config:
/etc/jupyter/jupyter_notebook_config.json
[D 14:18:39.643 LabApp] Paths used for configuration of jupyter_notebook_config:
/usr/local/etc/jupyter/jupyter_notebook_config.json
[D 14:18:39.643 LabApp] Paths used for configuration of jupyter_notebook_config:
/home/user/anaconda3/etc/jupyter/jupyter_notebook_config.d/jupyterlab.json
/home/user/anaconda3/etc/jupyter/jupyter_notebook_config.json
[D 14:18:39.644 LabApp] Paths used for configuration of jupyter_notebook_config:
/home/user/.jupyter/jupyter_notebook_config.json
[I 14:18:39.802 LabApp] The port 8888 is already in use, trying another port.
[I 14:18:39.810 LabApp] JupyterLab extension loaded from /home/user/anaconda3/lib/python3.7/site-packages/jupyterlab
[I 14:18:39.810 LabApp] JupyterLab application directory is /home/user/anaconda3/share/jupyter/lab
[I 14:18:39.812 LabApp] Serving notebooks from local directory: /home/tom
[I 14:18:39.812 LabApp] The Jupyter Notebook is running at:
[I 14:18:39.812 LabApp] http://localhost:8889/
[I 14:18:39.812 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Browser Output
vendors~main.701a79b7e5011430f0dc.js:216682 Download the React DevTools for a better development experience: https://fb.me/react-devtools
vendors~main.701a79b7e5011430f0dc.js:243884 Starting application in workspace: "/lab"
vendors~main.701a79b7e5011430f0dc.js:195930 Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in the componentWillUnmount method.
in Component (created by TOCTree)
warningWithoutStack @ vendors~main.701a79b7e5011430f0dc.js:195930
vendors~main.701a79b7e5011430f0dc.js:192867 Starting WebSocket: ws://localhost:8888/api/kernels/ce0ab287-1bf5-4c68-ba51-8b28a5a9eb95
vendors~main.701a79b7e5011430f0dc.js:192867 Starting WebSocket: ws://localhost:8888/api/kernels/fe2a751c-d415-4b41-a63b-50b5aad0b50c
vendors~main.701a79b7e5011430f0dc.js:192867 Starting WebSocket: ws://localhost:8888/api/kernels/899b826e-ce34-4b08-b23d-2e4dbfaba726
vendors~main.701a79b7e5011430f0dc.js:194264 Kernel: connected (ce0ab287-1bf5-4c68-ba51-8b28a5a9eb95)
vendors~main.701a79b7e5011430f0dc.js:194264 Kernel: connected (fe2a751c-d415-4b41-a63b-50b5aad0b50c)
vendors~main.701a79b7e5011430f0dc.js:194264 Kernel: connected (899b826e-ce34-4b08-b23d-2e4dbfaba726)
about:blank:1 Failed to load resource: net::ERR_UNKNOWN_URL_SCHEME
vendors~main.701a79b7e5011430f0dc.js:192867 Starting WebSocket: ws://localhost:8888/api/kernels/ce0ab287-1bf5-4c68-ba51-8b28a5a9eb95
vendors~main.701a79b7e5011430f0dc.js:194264 Kernel: connected (ce0ab287-1bf5-4c68-ba51-8b28a5a9eb95)
vendors~main.701a79b7e5011430f0dc.js:192867 Starting WebSocket: ws://localhost:8888/api/kernels/35e2a5d2-f54c-4247-b001-b4bc0a0d73e8
vendors~main.701a79b7e5011430f0dc.js:194264 Kernel: connected (35e2a5d2-f54c-4247-b001-b4bc0a0d73e8)
default.js:1453 Kernel: reconnecting (ce0ab287-1bf5-4c68-ba51-8b28a5a9eb95)
default.js:144 Connection lost, reconnecting in 1 seconds.
DefaultKernel._onWSClose @ default.js:144
default.js:1453 Kernel: reconnecting (fe2a751c-d415-4b41-a63b-50b5aad0b50c)
default.js:144 Connection lost, reconnecting in 1 seconds.
DefaultKernel._onWSClose @ default.js:144
default.js:1453 Kernel: reconnecting (899b826e-ce34-4b08-b23d-2e4dbfaba726)
default.js:144 Connection lost, reconnecting in 1 seconds.
DefaultKernel._onWSClose @ default.js:144
default.js:1453 Kernel: reconnecting (35e2a5d2-f54c-4247-b001-b4bc0a0d73e8)
default.js:144 Connection lost, reconnecting in 1 seconds.
DefaultKernel._onWSClose @ default.js:144
default.js:56 Starting WebSocket: ws://localhost:8888/api/kernels/ce0ab287-1bf5-4c68-ba51-8b28a5a9eb95
default.js:56 Starting WebSocket: ws://localhost:8888/api/kernels/fe2a751c-d415-4b41-a63b-50b5aad0b50c
default.js:56 Starting WebSocket: ws://localhost:8888/api/kernels/899b826e-ce34-4b08-b23d-2e4dbfaba726
default.js:56 Starting WebSocket: ws://localhost:8888/api/kernels/35e2a5d2-f54c-4247-b001-b4bc0a0d73e8
default.js:64 WebSocket connection to 'ws://localhost:8888/api/kernels/ce0ab287-1bf5-4c68-ba51-8b28a5a9eb95/channels?session_id=5a79cdec-50ba-442f-a411-c08131525ec9' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
DefaultKernel._createSocket @ default.js:64
default.js:144 Connection lost, reconnecting in 2 seconds.
DefaultKernel._onWSClose @ default.js:144
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
default.js:64 WebSocket connection to 'ws://localhost:8888/api/kernels/fe2a751c-d415-4b41-a63b-50b5aad0b50c/channels?session_id=86d7dd30-7f84-414c-96f0-7a424b9af77c' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
DefaultKernel._createSocket @ default.js:64
default.js:144 Connection lost, reconnecting in 2 seconds.
DefaultKernel._onWSClose @ default.js:144
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
default.js:64 WebSocket connection to 'ws://localhost:8888/api/kernels/899b826e-ce34-4b08-b23d-2e4dbfaba726/channels?session_id=a6a35c6a-3abc-4f25-95b3-4aedda24f71f' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
DefaultKernel._createSocket @ default.js:64
default.js:144 Connection lost, reconnecting in 2 seconds.
DefaultKernel._onWSClose @ default.js:144
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
default.js:64 WebSocket connection to 'ws://localhost:8888/api/kernels/35e2a5d2-f54c-4247-b001-b4bc0a0d73e8/channels?session_id=a1c21955-ad48-47ec-8137-f750cb4adadc' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
DefaultKernel._createSocket @ default.js:64
default.js:144 Connection lost, reconnecting in 2 seconds.
DefaultKernel._onWSClose @ default.js:144
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
default.js:56 Starting WebSocket: ws://localhost:8888/api/kernels/ce0ab287-1bf5-4c68-ba51-8b28a5a9eb95
default.js:64 WebSocket connection to 'ws://localhost:8888/api/kernels/ce0ab287-1bf5-4c68-ba51-8b28a5a9eb95/channels?session_id=5a79cdec-50ba-442f-a411-c08131525ec9' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
DefaultKernel._createSocket @ default.js:64
default.js:144 Connection lost, reconnecting in 4 seconds.
DefaultKernel._onWSClose @ default.js:144
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
default.js:56 Starting WebSocket: ws://localhost:8888/api/kernels/fe2a751c-d415-4b41-a63b-50b5aad0b50c
default.js:64 WebSocket connection to 'ws://localhost:8888/api/kernels/fe2a751c-d415-4b41-a63b-50b5aad0b50c/channels?session_id=86d7dd30-7f84-414c-96f0-7a424b9af77c' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
DefaultKernel._createSocket @ default.js:64
default.js:144 Connection lost, reconnecting in 4 seconds.
DefaultKernel._onWSClose @ default.js:144
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
default.js:56 Starting WebSocket: ws://localhost:8888/api/kernels/899b826e-ce34-4b08-b23d-2e4dbfaba726
default.js:64 WebSocket connection to 'ws://localhost:8888/api/kernels/899b826e-ce34-4b08-b23d-2e4dbfaba726/channels?session_id=a6a35c6a-3abc-4f25-95b3-4aedda24f71f' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
DefaultKernel._createSocket @ default.js:64
default.js:144 Connection lost, reconnecting in 4 seconds.
DefaultKernel._onWSClose @ default.js:144
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
default.js:56 Starting WebSocket: ws://localhost:8888/api/kernels/35e2a5d2-f54c-4247-b001-b4bc0a0d73e8
default.js:64 WebSocket connection to 'ws://localhost:8888/api/kernels/35e2a5d2-f54c-4247-b001-b4bc0a0d73e8/channels?session_id=a1c21955-ad48-47ec-8137-f750cb4adadc' failed: Error in connection establishment: net::ERR_CONNECTION_REFUSED
DefaultKernel._createSocket @ default.js:64
default.js:144 Connection lost, reconnecting in 4 seconds.
DefaultKernel._onWSClose @ default.js:144
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
error (async)
DefaultKernel._createSocket @ default.js:70
setTimeout (async)
DefaultKernel._onWSClose @ default.js:145
serverconnection.js:193 GET http://localhost:8888/api/sessions?1564140317085 net::ERR_CONNECTION_REFUSED
handleRequest @ serverconnection.js:193
makeRequest @ serverconnection.js:75
listRunning @ default.js:556
listRunning @ default.js:371
listRunning @ session.js:26
requestRunning @ manager.js:262
factory @ manager.js:45
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
dialog.js:32 Showing error: {message: "A connection to the Jupyter server could not be es…work connection or Jupyter server configuration.↵"}
showErrorMessage @ dialog.js:32
ConnectionLost @ connectionlost.js:12
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
_onConnectionFailure @ manager.js:105
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
(anonymous) @ manager.js:267
Promise.catch (async)
requestRunning @ manager.js:262
factory @ manager.js:45
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
serverconnection.js:193 GET http://localhost:8888/api/terminals?1564140317089 net::ERR_CONNECTION_REFUSED
handleRequest @ serverconnection.js:193
makeRequest @ serverconnection.js:75
listRunning @ default.js:329
listRunning @ terminal.js:59
requestRunning @ manager.js:211
factory @ manager.js:46
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
dialog.js:32 Showing error: {message: "A connection to the Jupyter server could not be es…work connection or Jupyter server configuration.↵"}
showErrorMessage @ dialog.js:32
ConnectionLost @ connectionlost.js:12
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
_onConnectionFailure @ manager.js:105
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
(anonymous) @ manager.js:216
Promise.catch (async)
requestRunning @ manager.js:211
factory @ manager.js:46
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
serverconnection.js:193 GET http://localhost:8888/api/contents/filter/run/waveguides?content=1&1564140321949 net::ERR_CONNECTION_REFUSED
handleRequest @ serverconnection.js:193
makeRequest @ serverconnection.js:75
get @ index.js:473
get @ index.js:169
cd @ model.js:218
factory @ model.js:76
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
requestAnimationFrame (async)
schedule @ poll.js:196
async function (async)
schedule @ poll.js:186
refresh @ poll.js:142
_onFileChanged @ model.js:503
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
_onFileChanged @ index.js:390
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
(anonymous) @ index.js:641
Promise.then (async)
save @ index.js:639
save @ index.js:264
(anonymous) @ context.js:537
Promise.then (async)
_maybeSave @ context.js:522
(anonymous) @ context.js:416
Promise.then (async)
_save @ context.js:414
(anonymous) @ context.js:196
Promise.then (async)
save @ context.js:195
_save @ savehandler.js:109
(anonymous) @ savehandler.js:88
serverconnection.js:193 GET http://localhost:8888/api/sessions?1564140327122 net::ERR_CONNECTION_REFUSED
handleRequest @ serverconnection.js:193
makeRequest @ serverconnection.js:75
listRunning @ default.js:556
listRunning @ default.js:371
listRunning @ session.js:26
requestRunning @ manager.js:262
factory @ manager.js:45
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
dialog.js:32 Showing error: {message: "A connection to the Jupyter server could not be es…work connection or Jupyter server configuration.↵"}
showErrorMessage @ dialog.js:32
ConnectionLost @ connectionlost.js:12
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
_onConnectionFailure @ manager.js:105
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
(anonymous) @ manager.js:267
Promise.catch (async)
requestRunning @ manager.js:262
factory @ manager.js:45
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
serverconnection.js:193 GET http://localhost:8888/api/terminals?1564140327125 net::ERR_CONNECTION_REFUSED
handleRequest @ serverconnection.js:193
makeRequest @ serverconnection.js:75
listRunning @ default.js:329
listRunning @ terminal.js:59
requestRunning @ manager.js:211
factory @ manager.js:46
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
dialog.js:32 Showing error: {message: "A connection to the Jupyter server could not be es…work connection or Jupyter server configuration.↵"}
showErrorMessage @ dialog.js:32
ConnectionLost @ connectionlost.js:12
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
_onConnectionFailure @ manager.js:105
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
(anonymous) @ manager.js:216
Promise.catch (async)
requestRunning @ manager.js:211
factory @ manager.js:46
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
serverconnection.js:193 GET http://localhost:8888/api/contents/filter/run/waveguides?content=1&1564140332099 net::ERR_CONNECTION_REFUSED
handleRequest @ serverconnection.js:193
makeRequest @ serverconnection.js:75
get @ index.js:473
get @ index.js:169
cd @ model.js:218
factory @ model.js:76
_execute @ poll.js:242
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
setTimeout (async)
schedule @ poll.js:199
async function (async)
schedule @ poll.js:186
(anonymous) @ poll.js:247
Promise.then (async)
_execute @ poll.js:243
execute @ poll.js:192
requestAnimationFrame (async)
schedule @ poll.js:196
async function (async)
schedule @ poll.js:186
refresh @ poll.js:142
_onFileChanged @ model.js:503
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
_onFileChanged @ index.js:390
invokeSlot @ index.js:475
emit @ index.js:433
push.qUp9.Signal.emit @ index.js:106
(anonymous) @ index.js:641
Promise.then (async)
save @ index.js:639
save @ index.js:264
(anonymous) @ context.js:537
Promise.then (async)
_maybeSave @ context.js:522
(anonymous) @ context.js:416
Promise.then (async)
_save @ context.js:414
(anonymous) @ context.js:196
Promise.then (async)
save @ context.js:195
_save @ savehandler.js:109
(anonymous) @ savehandler.js:88
tom-programming
All 48 comments
I just tried this with jlab 1.0.3, FF 68, macOS. Here's what happened for me:
- Start up notebook server on port 8888
- Start up SSH tunnel using
ssh -L 8889:localhost:8888 localhost - Connect to the notebook server via port 8889: http://localhost:8889/lab
- Start notebook
- Kill ssh tunnel
- At this point every few seconds the dialog you mention comes up. I clicked Dismiss every time. I think I waited until about 4-5 dialogs had appeared and I dismissed them, so maybe 30 seconds total?
- Reconnect the SSH tunnel
- Evaluate a cell in the notebook. It waited a few seconds, then evaluated just fine. In other words, the reconnection was automatic without me having to do anything.
At no point did I see the notebook kernel switch to "No kernel!". The notebook continued to show the python kernel in the toolbar throughout the entire process.
It sounds like from above that your kernel had switched to "No kernel!" before you hit the reconnect. In that case, there is no kernel to reconnect to, so the command doesn't do anything.
jasongrout
on 26 Jul 2019
Can you try exactly the steps I did, and tell us when the notebook switches to "No kernel!"?
jasongrout
on 26 Jul 2019
It happens exactly when the dialog appears
tom-programming
on 26 Jul 2019
So if you do exactly the steps I listed above, the notebook switches to "no kernel" step 6 when the very first dialog appears?
What kernel are you using? What version of the notebook server?
jasongrout
on 26 Jul 2019
Yes I did exactly as you listed and the notebook switches to 'no kernel' at point 6.
I am using a Python kernel (python version is 3.7.3)
For the version I do not know how to find it but here are the relevant lines (I believe) from conda list command
ipykernel 5.1.1 py37h39e3cac_0 ipython 7.6.1 py37h39e3cac_0 ipython_genutils 0.2.0 py37_0 ipywidgets 7.5.0 py_0 jupyter 1.0.0 py37_7 jupyter_client 5.3.1 py_0 jupyter_console 6.0.0 py37_0 jupyter_core 4.5.0 py_0 jupyterlab 1.0.2 py37hf63ae98_0 jupyterlab_server 1.0.0 py_0 python 3.7.3 h0371630_0
tom-programming
on 28 Jul 2019
I can confirm behavior reported by @tom-programming , kernel is set to 'no kernel' after connection is lost. jlab 1.0.2
arogozhnikov
on 1 Aug 2019
jlab 1.0.2
Can you try with jlab 1.0.4?
jasongrout
on 1 Aug 2019
@tom-programming - can you also try with JupyterLab 1.0.4?
jasongrout
on 1 Aug 2019
I have JupyterLab 1.0.4 and get the 'No Kernel!' after a connection failure. "Reconnect to Kernel" also does not actually reconnect.
magic-lantern
on 2 Aug 2019
Can you write down explicit steps to reproduce this, including creation of the new environment? You can see my steps above at https://github.com/jupyterlab/jupyterlab/issues/6910#issuecomment-515470632 (I think I created my environment from conda-forge packages with conda create -n myenv jupyterlab)
jasongrout
on 2 Aug 2019
Steps I've followed:
On my Linux server:
ssh myhost.domain.tl -L 8888:localhost:8888
tmux
conda create -y -n fastai python=3.6
conda activate fastai
conda install -y -c pytorch pytorch torchvision cudatoolkit=9.0
conda install -y -c fastai fastai
conda install -y ipykernel nbconvert ipywidgets
conda install scikit-learn
conda install -c conda-forge jupyterlab
conda install -c conda-forge jupytext
jupyter lab --no-browser
On local macOS machine with Firefox 68.01, open http://localhost:8888/lab
Kill SSH (put laptop to sleep, go home, get disconnected, etc.) Now "No Kernel!" shows. Using the "Reconnect to Kernel" command does nothing. Kernel is still running. Any attempts to evaluate a cell fail.
magic-lantern
on 2 Aug 2019
What is the minimum time the connection is down and it shows no kernel? Likely there is a timeout. Can you just kill it and immediately reconnect?
jasongrout
on 2 Aug 2019
And does it still happen with a minimal environment?
conda create -y -n fastai -c conda-forge python=3.6 jupyterlab
# don't install anything else
jupyter lab --no-browser
For me, the "No Kernel!" message appears as soon as the "Server Connection Error" appears. That takes 2 - 3 seconds to appear.
If I close the browser tab, reconnect to the server running Jupyter Lab, then re-open the http://localhost:8888/lab it appears to reconnect to the kernel. However, any unsaved changes are lost.
Alternatively, if I do not close the browser tab, I can save any previously unsaved changes, but cannot ever reconnect the kernel.
Also, running the follow to create a new environment results in the same behaviour:
conda create -y -n tempenv -c conda-forge python=3.6 jupyterlab
conda activate tempenv
jupyter lab --no-browser
One question: This server I'm using does have a a bunch of different kernels available. Even when launching Jupyter Lab inside the new minimal conda env the other kernels show up. Could that be related to this issue?
magic-lantern
on 2 Aug 2019
One question: This server I'm using does have a a bunch of different kernels available. Even when launching Jupyter Lab inside the new minimal conda env the other kernels show up. Could that be related to this issue?
@magic-lantern I have a single env, so probably that's unrelated
arogozhnikov
on 3 Aug 2019
FWIW a user of mine complained about server instability, turned out to be unreiable wifi and this UX. I can reproduce the root problem by turning off the wifi on my laptop, waiting for the message, and turning it back on again. Jupyterlab 1.1.3 with some extensions, not easy for me to try a newer. Doesn't come back, ever after waiting several minutes. Firefox, deployed via JupyterHub 1.0. Could this relate to JupyterLab/JupyterHub integration where JH is supposed to be more aggressive about making the
If you reload the tab, the kernel is there again.
I ended up telling users: when you see that message, hit refresh. If server still alive, you get back to your default state (hopefully not losing unsaved work). If our server is actually stopped, same effect as pushing "restart".
I think that UI could be improved here for unsuspecting users: can we detect the difference between hub responding "no server" and no network connection, and handle no network differently? Or, in existing dialog, include a note that "It could be network problems, just wait and it may reconnect", then handle the reconnection seamlessly (the original issue here). Just making "dismiss" work still leaves confusing UX I think.
Same issue: #7081.
rkdarst
on 17 Oct 2019
(hopefully not losing unsaved work)
Note that if server connection is restored, you should still be able to save the file even if kernel is not connected.
vidartf
on 18 Oct 2019
FWIW, I've experienced this issue as well on jlab 1.0.3 and the worst of it is when you're waiting for cell output and lose network connectivity. In classic jupyter, we could use 'reconnect to kernel' in such cases after restoring the network connection and the cell output would be retrieved if it completed since losing the connection. In jlab, it doesn't seem possible to recover the cell output on reconnect.
atronchi
on 27 Jan 2020
+1. This is happening to me as well in Jupyterlab 1.1.4 with Firefox on Ubuntu.
Summary of my experience with this issue:
Jupyter lab notebook does not reconnect to existing kernel on a remote server after the ssh connection is re-established. The notebook reports "No Kernel!" and the notebook does not execute any cells, however, I am able to save the notebook as usual. I can reconnect the the still running kernel by selecting it in the kernel list under the heading "Use Kernel from Preferred Session". However, I am able to save the notebook after the ssh connection is made even without a kernel. I notice this issue after my local machine has been in suspend mode for at least an hour or two.
Note, that I did not experience this problem in prior setups with Jupyterlab or Jupyter classic.
bkusenda
on 27 Jan 2020
FWIW, I've experienced this issue as well on jlab 1.0.3 and the worst of it is when you're waiting for cell output and lose network connectivity. In classic jupyter, we could use 'reconnect to kernel' in such cases after restoring the network connection and the cell output would be retrieved if it completed since losing the connection. In jlab, it doesn't seem possible to recover the cell output on reconnect.
Can you try with the new JupyterLab prerelease, 2.0b3? I just tried:
- Create a new environment with something like:
conda create -n connect -c conda-forge notebook nodejs pip conda activate connect pip install --pre jupyterlab - Start up jlab server on port 8888:
jupyter lab --port=8888 --no-browser - Start up SSH tunnel using
ssh -L 8889:localhost:8888 localhost - Connect to jlab via port 8889: http://localhost:8889/lab
- Start notebook and evaluate:
import time for i in range(100): print(i) time.sleep(1) - Kill ssh tunnel
- At this point every few seconds the dialog you mention comes up. I clicked Dismiss every time. I think I waited until about 4-5 dialogs had appeared and I dismissed them, so maybe 30 seconds total?
- Reconnect the SSH tunnel
- Either wait or execute the "Reconnect to Kernel" command from the command palette (I tried both). All of my missing output fills in without missing any numbers, and it keeps counting as well.
jasongrout
on 30 Jan 2020
Any idea when the pre-release will be generally released?
nelder
on 10 Feb 2020
Any idea when the pre-release will be generally released?
Are you asking when 2.0 final will be released? Hopefully very very soon (i.e., hopefully a matter of days). The RC is out now, and we're waiting for the dust to settle on one more thing before releasing final.
jasongrout
on 10 Feb 2020
I had the same issue. Running the Jupyter kernel from within a Linux screen might help. connect to the remote server and type:
screen
Then proceed to open a screen session, and type:
jupyter notebook --no-browser --port xxxx
Now open another terminal and tunnel:
ssh -N -L xxxx:localhost:xxxx [email protected]
After this the host jupyter should be able to recover even if connection is lost.
MohammadSamragh
on 31 Mar 2020
I have the same issue with Jlab 1.2.6. As soon as my ssh disconnects it switches to No Kernel! I use it for deep learning so I typically set it to run for several days, but because my computer is inactive during the night, when get up in the morning (like right now), it says "No Kernel!".
I use automatic reconnect to ssh on disconnect via MobaXTerm, so it would have been more or less instantaneous reconnect, but still Jlab is unable to reconnect, neither automatically nor via the reconnect to kernel command. What to do? I have a running kernel right now that I cannot reestablish connection to and I expect it to keep running for an additional 40 hours.
bjtho08
on 7 Apr 2020
For what it's worth, what has worked for me when you know your connection is going to be unstable (e.g. you are restarting your VPN connection), is to have your Chrome browser focused on a different tab than the JupyterLab tab. This somehow prevents the JupyterLab tab from noticing that it has lost the connection. When your connection is reestablished, you can come back to the JupyterLab tab, and the kernel should also be connected.
thoralf-gutierrez
on 7 Apr 2020
Same here.
agingnan
on 19 Apr 2020
This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:
https://discourse.jupyter.org/t/frequent-restart-dismiss-prompts-and-nokernel-problems/4066/1
meeseeksmachine
on 20 Apr 2020
I can confirm the same behavior, kernel is set to 'no kernel' after connection is lost (jlab 1.2.6).
We can see this behaviour using a tunnel or a jupyterhub.
jhgoebbert
on 30 Apr 2020
@MohammadSamragh I always start jupyter lab --no-browser in screen. But that does not solve this issue, as it is not a matter of the Jupyter Notebook Server stop running, but the connection to the kernel get lost.
jhgoebbert
on 30 Apr 2020
@jasongrout any updates on this? or suggestions for possible debugging steps apart from what has already been mentioned?
bjtho08
on 30 Apr 2020
Can you reproduce my experiment above with JupyterLab 2.1?
Can you come up with a similar simple reproducible case illustrating your issue with JupyterLab 2.1?
jasongrout
on 30 Apr 2020
@jasongrout JupyterLab 2.1 seems to solve the problem. Great! And thanks a lot!
Until now I have tested JupyterLab 2.1 setting up the tunnel manually and not yet using JupyterHub. I will have to make more tests to finally confirm, that the problem disappeared in version 2.1
This would be great news for all the DeepLearning users! No work-around with 'papermill' anymore :)
I noticed one minor thing: after reconnect the kernel icon in the top right corner of the notebook does not indicate that the kernel is working by showing the fully-filled circle, but it shows a not-filled one, even though the cell shows a [*] for 'running'.
jhgoebbert
on 30 Apr 2020
We did rewrite the connection logic in jlab 2.0, so that's great if it solves the problem!
When jlab initially connects to a kernel, it does not know the kernel status until it gets a status message, which may only happen after the current computation is finished. The cell shows [*] because it hasn't been told the computation was stopped, and the kernel icon doesn't show busy because it doesn't know the status. Perhaps that kernel icon should have a separate visual state to indicate that it does not know the kernel status yet.
jasongrout
on 30 Apr 2020
I can confirm that that 2.1 also solved the issue on my end! đź‘Ť
Regarding the kernel status, would it be possible to add some logic that pushes a notification to the client on reconnect to inform of the kernel status?
bjtho08
on 1 May 2020
Worked for me as well. If the connection is interrupted I return to the kernel, and also I can close the browser, open it again, and reconnect to the running kernel. Thank you very much!
noamraph
on 4 May 2020
@jasongrout can you help me reconnect to a running kernel from a different machine? I use X2Go to connect to a machine running jupyterlab and unfortunately X2Go sometimes crashes and I have to create a new session and then reconnect to the still running jupyterlab, but my session cookie appears to be different and I am unable to receive the buffered messages.
bjtho08
on 5 May 2020
The current buffering just covers the case of the same session reconnecting and is implemented in the notebook server. A separate effort is to move the underlying notebook state to the server, and is related to the efforts towards real-time collaboration. That would give more robust reconnection logic (similar to what Cocalc has, for example), but is significantly more work.
jasongrout
on 5 May 2020
Closing since it seems that the original issue seems to be solved with jlab 2.1. The real-time collaboration discussion (which also is part of the conversation to move the notebook state server-side) is over at #5382.
jasongrout
on 5 May 2020
Okay, so currently it is not possible? I should have switched to ssh once I upgraded to Jlab 2.1 - I only use this setup because that way I could preserve my session by suspending and resuming X2Go sessions, but now that it has started crashing on me, that really doesn't work at all. Of course it is terrible timing because I'm running a week-long experiment at the moment. Fortunately, I am writing my results to disk.
bjtho08
on 5 May 2020
Right now the logic in the Jupyter Notebook server is to replay buffered messages only to the same client session: https://github.com/jupyter/notebook/blob/d04cbb6a98b1c73b6cfb9fc8af969ca0c95c13d2/notebook/services/kernels/kernelmanager.py#L241-L263
jasongrout
on 5 May 2020
Fortunately, I am writing my results to disk.
+1 to checkpointing results!
jasongrout
on 5 May 2020
Right now the logic in the Jupyter Notebook server is to replay buffered messages only to the same client session: https://github.com/jupyter/notebook/blob/d04cbb6a98b1c73b6cfb9fc8af969ca0c95c13d2/notebook/services/kernels/kernelmanager.py#L241-L263
I haven't lookked at the code yet, but do you reckon it is too complicated to try and spoof the server to believe I am identical to the previous session?
bjtho08
on 5 May 2020
IIRC, even refreshing JupyterLab will generate a new client session id for the connection, and thus the messages will be dropped. One thing you could try is commenting out the code above that looks for the client session id, and just send the messages to whoever connects first. I haven't thought carefully through the ramifications of that behavior.
jasongrout
on 5 May 2020
Ah, yes, but that would require that I restart Jupyterlab, thereby killing the kernel, correct?
bjtho08
on 5 May 2020
Yes, correct for your current run. I meant trying to modify this logic for future runs.
jasongrout
on 5 May 2020
Okay, thanks for the suggestion. I think I will be switching to ssh forwarding after this run, though, so it might not be worth the trouble.
bjtho08
on 5 May 2020
I have now tried editing the session_keycheck on line 257 in the get_buffer() method, so that it only check if there is a session key rather than if the keys match:
def get_buffer(self, kernel_id, session_key):
"""Get the buffer for a given kernel
Parameters
----------
kernel_id : str
The id of the kernel to stop buffering.
session_key: str, optional
The session_key, if any, that should get the buffer.
If the session_key matches the current buffered session_key,
the buffer will be returned.
"""
self.log.debug("Getting buffer for %s", kernel_id)
if kernel_id not in self._kernel_buffers:
return
buffer_info = self._kernel_buffers[kernel_id]
- if buffer_info['session_key'] == session_key:
+ if buffer_info['session_key']:
# remove buffer
self._kernel_buffers.pop(kernel_id)
# only return buffer_info if it's a match
return buffer_info
else:
self.stop_buffering(kernel_id)
This did not allow me to resume after e.g. a browser crash or client reboot. Were you referring to something else, @jasongrout, or should this have worked, in your opinion?
bjtho08
on 13 May 2020
I think it should have worked, or at least got closer to working. The next step I would check is to see if the buffered messages were transferred over a kernel websocket connection to the frontend (and which kernel websocket).
jasongrout
on 13 May 2020
Related issues
rutgerhofste
·
45Comments
anshbansal
·
68Comments
rongmu
·
50Comments
minrk
·
126Comments
mgeier
·
48Comments
Most helpful comment
FWIW, I've experienced this issue as well on jlab 1.0.3 and the worst of it is when you're waiting for cell output and lose network connectivity. In classic jupyter, we could use 'reconnect to kernel' in such cases after restoring the network connection and the cell output would be retrieved if it completed since losing the connection. In jlab, it doesn't seem possible to recover the cell output on reconnect.