Notebook: can't conda update because phantom notebook server running

[eseiver]

I couldn't update my Python 3 conda environment today and the NotebookRunningError messages indicated it was because a Jupyter notebook server was running. I had just booted up my machine within the last hour and had not launched a notebook server. I'm on a MBP running macOS Sierra.

Here's the result of my command line tests:

Checking what notebook servers are "running"

jupyter notebook list: 
Currently running servers:
http://localhost:8888/ :: /Users/eseiver/anaconda3
http://localhost:8888/?token=[token] :: /Users/eseiver/PLOS_Corpus_Project

[also weird b/c I am using miniconda now not anaconda]

Checking if there is a background process in screen

screen -r: 
There is no screen to be resumed.

Check all processes

1) Following https://github.com/jupyter/notebook/issues/1950#issuecomment-318512976
pgrep jupyter
no output i.e. no processes running with 'jupyter' in name
[though I understand this doesn't work on Macs?]
2) Grepping the process list directly
ps aux | grep jupyter: eseiver 3144 0.0 0.0 2473332 4 s000 S 12:56PM 0:00.00 grep jupyter
[pretty sure this means it's only finding the grep process]
3) See which processes are listening on port 8888, based on this SO post
lsof -Pi :8888 -sTCP:LISTEN
no output, i.e. port 8888 is not registering any activity

Check that it is accessible in browser

http://localhost:8888/ tested in the browser; connection refused.

@mpacer @minrk @Carreau

[eseiver]

Here's some of the error output:

LinkError: post-link script failed for package conda-forge::jupyter_contrib_nbextensions-0.3.1-py36_0
running your command again with `-v` will provide additional information
location of failed script: /Users/eseiver/miniconda3/envs/py3/bin/.jupyter_contrib_nbextensions-post-link.sh
==> script messages <==
+ /Users/eseiver/miniconda3/envs/py3/bin/jupyter-contrib-nbextension install --sys-prefix
[I 10:54:16 InstallContribNbextensionsApp] jupyter contrib nbextension install --sys-prefix
[W 10:54:16 InstallContribNbextensionsApp] Error: Cannot configure while the Jupyter notebook server is running
[I 10:54:16 InstallContribNbextensionsApp] To perform actions even while a notebook server is running,you may use the flag
    --skip-running-check
Traceback (most recent call last):
  File "/Users/eseiver/miniconda3/envs/py3/bin/jupyter-contrib-nbextension", line 11, in <module>
    load_entry_point('jupyter-contrib-nbextensions==0.3.1', 'console_scripts', 'jupyter-contrib-nbextension')()
  File "/Users/eseiver/miniconda3/envs/py3/lib/python3.6/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/Users/eseiver/miniconda3/envs/py3/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/Users/eseiver/miniconda3/envs/py3/lib/python3.6/site-packages/jupyter_contrib_nbextensions/application.py", line 226, in start
    super(ContribNbextensionsApp, self).start()
  File "/Users/eseiver/miniconda3/envs/py3/lib/python3.6/site-packages/jupyter_core/application.py", line 256, in start
    self.subapp.start()
  File "/Users/eseiver/miniconda3/envs/py3/lib/python3.6/site-packages/jupyter_contrib_nbextensions/application.py", line 173, in start
    toggle_install_files(self._toggle_value, **kwargs_files)
  File "/Users/eseiver/miniconda3/envs/py3/lib/python3.6/site-packages/jupyter_contrib_nbextensions/install.py", line 60, in toggle_install_files
    'Cannot configure while the Jupyter notebook server is running')
jupyter_contrib_nbextensions.install.NotebookRunningError: Cannot configure while the Jupyter notebook server is running

[eseiver]

Another update after much detective work: checked runtime info in Library/Jupyter/runtime/ and there's an nbserver json file that includes server information:

{
  "base_url": "/",
  "hostname": "localhost",
  "notebook_dir": "/Users/eseiver/PLOS_Corpus_Project",
  "password": false,
  "pid": 1287,
  "port": 8888,
  "secure": false,
  "token": [TOKEN],
  "url": "http://localhost:8888/"
}

I'll delete this file, but I suspect there's some kind of caching/clean-up issue at play here.

Update: running notebook.notebookapp.list_running_servers() (code) did NOT delete that file

[mpacer]

So looking at the linked code:

https://github.com/jupyter/notebook/blob/d4c6fe46cea2d7daa6dbe2647d766eec57ebb56e/notebook/notebookapp.py#L1525-L1532

It would be nice to have the cleanup functionality separated from the listing functionality. That way we could define a separate app just to handle this kind of use case. Telling people that the only way to clean up these files automatically is via the "list" functionality which magically cleans stuff up in the background seems confusing at best.

At the least we should have a way to raise an explicit error rather than silently pass if removing a file fails. Ideally, that error would include a message informing the user of which file they might need to rm manually because the auto-removal failed.

[Carreau]

The "cannot upgrade" should also be fixed in the contrib repository. There should be a way to force upgrade, and not track that file down manually.

The file not being deleted is likely because PID are reused – they are assigned in sequence –, we could – for example – store in the json file the date at which the server was started, and check if that was before (now minus machine uptime), in which case we can remove the file, as the PID has likely been given to another process.

[Carreau]

There seem to be an uptime package on PyPI that provides a boottime() function.

[mpacer]

I wanted to say "Couldn't we also check the process name currently attached to the PID?" and then proceed to suggest appropriate logic to handle watching for "jupyter/ipython" named processes (though that wouldn't be foolproof)… however I just spent some time looking at how you would get the name of a process given it's PID… and it seems pretty hard to do in an OS independent way… and even if we continued our OS dependence in (our code) it seems like it's hard to do without introducing a new dependency like this SO post suggests with a package like psutil.

[Carreau]

Yes, and in many case we could not even find "jupyter" or "ipython" in the process name, we would just get python. The other possibility would be to make a request to given ip/port and see if we get a reply. But I'm unsure this is reasonable.

[minrk]

The other possibility would be to make a request to given ip/port and see if we get a reply.

This ought to work. It should identify itself as a tornado server in the headers, at least. We do have something tricky in this particular case because there are two leftover runtime files, each claiming to be on the same http endpoint (unsurprising, since it's the default). That's probably pretty common, come to think of it.

@eseiver I think you're exactly right that a runtime file was leftover after a server stopped, for whatever reason. Removing it by hand is an AOK workaround while we figure out why the file was leftover and how we can do better cleaning up after this event occurs. In general, the runtime directory should be safe to delete in its entirety if no Jupyter processes are running.