Zero-to-jupyterhub-k8s: Unable to start singleuser when user pv is full

[clkao]

This is probably a jupyterlab issue, but worth tracking here as full user pv will render the singleuser notebook unusable and require manual intervention. There should be a fail-safe mode that the server can be started and user is advised to cleanup home directory.

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-labhub", line 11, in <module>
    load_entry_point('jupyterlab==0.31.3', 'console_scripts', 'jupyter-labhub')()
  File "/opt/conda/lib/python3.6/site-packages/jupyterlab/labhubapp.py", line 32, in main
    return SingleUserLabApp.launch_instance(argv)
  File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 368, in start
    super(SingleUserNotebookApp, self).start()
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1453, in start
    self.write_server_info_file()
  File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1412, in write_server_info_file
    json.dump(self.server_info(), f, indent=2, sort_keys=True)
OSError: [Errno 28] No space left on device

[willingc]

cc/ @blink1073 Have you seen this issue?

[yuvipanda]

@minrk any idea how to deal with full disks from notebook?

Thanks for reporting this, @clkao!

[blink1073]

No, I have not seen this issue. Min would know better if the server could function without creating this file. AFAIK no other files are created, since most state is in in-memory sqlite dbs.

[minrk]

This file is only used for things like jupyter notebook list, so it should be safe to launch without it. I suspect there are several stages where small runtime files are created during startup (e.g. the notebook signatures database file, which is needed, but probably already exists on disk if it's full), so it may be a bit of whack-a-mole to make sure that it starts up without touching any files.

[clkao]

Apart from this, there's actually nowhere in the UI for emptying trash. Perhaps the notebook image should just do that in the startup script.

[minrk]

I think automatically emptying trash without user input is something that shouldn't be done in general. However, if this is being caused by the new support for send2trash in the notebook, I think it makes sense to disable that by default in jupyterhub-singleuser as you say, since there is no UI for the trash in a hosted deployment.

[minrk]

https://github.com/jupyter/notebook/pull/3384 makes these non-fatal errors and https://github.com/jupyterhub/jupyterhub/pull/1688 disables the trash mechanism by default in jupyterhub