Jupyterhub: 500 Internal Server Error

[danielrychel]

I have configured jupyterhub through my AWS instance using the following commands:

wget https://gist.githubusercontent.com/lambdalisue/f01c5a65e81100356379/raw/ecf427429f07a6c2d6c5c42198cc58d4e332b425/jupyterhub
sudo mv jupyterhub /etc/init.d/jupyterhub
sudo chmod +x /etc/init.d/jupyterhub
sudo mkdir /etc/jupyterhub
sudo jupyterhub --generate-config -f /etc/jupyterhub/jupyterhub_config.py
sudo service jupyterhub start

I am using GitHub Authentication with my jupyterhub and I am still able to sign in to it on my domain and everything looks like its working correctly but when I click on "Start a Server" I get a 500 error (Internal Server Error, Spawner failed to start [status=1]) I have looked around GitHub for a solution but have not found anything yet that seems to apply to me.

Here is what is logged:

[I 2018-05-31 16:51:15.662 JupyterHub spawner:978] Spawning jupyterhub-singleuser --port=33772 '--notebook-dir="~/assignments"' --NotebookApp.default_url=/notebooks/Welcome.ipynb
[W 2018-05-31 16:51:25.683 JupyterHub web:1618] 500 GET /hub/user/danielrychel/ (::ffff:130.126.255.107): Spawner failed to start [status=1]. The logs for danielrychel may contain details.
[E 2018-05-31 16:51:25.688 JupyterHub log:114] {
      "Upgrade-Insecure-Requests": "1",
      "X-Forwarded-Port": "443",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36",
      "Accept-Encoding": "gzip, deflate, br",
      "Connection": "close",
      "Host": "zeus.uiuc-bdeep.org",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
      "Accept-Language": "en-US,en;q=0.9",
      "X-Forwarded-For": "::ffff:130.126.255.107",
      "X-Forwarded-Host": "zeus.uiuc-bdeep.org",
      "X-Forwarded-Proto": "https",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1527706761|17:jupyter-hub-token|44:MTBkZjgwYTIzZThlNDFmYWEyOTVmNDA4N2QzMmZlMjU=|3d716ae4389c01961f7e83e5c78110a05e0850b1e473ef351afa66049b21b9a3\"",
      "Referer": "https://zeus.uiuc-bdeep.org/hub/home"
    }
[E 2018-05-31 16:51:25.688 JupyterHub log:122] 500 GET /hub/user/danielrychel/ (danielrychel@::ffff:130.126.255.107) 10060.07ms
[W 2018-05-31 16:51:45.704 JupyterHub user:458] danielrychel's server never showed up at http://127.0.0.1:33772/user/danielrychel/ after 30 seconds. Giving up
[E 2018-05-31 16:51:45.715 JupyterHub gen:940] Exception in Future <Future finished exception=TimeoutError("Server at http://127.0.0.1:33772/user/danielrychel/ didn't respond in 30 seconds",)> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 936, in error_callback
        future.result()
      File "/usr/lib/python3.4/asyncio/futures.py", line 275, in result
        raise self._exception
      File "/usr/local/lib/python3.4/dist-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
        yield spawn_future
      File "/usr/lib/python3.4/asyncio/futures.py", line 275, in result
        raise self._exception
      File "/usr/local/lib/python3.4/dist-packages/jupyterhub/user.py", line 476, in spawn
        raise e
      File "/usr/local/lib/python3.4/dist-packages/jupyterhub/user.py", line 450, in spawn
        resp = yield server.wait_up(http=True, timeout=spawner.http_timeout)
      File "/usr/lib/python3.4/asyncio/futures.py", line 275, in result
        raise self._exception
      File "/usr/local/lib/python3.4/dist-packages/jupyterhub/utils.py", line 180, in wait_for_http_server
        timeout=timeout
      File "/usr/lib/python3.4/asyncio/futures.py", line 275, in result
        raise self._exception
      File "/usr/local/lib/python3.4/dist-packages/jupyterhub/utils.py", line 135, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://127.0.0.1:33772/user/danielrychel/ didn't respond in 30 seconds

I followed https://github.com/jupyterhub/sudospawner/issues/44 and changed the spawner ip to 127.0.0.1 but that did not fix the problem, so I'm fairly certain that the TimeoutError is caused by jupyterhub not starting.

Here is my output from jupyter troubleshoot:

```
$PATH:
/usr/local/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/games
/usr/local/games

sys.path:
/usr/local/bin
/usr/lib/python3.4
/usr/lib/python3.4/plat-x86_64-linux-gnu
/usr/lib/python3.4/lib-dynload
/usr/local/lib/python3.4/dist-packages
/usr/lib/python3/dist-packages

sys.executable:
/usr/bin/python3

sys.version:
3.4.3 (default, Nov 28 2017, 16:41:13)
[GCC 4.8.4]

platform.platform():
Linux-3.13.0-143-generic-x86_64-with-Ubuntu-14.04-trusty

which -a jupyter:
/usr/local/bin/jupyter
/usr/local/bin/jupyter

pip list:
Package Version
------------------- ----------------
alembic 0.9.9
backcall 0.1.0
backports-abc 0.5
bleach 2.1.3
chardet 2.2.1
colorama 0.2.5
command-not-found 0.3
decorator 4.3.0
entrypoints 0.2.3
html5lib 1.0.1
ipykernel 4.8.2
ipython 6.4.0
ipython-genutils 0.2.0
ipywidgets 7.2.1
jedi 0.12.0
Jinja2 2.10
jsonschema 2.6.0
jupyter-client 5.2.3
jupyter-core 4.4.0
jupyterhub 0.8.1
language-selector 0.1
Mako 1.0.7
MarkupSafe 1.0
mistune 0.8.3
nbconvert 5.3.1
nbformat 4.4.0
notebook 5.5.0
oauthenticator 0.7.3
pamela 0.3.0
pandocfilters 1.4.2
parso 0.2.1
pexpect 4.6.0
pickleshare 0.7.4
pip 10.0.1
prompt-toolkit 1.0.15
ptyprocess 0.5.2
pycurl 7.19.3
Pygments 2.2.0
pygobject 3.12.0
pyparsing 2.2.0
python-apt 0.9.3.5ubuntu2
python-dateutil 2.7.3
python-editor 1.0.3
python-oauth2 1.1.0
pyzmq 17.0.0
requests 2.2.1
Send2Trash 1.5.0
setuptools 39.2.0
simplegeneric 0.8.1
six 1.11.0
SQLAlchemy 1.2.7
terminado 0.8.1
testpath 0.3.1
tornado 5.0.2
traitlets 4.3.2
typing 3.6.4
ufw 0.34-rc-0ubuntu2
unattended-upgrades 0.1
urllib3 1.7.1
wcwidth 0.1.7
webencodings 0.5.1
wheel 0.24.0
widgetsnbextension 3.2.1```

[minrk]

Can you start jupyterhub with --debug? What Spawner are you using? Can your share your jupyterhub_config.py (minus any sensitive info)?

[danielrychel]

@minrk Thanks for the response, sorry I got wrapped up working on some other aspects of the hub. I switched to follow this video and added github authentication. This got me past the 500 Internal Server Error when signed in to the base 'ubuntu' account, which is able to spawn servers perfectly but the 500 error comes back when I try to start a server on any other account. Here is the relevant parts of my jupyterhub_config.py file:

c.JupyterHub.ssl_key = '/etc/letsencrypt/live/jupyter.uiuc-bdeep.org/privkey.pem'
c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/jupyter.uiuc-bdeep.org/fullchain.pem'
c.JupyterHub.port = 443
from oauthenticator.github import LocalGitHubOAuthenticator
c.JupyterHub.authenticator_class = LocalGitHubOAuthenticator
c.LocalGitHubOAuthenticator.create_system_users = True
c.Authenticator.whitelist = {'danielrychel', ...}
c.Authenticator.admin_users = {'danielrychel'}

As for the spawner I am using, it looks like it is the default single-user spawner. I tried to set this up with a sudospawner according to this link but that did not work either. Here is my error/debug log when I attempt to spawn a server from my github account (not the 'ubuntu' account):

[I 2018-06-11 19:49:25.320 JupyterHub log:122] 200 GET /hub/login (@::ffff:130.126.255.107) 5.42ms
[I 2018-06-11 19:49:27.184 JupyterHub oauth2:82] OAuth redirect: 'https://jupyter.uiuc-bdeep.org/hub/oauth_callback'
[I 2018-06-11 19:49:27.186 JupyterHub log:122] 302 GET /hub/oauth_login?next= → https://github.com/login/oauth/authorize?redirect_uri=https%3A%2F%2Fjupyter.uiuc-bdeep.org%2Fhub%2Foauth_callback&client_id=0bc9d83c63045fce2f4b&response_type=code&state=eyJzdGF0ZV9pZCI6ICJlMjNjZDc5ODMxM2U0ZTQxOWVmMjkwMjhkYTFiMmE1NyIsICJuZXh0X3VybCI6ICIifQ%3D%3D (@::ffff:130.126.255.107) 1.31ms
[D 2018-06-11 19:49:27.650 JupyterHub base:268] Setting cookie for danielrychel: jupyter-hub-token, {'httponly': True, 'secure': True}
[I 2018-06-11 19:49:27.651 JupyterHub base:346] User logged in: danielrychel
[I 2018-06-11 19:49:27.651 JupyterHub log:122] 302 GET /hub/oauth_callback?code=fd1931491b656fe521d9&state=eyJzdGF0ZV9pZCI6ICJlMjNjZDc5ODMxM2U0ZTQxOWVmMjkwMjhkYTFiMmE1NyIsICJuZXh0X3VybCI6ICIifQ%3D%3D → /hub/ (@::ffff:130.126.255.107) 185.43ms
[D 2018-06-11 19:49:27.751 JupyterHub pages:54] User is not running: /hub/home
[I 2018-06-11 19:49:27.753 JupyterHub log:122] 302 GET /hub/ → /hub/home (danielrychel@::ffff:130.126.255.107) 1.76ms
[D 2018-06-11 19:49:27.853 JupyterHub log:122] 304 GET /hub/home (danielrychel@::ffff:130.126.255.107) 3.24ms
[I 2018-06-11 19:49:29.146 JupyterHub log:122] 302 GET /hub/spawn → /user/danielrychel/ (danielrychel@::ffff:130.126.255.107) 4.12ms
[I 2018-06-11 19:49:29.225 JupyterHub log:122] 302 GET /user/danielrychel/ → /hub/user/danielrychel/ (@::ffff:130.126.255.107) 0.45ms
[D 2018-06-11 19:49:29.311 JupyterHub base:421] Initiating spawn for danielrychel
[I 2018-06-11 19:49:29.341 JupyterHub spawner:978] Spawning jupyterhub-singleuser --port=52776
[E 2018-06-11 19:49:29.349 JupyterHub user:427] Unhandled error starting danielrychel's server: Exception occurred in preexec_fn.
[D 2018-06-11 19:49:29.360 JupyterHub base:427] 0/100 concurrent spawns
[D 2018-06-11 19:49:29.360 JupyterHub base:430] 0 active servers
[E 2018-06-11 19:49:29.360 JupyterHub web:1621] Uncaught exception GET /hub/user/danielrychel/ (::ffff:130.126.255.107)
    HTTPServerRequest(protocol='https', host='jupyter.uiuc-bdeep.org', method='GET', uri='/hub/user/danielrychel/', version='HTTP/1.1', remote_ip='::ffff:130.126.255.107')
    Traceback (most recent call last):
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/site-packages/tornado/web.py", line 1543, in _execute
        result = yield result
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/asyncio/futures.py", line 274, in result
        raise self._exception
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 747, in get
        yield self.spawn_single_user(current_user)
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/asyncio/futures.py", line 274, in result
        raise self._exception
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 475, in spawn_single_user
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future)
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/asyncio/futures.py", line 274, in result
        raise self._exception
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/site-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
        yield spawn_future
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/asyncio/futures.py", line 274, in result
        raise self._exception
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/site-packages/jupyterhub/user.py", line 439, in spawn
        raise e
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/site-packages/jupyterhub/user.py", line 378, in spawn
        ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/asyncio/futures.py", line 274, in result
        raise self._exception
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/types.py", line 243, in wrapped
        coro = func(*args, **kwargs)
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/site-packages/jupyterhub/spawner.py", line 988, in start
        self.proc = Popen(cmd, **popen_kwargs)
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/subprocess.py", line 950, in __init__
        restore_signals, start_new_session)
      File "/home/ubuntu/anaconda3/envs/jupyterhub-tutorial/lib/python3.5/subprocess.py", line 1541, in _execute_child
        raise child_exception_type(err_msg)
    subprocess.SubprocessError: Exception occurred in preexec_fn.
[D 2018-06-11 19:49:29.364 JupyterHub base:633] No template for 500
[E 2018-06-11 19:49:29.372 JupyterHub log:114] {
      "X-Forwarded-For": "::ffff:130.126.255.107",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.79 Safari/537.36",
      "X-Forwarded-Host": "jupyter.uiuc-bdeep.org",
      "X-Forwarded-Proto": "https",
      "Connection": "close",
      "Accept-Language": "en-US,en;q=0.9",
      "Host": "jupyter.uiuc-bdeep.org",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
      "X-Forwarded-Port": "443",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1528746567|17:jupyter-hub-token|44:NjIzMWRlZWU4NWQ5NGJkMjgyNDNiMWViZmNjNmQ0ODc=|f82262e4cf81186434be6c5f5e427269ecd52aa82f9cc9aceb0b69b3668c84a3\"; _xsrf=2|a03036a7|02ce5338dc5be67e280f1c1634c6c8f7|1528742754",
      "Referer": "https://jupyter.uiuc-bdeep.org/hub/home",
      "Upgrade-Insecure-Requests": "1"
    }
[E 2018-06-11 19:49:29.372 JupyterHub log:122] 500 GET /hub/user/danielrychel/ (danielrychel@::ffff:130.126.255.107) 61.17ms

Any advice or tutorials from some other spawner would be greatly appreciated!

[danielrychel]

I was able to resolve the issue using the sudospawner, I just had an incorrect path to it. I can now use it to run multiple servers simultaneously, so I will close the issue now.

[gabefair]

How can I use the sudospawner?