Jupyterhub: Problem when runing jupyterhub.

[beeva-luisgonzalez]

Hi guys,
I've really looked into the forums (usergroup, internet, stackoverflow, etc..), and closed bugs but I'm unable to find info on a similar problem.

After installing installing all the requirements, whenever I try to run the jupyterhub command I get the following error (and I'm unable to know whats causing it).

[luix@boxita jupyterhub]$ jupyterhub
Traceback (most recent call last):
  File "/usr/bin/jupyterhub", line 3, in <module>
    from jupyterhub.app import main
  File "/usr/lib/python3.4/site-packages/jupyterhub/app.py", line 28, in <module>
    import tornado.httpserver
  File "/usr/lib/python3.4/site-packages/tornado-4.2b1-py3.4-linux-x86_64.egg/tornado/httpserver.py", line 34, in <module>
    from tornado.http1connection import HTTP1ServerConnection, HTTP1ConnectionParameters
  File "/usr/lib/python3.4/site-packages/tornado-4.2b1-py3.4-linux-x86_64.egg/tornado/http1connection.py", line 26, in <module>
    from tornado.concurrent import Future
  File "/usr/lib/python3.4/site-packages/tornado-4.2b1-py3.4-linux-x86_64.egg/tornado/concurrent.py", line 37, in <module>
    from concurrent import futures
  File "/usr/lib/python3.4/site-packages/concurrent/futures/__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File "/usr/lib/python3.4/site-packages/concurrent/futures/_base.py", line 355
    raise type(self._exception), self._exception, self._traceback
                               ^
SyntaxError: invalid syntax

my system information is

[luix@boxita jupyterhub]$ python --version
Python 3.4.3
[luix@boxita jupyterhub]$ ipython --version
3.2.1
[luix@boxita jupyterhub]$ uname -a
Linux boxita 4.1.2-2-ARCH #1 SMP PREEMPT Wed Jul 15 08:30:32 UTC 2015 x86_64 GNU/Linux

any ideas what can be??

[minrk]

I think you've got a somewhat peculiar version of the concurrent package, which is a backport of part of the stdlib in Python 3.4, only for use on Python 2. My guess is this is a consequence of using easy_install, which is no longer recommended for anything.

Try:

pip uninstall concurrent

[beeva-luisgonzalez]

I don't have installed concurrent apparently

[luix@boxita jupyterhub]$ pip uninstall concurrent
Cannot uninstall requirement concurrent, not installed
[luix@boxita jupyterhub]$ pip2 uninstall concurrent
Cannot uninstall requirement concurrent, not installed
[luix@boxita jupyterhub]$ pip3 uninstall concurrent
Cannot uninstall requirement concurrent, not installed

[minrk]

Ah, sorry. I think the package is called futures. I think you want pip3 uninstall futures. What do you get from pip3 list?

[bjlange]

Was there ever any resolution on this? I'm running into the same issue now. I'm thinking it has to do with the fact that I have Python 2 Anaconda (4.0) installed on the system and used for most purposes, and also Python 3 where JupyterHub is installed. I was able to get it running in the past, but something's changed it's not working now.

I'll post anything I discover in here.

[minrk]

@bjlange there should be no concurrent directory in your site-packages folder. This is a standard-library module in Python 3. I recommended removing it with the command above, but manual intervention may be necessary (you can delete the folder). Also, if you have any easy-install.pth files in any site-packages directory, those should probably also be removed unless you have development-installs of packages (pip install -e or setup.py develop). They are used by setuptools/easy-install to do deprecated, extremely problematic things that are no longer necessary and actively destructive when you use pip.

[bjlange]

Thanks @minrk, this was really helpful. I didn't have concurrent in my Python 3 packages. But I did have it in my 2.7 packages, which would be fine except that that I was setting the PYTHONPATH globally via .bashprofile to look in Python 2.7 _first_, so JupyterHub loaded that concurrent even though it was being executed with Python 3. Removing that seemed to resolve the issue.

[andreapago]

Hi,
I am experiencing the same issue as above.

if I issue the command

./pip3 uninstall futures
Cannot uninstall requirement futures, not installed

I don't have the future in the site-packages but it is in other locations

/usr/lib64/python3.4/concurrent/futures
/opt/rh/rh-python34/root/usr/lib64/python3.4/concurrent/futures

Any idea?

Thanks in advance

[minrk]

@andreapago your concurrent.futures look appropriate and correct. Are you seeing the same SyntaxError? If so, what is the file where the exception occurs? Are you perhaps using PYTHONPATH? If so, what is it's value?

[andreapago]

Hi @minrk

thanks for your reply.
I am seeing the following:

[E 2016-06-06 08:33:21.460 JupyterHub user:237] Unhandled error starting andrea's server: [Errno 13] Permission denied
[E 2016-06-06 08:33:21.481 JupyterHub web:1524] Uncaught exception POST /hub/login?next= (127.0.0.1)
    HTTPServerRequest(protocol='http', host='127.0.0.1:8000', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Connection': 'close', 'Accept-Encoding': 'gzip, deflate', 'Host': '127.0.0.1:8000', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0', 'Content-Length': '31', 'X-Forwarded-For': '127.0.0.1', 'X-Forwarded-Port': '8000', 'Content-Type': 'application/x-www-form-urlencoded', 'X-Forwarded-Proto': 'http', 'Accept-Language': 'en-US,en;q=0.5', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Referer': 'http://127.0.0.1:8000/hub/login?next='})
    Traceback (most recent call last):
      File "/opt/rh/rh-python34/root/lib64/python3.4/site-packages/tornado/web.py", line 1445, in _execute
        result = yield result
      File "/opt/rh/rh-python34/root/lib/python3.4/site-packages/jupyterhub/handlers/login.py", line 79, in post
        yield self.spawn_single_user(user)
      File "/opt/rh/rh-python34/root/lib/python3.4/site-packages/jupyterhub/handlers/base.py", line 312, in spawn_single_user
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f)
      File "/opt/rh/rh-python34/root/lib/python3.4/site-packages/jupyterhub/user.py", line 247, in spawn
        raise e
      File "/opt/rh/rh-python34/root/lib/python3.4/site-packages/jupyterhub/user.py", line 228, in spawn
        yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/opt/rh/rh-python34/root/lib/python3.4/site-packages/jupyterhub/spawner.py", line 470, in start
        start_new_session=True, # don't forward signals
      File "/opt/rh/rh-python34/root/lib64/python3.4/subprocess.py", line 858, in __init__
        restore_signals, start_new_session)
      File "/opt/rh/rh-python34/root/lib64/python3.4/subprocess.py", line 1456, in _execute_child
        raise child_exception_type(errno_num, err_msg)
    PermissionError: [Errno 13] Permission denied

[E 2016-06-06 08:33:21.509 JupyterHub log:99] {
      "Connection": "close",
      "Accept-Encoding": "gzip, deflate",
      "Host": "127.0.0.1:8000",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0",
      "Content-Length": "31",
      "X-Forwarded-For": "127.0.0.1",
      "X-Forwarded-Port": "8000",
      "Content-Type": "application/x-www-form-urlencoded",
      "X-Forwarded-Proto": "http",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
      "Referer": "http://127.0.0.1:8000/hub/login?next="
    }
[E 2016-06-06 08:33:21.509 JupyterHub log:100] 500 POST /hub/login?next= (@127.0.0.1) 376.24ms

No I am not using PYTHONPATH

[minrk]

Ah, that's a different error. That looks like your Hub doesn't have permission to become the user. Are you running the Hub as root?

[andreapago]

Yes I run the hub as a root.

[minrk]

Are you running SELinux?

[andreapago]

My understanding is yes (I was not aware of that) given the following:

sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28

Shall I disable it?

[andreapago]

Thanks for help :-)

[minrk]

It's a specific piece of the PAMAuthenticator that runs afoul of SELinux. You can disable that with:

c.PAMAuthenticator.open_sessions = False

and see if that's enough to fix the problem.

[andreapago]

Unfortunately not.
I have put that line in the jupyterhub config file, restarted the hub, but I get the same error... Any other tricks?

[minrk]

Can you check the permissions on $(which jupyterhub-singleuser)?

[andreapago]

here are the permission on the file
-rwxr-xr-x. 1 root root 9901 Jun 6 07:37 jupyterhub-singleuser

[minrk]

And what's the location of the executable?

[andreapago]

it is in
/opt/rh/rh-python34/root/bin

[minrk]

And can all users read that directory? e.g. sudo -u [someone] /opt/rh/rh-python34/root/bin/jupyterhub-singleuser -h?

[andreapago]

Yes, my other user in the OS succesfully executes the command, and looking at the files in that directory all files except the following can be read and executed by any user

jupyterhub_config.py jupyterhub_cookie_secret jupyterhub.sqlite

[minrk]

Hm, if those permissions are all correct, perhaps it is SELinux policy preventing some action. I'm not sure what it would be, I've never used a system with SELinux, but they seem to regularly need extra work to allow JupyterHub to do what it needs to do.

[andreapago]

I see....I will try to disable it and let you know any new info...thanks for your help

[andreapago]

I have disabled the SELinux
Configure SELINUX=disabled in the /etc/selinux/config
Rebooted the machine
Same error as before...

[minrk]

There are three operations that spawning the single user server does:

1. switch to the given user with setuid
2. cd to the user's home directory
3. start jupyterhub-singleuser

One of these operations is failing. Either the Hub cannot become the user (unlikely if SELinux is disabled), the user cannot read a directory, either their home or the directory where the Hub runs, or the user cannot read something required to run jupyterhub-singleuser. This could either be the Python packages it is importing (is there a Python env or anything installed in a user-specific way?) or any parent directory of the jupyterhub-singleuser script or any of the Python modules it uses being not accessible to the user.

[andreapago]

If I manually try those operation opening another console with the non root user and I run
jupytherhub-singleuser, I get the following:

File "./jupyterhub-singleuser", line 297, in
main()
File "./jupyterhub-singleuser", line 293, in main
return SingleUserNotebookApp.launch_instance()
File "/opt/rh/rh-python34/root/lib/python3.4/site-packages/jupyter_core/application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, *_kwargs)
File "/opt/rh/rh-python34/root/lib/python3.4/site-packages/traitlets/config/application.py", line 595, in launch_instance
app.initialize(argv)
File "", line 2, in initialize
File "/opt/rh/rh-python34/root/lib/python3.4/site-packages/traitlets/config/application.py", line 74, in catch_config_error
return method(app, *args, *_kwargs)
File "/opt/rh/rh-python34/root/lib/python3.4/site-packages/notebook/notebookapp.py", line 1060, in initialize
self.init_webapp()
File "./jupyterhub-singleuser", line 261, in init_webapp
s['hub_api_key'] = env.pop('JPY_API_TOKEN')
File "/opt/rh/rh-python34/root/lib64/python3.4/_collections_abc.py", line 552, in pop
value = self[key]
File "/opt/rh/rh-python34/root/lib64/python3.4/os.py", line 633, in __getitem__
raise KeyError(key) from None
KeyError: 'JPY_API_TOKEN'

Does it give any other clue?

[minrk]

Afraid not, that's the normal behavior when you try to launch the single-user script without the proper context that JupyterHub provides. Perhaps it has something to do with the user's home directory?

[andreapago]

could that be that the underlying Python 2.7 installation (installed by default) that if the user invokes a python command it is issued with that "old" version and not the 3.4 that I installed manually...I don't see that in relationship with the user privileges but was just thinking out loud...

[andreapago]

@minrk
Hey Minrk, I solved the issue: I have first changed the permission of the files that the hub was complaining about for the lack of rights, set the SELinux to disabled (also applied this for the SELinux https://github.com/jupyterhub/jupyterhub/wiki/Using-sudo-to-run-JupyterHub-without-root-privileges), and finally set the c.Spawner.cmd = to my location of the single user jupyter.
Now it works, very happy. Thanks for help!

[willingc]

@andreapago That's great that you solved the issue. Thanks for passing along your solution as well.

@minrk I'm going to label this issue as reference for possible future documentation and close the open issue.

[hustlijian]

check all the directory to$(which jupyterhub-singleuser) to see ,if has read&execute permission。
eg: /opt/rh/rh-python34/root/bin
you need to check /opt/, /opt/rh, /opt/rh/rh-python34/, /opt/rh/rh-python34/root, /opt/rh/rh-python34/root/bin

[aymaliev]

I had the same issue. Disabling SELINUX helped. Thank you.

[dturaev]

Same thing happened here on CentOS 7, setting
c.PAMAuthenticator.open_sessions = False
in jupyterhub_config.py helped. Took just 2 hours to figure it out. I guess, the "If any errors are encountered when opening/closing PAM sessions, this is automatically set to False." part didn't work for me...