Jupyterhub: jinja2 in Python 3 import error

[statueofmike]

I installed Jupyterhub on Ubuntu 14.04, then run it as sudo. When logging in, the following error comes up:

[I 2016-02-01 13:28:07.586 JupyterHub spawner:409] Spawning jupyterhub-singleuser --user=michael --port=51590 --cookie-name=jupyter-hub-token-michael --base-url=/user/michael --hub-prefix=/hub/ --hub-api-url=http://localhost:8081/hub/api --ip=localhost
Traceback (most recent call last):

File "/usr/local/bin/jupyterhub-singleuser", line 15, in

from jinja2 import ChoiceLoader, FunctionLoader

ImportError: cannot import name 'ChoiceLoader'

I tried reinstalling Jupyterhub and jinja2.

When I run python3, I can do import jinja2; dir(jinja2) and it looks like something is wrong there:

import jinja2; dir(jinja2)
['doc', 'loader', 'name', 'package', 'path', 'spec']

Here is my system information:

$ python3 -c "import IPython; print(IPython.sys_info())"
{'codename': 'An Afternoon Hack',
'commit_hash': '9c4f59d',
'commit_source': 'installation',
'default_encoding': 'UTF-8',
'ipython_path': '/usr/lib/python3/dist-packages/IPython',
'ipython_version': '1.2.1',
'os_name': 'posix',
'platform': 'Linux-3.16.0-30-generic-x86_64-with-Ubuntu-14.04-trusty',
'sys_executable': '/usr/bin/python3',
'sys_platform': 'linux',
'sys_version': '3.4.3 (default, Oct 14 2015, 20:28:29) \n[GCC 4.8.4]'}

[minrk]

What's jinja2.__version__ and/or jinja2.__path__?

[statueofmike]

>>> jinja2.__path__

_NamespacePath(['/usr/local/lib/python3.4/dist-packages/jinja2'])

there's no __version__

[minrk]

Then it looks like you have a partial install of jinja2 (i.e. missing everything). Try:

pip install --upgrade jinja2

If that doesn't work, you might try uninstalling it first.

[statueofmike]

I'm really at a loss here. Is it a requirement of python 2.7, or 3.4? Those are the two I have installed locally, but I believe Jupyterhub uses 3.4.

sudo pip uninstall jinja2

Not uninstalling Jinja2 at /usr/lib/python2.7/dist-packages, owned by OS

sudo pip3 uninstall jinja2

...
Successfully uninstalled Jinja2

Want to verify that it was removed:
sudo ls /usr/local/lib/python3.4/dist-packages/jinja*

ls: cannot access /usr/local/lib/python3.4/dist-packages/jinja*: No such file or directory

Looks like it's gone, so reinstall:
sudo pip3 install --upgrade jinja2

Downloading/unpacking jinja2
Downloading Jinja2-2.8-py2.py3-none-any.whl (263kB): 263kB downloaded
Requirement already up-to-date: MarkupSafe in /usr/local/lib/python3.4/dist-packages (from jinja2)
Installing collected packages: jinja2
Successfully installed jinja2
Cleaning up...

Now check if it was completely installed, looks like NO!:
python3

Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.

import jinja2
dir(jinja2)
['doc', 'loader', 'name', 'package', 'path', 'spec']

[minrk]

And what's jinja2.__path__ after that, and what files do you see when you ls the directory that you find?

[statueofmike]

>>> jinja2.__path__

_NamespacePath(['/usr/local/lib/python3.4/dist-packages/jinja2'])

$ sudo ls /usr/local/lib/python3.4/dist-packages/jinja2

bccache.py constants.py environment.py filters.py loaders.py optimizer.py runtime.py tests.py
_compat.py debug.py exceptions.py __init__.py meta.py parser.py sandbox.py utils.py
compiler.py defaults.py ext.py lexer.py nodes.py pycache _stringdefs.py visitor.py

[minrk]

Something's really fishy. What do you get from from jinja2 import loaders?

When you uninstall jinja2, does that directory still exist?

[statueofmike]

>>> from jinja2 import loaders

Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name 'loaders'

When I'd uninstalled jinja2, the directory was gone:
sudo ls /usr/local/lib/python3.4/dist-packages/jinja*

ls: cannot access /usr/local/lib/python3.4/dist-packages/jinja*: No such file or directory

But it is back now after the reinstall:
sudo ls /usr/local/lib/python3.4/dist-packages/jinja*

bccache.py constants.py environment.py filters.py loaders.py optimizer.py runtime.py tests.py
_compat.py debug.py exceptions.py __init__.py meta.py parser.py sandbox.py utils.py
compiler.py defaults.py ext.py lexer.py nodes.py pycache _stringdefs.py visitor.py

[statueofmike]

It looks like there is something wrong with my pip3. I worked around the above errors by using apt-get to install the packages jinja2,tornado under Python 3.4. Now jinja2 and tornado are installed to /usr/lib/python3/dist-packages/ and seem to be installed correctly. Pip3 installed these incompletely to /usr/local/lib/python3.4/dist-packages, but apt-get seems to have installed them correctly.

Unfortunately, now there is an error under notebook:

[I 2016-02-02 17:32:58.097 JupyterHub spawner:409] Spawning jupyterhub-singleuser --user=michael --port=42835 --cookie-name=jupyter-hub-token-michael --base-url=/user/michael --hub-prefix=/hub/ --hub-api-url=http://localhost:8081/hub/api --ip=localhost
Traceback (most recent call last):
File "/usr/local/bin/jupyterhub-singleuser", line 43, in
from notebook.notebookapp import NotebookApp, aliases as notebook_aliases
ImportError: No module named 'notebook.notebookapp'

I tried also installing ipython-notebook for python3 with apt-get, but that wasn't found in the class path for jupyterhub. It appears that my pip3-installed packages are just not installing completely.

[minrk]

Try this:

pip3 install --upgrade setuptools pip
pip3 uninstall jinja2 tornado
pip3 install --upgrade notebook jupyterhub

[statueofmike]

pip3 install --upgrade notebook jupyterhub ended with this:

Installing collected packages: jinja2, tornado, notebook, jupyterhub, ipython
  Found existing installation: ipython 4.0.3
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/usr/local/lib/python3.4/dist-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python3.4/dist-packages/pip/req/req_set.py", line 725, in install
    requirement.uninstall(auto_confirm=True)
  File "/usr/local/lib/python3.4/dist-packages/pip/req/req_install.py", line 709, in uninstall
    for path in pip.wheel.uninstallation_paths(dist):
  File "/usr/local/lib/python3.4/dist-packages/pip/wheel.py", line 532, in unique
    for item in fn(*args, **kw):
  File "/usr/local/lib/python3.4/dist-packages/pip/wheel.py", line 551, in uninstallation_paths
    r = csv.reader(FakeFile(dist.get_metadata_lines('RECORD')))
  File "/usr/local/lib/python3.4/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1624, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/local/lib/python3.4/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1621, in get_metadata
    return self._get(self._fn(self.egg_info, name)).decode("utf-8")
  File "/usr/local/lib/python3.4/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 1727, in _get
    with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.4/dist-packages/ipython-4.0.3.dist-info/RECORD'

[minrk]

Arg, that seems like setuptools and/or pip has really messed itself up. When it gets to that degree, I'm tempted to trash the entirety of /usr/local/lib/python3.4/dist-packages and start over, but be careful when doing something that destructive.

[willingc]

@statueofmike If you installed pip or pip3 using apt-get it is possible that you may have an old version of pip or pip3. Before trashing the entirety of /usr/local/lib/python3.4/dist-packages, you may wish to just remove pip and pip3 (and possibly setuptools). http://stackoverflow.com/questions/28917534/pip-broken-on-ubuntu-14-4-after-package-upgrade

[yuvipanda]

You could also use a virtualenv to see if the problem goes away there, instead of trashing your /usr/local/lib/

[statueofmike]

SOLVED

I tried reinstalling pip and pip3 altogether first. This didn't seem to help.

Then I trashed my entire /usr/local/lib/python3.4/dist-packages (backed up).

Here I realized the problem: for some reason, the file system permissions were restricting user Python sessions from loading these packages properly.

$ python3
Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jinja2
>>> dir(jinja2)
['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

sudo chmod -R 775 /usr/local/lib/python3.4/dist-packages/
$ python3
>>> import jinja2
>>> dir(jinja2)
['BaseLoader', 'BytecodeCache', 'ChoiceLoader', 'DebugUndefined', 'DictLoader', 'Environment', 'FileSystemBytecodeCache', 'FileSystemLoader', 'FunctionLoader', 'Markup', 'MemcachedBytecodeCache', 'ModuleLoader', 'PackageLoader', 'PrefixLoader', 'StrictUndefined', 'Template', 'TemplateAssertionError', 'TemplateError', 'TemplateNotFound', 'TemplateSyntaxError', 'TemplatesNotFound', 'Undefined', 'UndefinedError', '__all__', '__builtins__', '__cached__', '__doc__', '__docformat__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_compat', '_stringdefs', 'bccache', 'clear_caches', 'compiler', 'contextfilter', 'contextfunction', 'defaults', 'environment', 'environmentfilter', 'environmentfunction', 'escape', 'evalcontextfilter', 'evalcontextfunction', 'exceptions', 'filters', 'is_undefined', 'lexer', 'loaders', 'make_logging_undefined', 'nodes', 'optimizer', 'parser', 'runtime', 'tests', 'utils', 'visitor']

Now I seem to have created a NEW problem. The Jupyterhub launcher also had too restrictive permissions. I'd installed with sudo -H pip3 install jupyterhub; sudo -H pip3 install notebook

$ ls -lah /usr/local/bin
-rw-r--r--  1 root root   59 Feb 15 11:47 jupyterhub
-rw-r--r--  1 root root 7.9K Feb 15 11:47 jupyterhub-singleuser

Open Question

Why did I have this file permission issue? Should I have installed differently?

[minrk]

That's quite strange. Do you see similar permission issues on any other packages?

[statueofmike]

I'm seeing other strange errors now.

$ pip3 install matplotlib
Collecting matplotlib
  Downloading matplotlib-1.5.1.tar.gz (54.0MB)
    100% |████████████████████████████████| 54.0MB 8.9kB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/tmp/pip-build-y9xlz1k7/matplotlib/distribute_setup.py", line 152, in use_setuptools
        pkg_resources.require("setuptools>=0.7b")
    AttributeError: 'module' object has no attribute 'require'

Jupyterhub throws the following error:

``````
[I 2016-02-15 12:53:31.558 JupyterHub spawner:409] Spawning jupyterhub-singleuser --user=michael --port=59855 --cookie-name=jupyter-hub-token-michael --base-url=/user/michael --hub-prefix=/hub/ --hub-api-url=http://localhost:8081/hub/api --ip=localhost
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/path.py", line 123, in
__version__ = pkg_resources.require('path.py')[0].version
AttributeError: 'module' object has no attribute 'require'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/jupyterhub-singleuser", line 21, in
from IPython.utils.traitlets import (
File "/usr/local/lib/python3.4/dist-packages/IPython/__init__.py", line 49, in
from .terminal.embed import embed
File "/usr/local/lib/python3.4/dist-packages/IPython/terminal/embed.py", line 16, in
from IPython.core.interactiveshell import DummyMod
File "/usr/local/lib/python3.4/dist-packages/IPython/core/interactiveshell.py", line 31, in
from pickleshare import PickleShareDB
File "/usr/local/lib/python3.4/dist-packages/pickleshare.py", line 41, in
from path import Path
File "/usr/local/lib/python3.4/dist-packages/path.py", line 126, in
except pkg_resources.DistributionNotFound:
AttributeError: 'module' object has no attribute 'DistributionNotFound'```
``````

[minrk]

That means that the pkg_resources module, the underpinning of setuptools itself, is completely broken or missing. When you do

import pkg_resources
print(pkg_resources.__file__)

what do you see?

[statueofmike]

I see this:

>>> import pkg_resources
>>> print(pkg_resources.__file__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__file__'
>>> dir(pkg_resources)
['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> print(pkg_resources.__path__)
_NamespacePath(['/usr/local/lib/python3.4/dist-packages/pkg_resources'])

[minrk]

And ls -la /usr/local/lib/python3.4/dist-packages/pkg_resources?

[statueofmike]

$ sudo ls -la /usr/local/lib/python3.4/dist-packages/pkg_resources
total 114
drwxr-Sr--  5 root staff   1024 Feb 15 12:46 .
drwxrwsr-x 64 root staff   3072 Feb 15 12:46 ..
drwxr-Sr--  3 root staff   1024 Feb 15 12:46 extern
-rw-r--r--  1 root staff 108616 Feb 15 12:46 __init__.py
drwxr-Sr--  2 root staff   1024 Feb 15 12:46 __pycache__
drwxr-Sr--  4 root staff   1024 Feb 15 12:46 _vendor

[minrk]

All users should have x permissions on directories in there, otherwise they can't read the files. The fact that you had to sudo ls is the same problem. In order for these packages to be imported, you have to be able to cd/ls/read all files in there.

If this is the only issue, you can fix it by ensuring that all directories are world-readable:

sudo find /usr/local/lib/python3.4/dist-packages -type d -exec chmod a+x "{}" \;

That's: find all directories in dist-packages and give everyone execute (aka listing) permissions on those directories

[statueofmike]

I ran the command to enable world-readable directories. pip3 has the same problem trying to install e.g. matplotlib.

Is there a safe way to just remove all python altogether and reinstall it? Maybe I used easy_install early on and messed things up that way?

[minrk]

You can remove all of the pip-installed packages easily enough, by removing /usr/local/lib/python3.4/, and start again with:

sudo python3 -m pip install --upgrade setuptools pip

That will leave some broken entrypoints in /usr/local/bin, but that shouldn't be too problematic as they will be replaced by subsequent installations.

It would be unwise to remove the system-installed packages without going through apt.

[statueofmike]

is it supposed to be sudo python3 -m pip ... or ... -m pip3 ... ?

[willingc]

@statueofmike It's as written by @minrk. :sunny:

[willingc]

Closing this issue as it's been inactive for several months. Please feel free to open a new issue at the jupyter/help repo or leave a comment requesting to reopen this issue. Thanks! :sunflower: