Jupyterhub: Jupyterhub Docker not able to find any jinja templates

[nayan1247]

I wanted to run jupyterhub on a docker and used the below -
docker run -p 8000:8000 -d --name jupyterhub jupyterhub/jupyterhub jupyterhub
Unable to find the login/error jinja templates

Tried the below code as well still the same -

FROM jupyterhub/jupyterhub:master

RUN apt-get update --fix-missing && apt-get -y install \
wget git rsync openssh-server build-essential gcc openjdk-8-jdk-headless && \
apt install -y vim vim-tiny && \
rm -rf /var/lib/apt/lists/*

COPY requirements.txt /tmp/requirements.txt

RUN pip install -r /tmp/requirements.txt

RUN echo "127.0.0.1 $(hostname) localhost" > /etc/hosts
RUN echo "echo \"127.0.0.1 $(hostname) localhost\" > /etc/hosts" >> /etc/profile

CMD jupyterhub --ip 0.0.0.0 --port 8000 --Spawner.default_url='/lab'

Gives the following error

[theEndBeta]

I'm seeing this as well. If I setc.JupyterHub.data_files_path = '/usr/local/share/jupyterhub' in the config file, I get a login page, but little/no styling.

[theEndBeta]

I was able to get around this by specifying FROM jupyterhub/jupyterhub:1.0.0 in my dockerfile. Using jupyterhub/jupyterhub:1.0 pulls version 1.0.1dev, which makes it likely that that version is where the issue is. That being said, I would not expect to get a dev version when specifying 1.0.

[nayan1247]

Thanks @steinad.

I overcame by uninstalling jupyterhub (1.0.1dev) and again installing jupyterhub 1.0.0.
This solved the issue for me and its working normally.
Agree that we shouldn't be seeing a dev version of jupyterhub when mentioning 1.0 or master.

[Dmitry1987]

lol same issue, spent half day trying to figure out what's wrong ....

[bigbosst]

Same issue here, updated 1.0 got the dev version with missing jinja, added the jinja pathing like @steinad and got the all the missing CSS issues. Switching to 1.0.0 got the stable version and that went back to working like expected.

[consideRatio]

The Dockerfile has been changes significantly recently, can you identify what caused the issue? Is it the version bump of JupyterHub, or is it the Dockerfile that did something?

If someone has a minimalistically reproducible example, that would be very helpful as well to resolve this!

[bigbosst]

docker pull jupyterhub/jupyterhub:1.0
docker run -dt --rm -p 8000:8000 --name jupyterhubtest jupyterhub/jupyterhub:1.0

Try going to the port http://server:8000
auto redirects to :8000/hub/login with a 500 error

docker logs jupyterhubtest

...

[E 2019-12-20 19:02:25.039 JupyterHub log:174] 500 GET /hub/login (@::ffff:10.40.229.10) 4.27ms

[E 2019-12-20 19:02:26.574 JupyterHub web:1788] Uncaught exception GET /hub/login (::ffff:10.40.229.10)
    HTTPServerRequest(protocol='http', host='server:8000', method='GET', uri='/hub/login', version='HTTP/1.1', remote_ip='::ffff:10.40.229.10')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/login.py", line 135, in get
        self.finish(self._render(username=username))
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/login.py", line 102, in _render
        {'next': self.get_argument('next', '')},
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1119, in render_template
        template = self.get_template(name)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1113, in get_template
        return self.settings['jinja2_env'].get_template(name)
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 830, in get_template
        return self._load_template(name, self.make_globals(globals))
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 804, in _load_template
        template = self.loader.load(self, name, globals)
      File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 408, in load
        raise TemplateNotFound(name)
    jinja2.exceptions.TemplateNotFound: login.html

[E 2019-12-20 19:02:26.575 JupyterHub web:1216] Uncaught exception in write_error
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1214, in send_error
        self.write_error(status_code, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1203, in write_error
        html = self.render_template('error.html', **ns)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1119, in render_template
        template = self.get_template(name)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1113, in get_template
        return self.settings['jinja2_env'].get_template(name)
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 830, in get_template
        return self._load_template(name, self.make_globals(globals))
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 804, in _load_template
        template = self.loader.load(self, name, globals)
      File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 408, in load
        raise TemplateNotFound(name)
    jinja2.exceptions.TemplateNotFound: error.html
...

[wangsijie]

docker pull jupyterhub/jupyterhub:1.0
docker run -dt --rm -p 8000:8000 --name jupyterhubtest jupyterhub/jupyterhub:1.0

Try going to the port http://server:8000
auto redirects to :8000/hub/login with a 500 error

docker logs jupyterhubtest

...

[E 2019-12-20 19:02:25.039 JupyterHub log:174] 500 GET /hub/login (@::ffff:10.40.229.10) 4.27ms

[E 2019-12-20 19:02:26.574 JupyterHub web:1788] Uncaught exception GET /hub/login (::ffff:10.40.229.10)
    HTTPServerRequest(protocol='http', host='server:8000', method='GET', uri='/hub/login', version='HTTP/1.1', remote_ip='::ffff:10.40.229.10')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1699, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/login.py", line 135, in get
        self.finish(self._render(username=username))
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/login.py", line 102, in _render
        {'next': self.get_argument('next', '')},
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1119, in render_template
        template = self.get_template(name)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1113, in get_template
        return self.settings['jinja2_env'].get_template(name)
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 830, in get_template
        return self._load_template(name, self.make_globals(globals))
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 804, in _load_template
        template = self.loader.load(self, name, globals)
      File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 408, in load
        raise TemplateNotFound(name)
    jinja2.exceptions.TemplateNotFound: login.html

[E 2019-12-20 19:02:26.575 JupyterHub web:1216] Uncaught exception in write_error
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1214, in send_error
        self.write_error(status_code, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1203, in write_error
        html = self.render_template('error.html', **ns)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1119, in render_template
        template = self.get_template(name)
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 1113, in get_template
        return self.settings['jinja2_env'].get_template(name)
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 830, in get_template
        return self._load_template(name, self.make_globals(globals))
      File "/usr/local/lib/python3.6/dist-packages/jinja2/environment.py", line 804, in _load_template
        template = self.loader.load(self, name, globals)
      File "/usr/local/lib/python3.6/dist-packages/jinja2/loaders.py", line 408, in load
        raise TemplateNotFound(name)
    jinja2.exceptions.TemplateNotFound: error.html
...

jupyterhub/jupyterhub:1.0.0

Don't miss the last 0

[bigbosst]

@wangsijie

Don't miss the last 0

According to the comments here and my testing, the issue is on 1.0, 1.0.1dev, the issue is not on 1.0.0. I am not sure what the point of this comment was.

[rkdarst]

Problem seems to be in jupyterhub/_data.py. Steps to verify:

# from within the miage
>>> import jupyterhub._data
>>> jupyterhub._data.DATA_FILES_PATH

The offending section is:

            if all(exists(join(static, f)) for f in ['components', 'css']):
                return share_jupyterhub

... which tries to find the right JupyterHub data path by looking for a share/jupyterhub/ dir that contains the subdirs components and css. But these are now missing from the latest Docker image.

I'm now trying to figure out the root cause but it's a bit beyond my expertise.

[rkdarst]

Actually the missing files are in /usr/local/share/jupyterhub/static/.

In current dockerfile:

# ls static/
favicon.ico  images  js  less

In same dockerfile after python3 -m pip install --force --no-deps jupyterhub==1.1.0.b1:

# ls static/
components  css  favicon.ico  images  js  less

my workaround was adding the pip install line above into my own Dockerfile. Since regular pip install works, my guess is it's somehow related to recent changes in the Dockerfile, e.g. #2840 (or maybe #2816)

• I note in #2840, here it copies share/ in the builder container: https://github.com/jupyterhub/jupyterhub/pull/2840/files#diff-3254677a7917c6c01f55212f86c57fbfR46
• but here it only copies things from wheelhouse to the final image: https://github.com/jupyterhub/jupyterhub/pull/2840/files#diff-3254677a7917c6c01f55212f86c57fbfR86 . The wheel does not include share/.

Anyone around to provide a better analysis?

[rkdarst]

I think #2864 should fix it, I built the image and the missing stuff is there. Not sure if this is the right answer, and I haven't actually tested it...

[consideRatio]

Reproducing / debugging current master

IMG=jupyterhub:dev
docker pull jupyterhub/jupyterhub:master

docker stop jh-dev || true
docker run -dt --rm -p 8000:8000 --name jh $IMG
sleep 3
curl localhost:8000/hub/login
docker logs jh

Debugging locally

IMG=jupyterhub:dev
docker build -t $IMG .

docker stop jh-dev || true
docker run -dt --rm -p 8000:8000 --name jh $IMG
sleep 3
curl localhost:8000/hub/login
docker logs jh

Conclusion

Currently, everything works if I built it locally... I'm on Ubuntu 19.10 and Docker CE 19.03.3, but the latest docker image in the master tag built by docker-hub does not work. Hmmm... Why?

I found this recent issue about different outcomes of building locally and remotely etc. I would ignore it if it wasn't for you mentioning the C compilation need. Hmmm.. Not yet at all confident about anything except that the bug is that there is a difference of local function and remote function when we build the image.

[consideRatio]

Here are DockerHub information from our build btw:

KernelVersion: 4.4.0-1060-aws
Components: [{u'Version': u'18.03.1-ee-3', u'Name': u'Engine', u'Details': {u'KernelVersion': u'4.4.0-1060-aws', u'Os': u'linux', u'BuildTime': u'2018-08-30T18:42:30.000000000+00:00', u'ApiVersion': u'1.37', u'MinAPIVersion': u'1.12', u'GitCommit': u'b9a5c95', u'Arch': u'amd64', u'Experimental': u'false', u'GoVersion': u'go1.10.2'}}]
Arch: amd64
BuildTime: 2018-08-30T18:42:30.000000000+00:00
ApiVersion: 1.37
Platform: {u'Name': u''}
Version: 18.03.1-ee-3
MinAPIVersion: 1.12
GitCommit: b9a5c95
Os: linux
GoVersion: go1.10.2

[consideRatio]

I could no longer reproduce it locally, I think I must have used an old master tag or similar!

[matteoipri]

I am still encountering the issue today.
I cloned the repository and built the image locally and I got error 500.

$ mkdir jupyterhub
$ cd jupyterhub/
$ git clone https://github.com/jupyterhub/jupyterhub.git
$ cd jupyterhub/
$ ls -l
$ sudo docker image build -t jhmi .
$ sudo docker run -dit -p 8003:8000 --name jhmi jhmi
$ sudo docker logs jhmi
Running JupyterHub version 1.2.0dev
...
    jinja2.exceptions.TemplateNotFound: login.html
    jinja2.exceptions.TemplateNotFound: error.html
...
$ sudo docker exec jhmi ls -Al /usr/local/share/jupyterhub/static/
total 20
-rw-r--r-- 1 root root 4286 Jan 17 18:46 favicon.ico
drwxr-xr-x 2 root root 4096 Jan 17 18:46 images
drwxr-xr-x 2 root root 4096 Jan 17 18:46 js
drwxr-xr-x 2 root root 4096 Jan 17 18:46 less
$ sudo docker exec jhmi ls -Al /src/jupyterhub/share/jupyterhub/static
total 20
-rw-r--r-- 1 root root 4286 Jan 17 18:43 favicon.ico
drwxr-xr-x 2 root root 4096 Jan 17 18:43 images
drwxr-xr-x 2 root root 4096 Jan 17 18:43 js
drwxr-xr-x 2 root root 4096 Jan 17 18:43 less

Using the docker image tagged as 1.1.0 from hub.docker.com, I end with a 500 error page with no template.

$ sudo docker image pull jupyterhub/jupyterhub:1.1.0
$ sudo docker run -dit -p 8001:8000 --name jh110 jupyterhub/jupyterhub:1.1.0
$ sudo docker logs jh110
Running JupyterHub version 1.1.0
...
    jinja2.exceptions.TemplateNotFound: login.html
    jinja2.exceptions.TemplateNotFound: error.html
...
$ sudo docker exec jh110 ls -Al /usr/local/share/jupyterhub/static/
total 20
-rw-r--r-- 1 root root 4286 Jan 17 13:03 favicon.ico
drwxr-xr-x 2 root root 4096 Jan 17 13:04 images
drwxr-xr-x 2 root root 4096 Jan 17 13:04 js
drwxr-xr-x 2 root root 4096 Jan 17 13:04 less
$ sudo docker exec jh110 ls -Al /src/jupyterhub/share/jupyterhub/static
total 20
-rw-r--r-- 1 root root 4286 Jan 17 12:57 favicon.ico
drwxr-xr-x 2 root root 4096 Jan 17 12:57 images
drwxr-xr-x 2 root root 4096 Jan 17 12:57 js
drwxr-xr-x 2 root root 4096 Jan 17 12:57 less

Instead with the docker image tagged as 1.0.0

$ sudo docker image pull jupyterhub/jupyterhub:1.0.0
$ sudo docker run -dit -p 8002:8000 --name jh100 jupyterhub/jupyterhub:1.0.0
$ sudo docker exec jh100 ls -Al /usr/local/share/jupyterhub/static/
ls: cannot access '/usr/local/share/jupyterhub/static/': No such file or directory
$ sudo docker exec jh100 ls -Al /src/jupyterhub/share/jupyterhub/static
ls: cannot access '/src/jupyterhub/share/jupyterhub/static': No such file or directory

leads me to a login page as expected.

Any suggestion?

[consideRatio]

Thanks for showing how to reproduce this! Opening it back again

[shingo78]

I built the image locally, found the following errors in the build log.

  Building wheel for jupyterhub (PEP 517): started
  Running command /usr/bin/python3 /usr/local/lib/python3.6/dist-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/
tmpr88qla89
  running bdist_wheel
  running build
  running build_py
  running js

  > [email protected] postinstall /tmp/pip-req-build-mnx4w6t5
  > python ./bower-lite

  python: can't open file './bower-lite': [Errno 2] No such file or directory

  npm WARN [email protected] license should be a valid SPDX license expression
  npm ERR! Linux 3.10.0-1062.1.1.el7.x86_64
  npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "install" "--progress=false" "--unsafe-perm"
  npm ERR! node v8.10.0
  npm ERR! npm  v3.5.2
  npm ERR! code ELIFECYCLE
  npm ERR! [email protected] postinstall: `python ./bower-lite`
  npm ERR! Exit status 2
  npm ERR!
  npm ERR! Failed at the [email protected] postinstall script 'python ./bower-lite'.
  npm ERR! Make sure you have the latest version of node.js and npm installed.
  npm ERR! If you do, this is most likely a problem with the jupyterhub-deps package,
  npm ERR! not with npm itself.
  npm ERR! Tell the author that this fails on your system:
  npm ERR!     python ./bower-lite
  npm ERR! You can get information on how to open an issue for this project with:
  npm ERR!     npm bugs jupyterhub-deps
  npm ERR! Or if that isn't available, you can get their info via:
  npm ERR!     npm owner ls jupyterhub-deps
  npm ERR! There is likely additional logging output above.

  npm ERR! Please include the following file with any support request:
  npm ERR!     /tmp/pip-req-build-mnx4w6t5/npm-debug.log

It seems that bower-lite is not found in /src/jupyterhub while building wheel.
The package.json depends bower-lite.

If I add bower-lite to the COPY src arguments at https://github.com/jupyterhub/jupyterhub/blob/master/Dockerfile#L45-L50 and rebuild, I get a login page as expected.

[matteoipri]

I confirm that adding bower-lite to that COPY statement leads me to jupyterhub docker image that seems to work.
I say only seems because I performed limited testing.

[consideRatio]

Here is the bower-lite script, copying content from node-modules to other folders it seems.

https://github.com/jupyterhub/jupyterhub/blob/master/bower-lite

[consideRatio]

The error does reproduce.

cd /tmp
git clone https://github.com/jupyterhub/jupyterhub.git
cd jupyterhub
IMG=jh
docker build -t $IMG .
docker stop jh || true
docker run -dt --rm -p 8000:8000 --name jh $IMG
sleep 3
curl localhost:8000/hub/login
docker logs jh

And... Copying the bower-lite script to the build stage does solve the issue!

[analytically]

This is also present in 1.1.0

[manics]

@analytically It should be fixed by https://github.com/jupyterhub/jupyterhub/pull/2905 but that's not released yet

[analytically]

I'd say this merits a note in the changelog about 1.1.0 being broken for container-based setups.

[Dmitry1987]

I just tried version "1.1.0" and got this issue again. had to revert to "1.0.0" :( the images in dockerhub are broken (or their tags are wrong)

[diurnalist]

Update: #2905 does in fact fix the issue but is still not released as a patch to 1.1

---

The root cause of this is because client dependencies have not been reliably built in the Dockerfile. #3082 has been submitted to address.

The MANIFEST.in is doing its job of ensuring the static files are built as part of the wheel, from what I can tell. The inclusion of an npm install step is the missing bit. Not sure how this was working before, but it's possible that because the static/components directory is untracked, it's been sitting around in CI caches for a long time, masking the problem.

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/custom-spawner-with-jupyterhub-on-k8s/5772/2