Nbconvert: TemplateNotFound error when extending built-in template

[milliams]

I have the following code:

import nbformat
from nbconvert import HTMLExporter

html_exporter = HTMLExporter(template_file="foo.tpl")
with open("foo.ipynb") as infile:
    nb = nbformat.read(infile, as_version=4)
    html = html_exporter.from_notebook_node(nb)
    print(html)

and a template (which does nothing but still fails if it does do something) in the cwd, foo.tpl:

{% extends 'lab/index.html.j2' %}

I get the error (with the venv path replaced with <venv> for brevity):

Traceback (most recent call last):
  File "convert_test/__init__.py", line 7, in <module>
    html = html_exporter.from_notebook_node(nb)
  File "<venv>/lib64/python3.8/site-packages/nbconvert/exporters/html.py", line 122, in from_notebook_node
    return super().from_notebook_node(nb, resources, **kw)
  File "<venv>/lib64/python3.8/site-packages/nbconvert/exporters/templateexporter.py", line 382, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "<venv>/lib64/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "<venv>/lib64/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "<venv>/lib64/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/matt/temp/convert_test/foo.tpl", line 1, in top-level template code
    {% extends 'lab/index.html.j2'%}
  File "<venv>/share/jupyter/nbconvert/templates/lab/index.html.j2", line 1, in top-level template code
    {%- extends 'base.html.j2' -%}
jinja2.exceptions.TemplateNotFound: base.html.j2

It seems to be finding my foo.tpl file correctly, seeing the extends 'lab/index.html.j2' line and then correctly finding <venv>/share/jupyter/nbconvert/templates/lab/index.html.j2. This file, inside the install directory says it extends 'base.html.j2' but nbconvert fails to find that file.

If I do ls <venv>/share/jupyter/nbconvert/templates/lab/ then I see:

static  base.html.j2  conf.json  index.html.j2

so base.html.j2 is sitting there right next to index.html.j2.

This process works fine with nbconvert 5. I realise that templates have changed and I read https://nbconvert.readthedocs.io/en/latest/customizing.html but it doesn't seem to tell me how to create a custom template, only how the built-in templates are organised.

[desilinguist]

I am having the exact same issue with the following API code in my tool.

report_config = Config({'ExecutePreprocessor': {'enabled': True,
                                                'timeout': 3600},
                        'HTMLExporter': {'template_paths': [template_path],
                                         'template_file': 'report.tpl'}})

exportHtml = HTMLExporter(config=report_config)
output, _ = exportHtml.from_filename(notebook_file)

template_path is where my custom template lives and looks like this:

{%- extends "full.tpl" -%}

{% block input %}
{%- endblock input %}
{% block in_prompt %}
{%- endblock in_prompt %}

It used to work just perfectly with the previous versions of nbconvert but since 6.0.2 came out, I am seeing the following error:

Executing notebook with kernel: python3
Traceback (most recent call last):
  File "/Users/nmadnani/anaconda/envs/rsmdev/bin/rsmtool", line 33, in <module>
    sys.exit(load_entry_point('rsmtool', 'console_scripts', 'rsmtool')())
  File "/Users/nmadnani/work/rsmtool/rsmtool/rsmtool.py", line 374, in main
    overwrite_output=args.force_write)
  File "/Users/nmadnani/work/rsmtool/rsmtool/rsmtool.py", line 321, in run_experiment
    figdir)
  File "/Users/nmadnani/work/rsmtool/rsmtool/reporter.py", line 688, in create_report
    join(reportdir, '{}.html'.format(report_name)))
  File "/Users/nmadnani/work/rsmtool/rsmtool/reporter.py", line 327, in convert_ipynb_to_html
    output, _ = exportHtml.from_filename(notebook_file)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 182, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 200, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/nbconvert/exporters/html.py", line 122, in from_notebook_node
    return super().from_notebook_node(nb, resources, **kw)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/nbconvert/exporters/templateexporter.py", line 382, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/Users/nmadnani/work/rsmtool/rsmtool/notebooks/templates/report.tpl", line 1, in top-level template code
    {%- extends "full.tpl" -%}
jinja2.exceptions.TemplateNotFound: full.tpl

I also tried using {%- extends 'compatibility/full.tpl' -%} but that has the same issue.

[MSeal]

I see the issue. It's a bug in how the templates are referring to local files when being imported from a different template directory. @maartenbreddels @SylvainCorlay could use your eyes on this to brainstorm how we might remedy the transitive pathing issue here. For compatibility templates in https://github.com/jupyter/nbconvert/pull/1387 I added the extra template paths explicitly to handle this but seeing these reports of issues I think that might be the wrong approach to that aspect of the problem.

[Sieboldianus]

In the meantime, until this is resolved, is there a possible workaround? I have the same issue, using a custom template that extends full.tpl.

  File "/opt/conda/envs/worker_env/share/jupyter/nbconvert/templates/compatibility/full.tpl", line 2, in top-level template code
    {%- extends 'lab/index.html.j2' -%}
  File "/opt/conda/envs/worker_env/share/jupyter/nbconvert/templates/lab/index.html.j2", line 1, in top-level template code
    {%- extends 'base.html.j2' -%}
jinja2.exceptions.TemplateNotFound: base.html.j2

I tried to copy all files from https://github.com/jupyter/nbconvert/tree/master/share/jupyter/nbconvert/templates/base to the top folder. This fixes all *.tpl issues, but I don't know how to proceed with this:

File "/opt/conda/envs/worker_env/lib/python3.8/site-packages/nbconvert/exporters/html.py", line 127, in resources_include_css
    code = """<style type="text/css">\n%s</style>""" % (env.loader.get_source(env, name)[0])
  File "/opt/conda/envs/worker_env/lib/python3.8/site-packages/jinja2/loaders.py", line 420, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: static/index.css

Found the css files: share/jupyter/nbconvert/templates/lab/static

If these are also present locally (from where nbconvert is run), then everything works. However, all code-cells have no syntax highlighted in the output HTML.

I finally used the workaround conda install -c conda-forge nbconvert=5.6.1.

[MSeal]

Until I get the PR in to fix transitive dependency pathing, you can add share/jupyter/nbconvert/templates/lab to the template_paths attribute of the TemplateExporter instance, but that's not exposed on the command line. I'll put effort into getting a resolved path on this for this week.

[MSeal]

This overall issue should now be resolved for backwards compatibility in 6.0.7 if you additionally set the template_name to be the base template you are using (lab I think in the cases above). Ideally though if you are using a 6.0 template you can follow https://nbconvert.readthedocs.io/en/latest/customizing.html#conf-json and set the base template name there, which will fix any template inheritance for importing those base template files as local files in jinja.