Nbconvert: Feature: Update documentation to show how to configure custom template paths

[txoof]

Problem: I am trying to build a system for maintaining nbconvert templates over several platforms (Mac OS, Ubuntu, Debian) that use various paths for storing templates
Raspbian:

['/home/pi/.local/share/jupyter/nbconvert/templates', '/usr/local/share/jupyter/nbconvert/templates', '/usr/share/jupyter/nbconvert/templates']

Mac OS with Pyenv:

['/Users/ac/Library/Jupyter/nbconvert/templates', '/Users/ac/.pyenv/versions/3.8.5/Python.framework/Versions/3.8/share/jupyter/nbconvert/templates', '/usr/local/share/jupyter/nbconvert/templates', '/usr/share/jupyter/nbconvert/templates']

I understand that I can drop my templates into one of the paths above, but that requires a different solution for each platform making updates more challenging.

Proposed Solution
Update the documentation to clearly show how to specify an alternative or additional path to templates.

The current documentation does not indicate how to do this.

There is a long discussion in Issue #1028 around this, but I could not determine how it was resolved.

TemplateExporter.template_paths : List
   Default: ['.']

   No description

PythonExporter.template_paths : List
   Default: ['.']

   No description



md5-fccf6d878fee82d9b746e6843871a9a4



c.PythonExporter.template_path: ['.', '/path/to/.jupyter/templates/my_python_template']



md5-ad5f9378a6f4dbae2b4ed3885abac92f



jinja2.exceptions.TemplateNotFound: null.j2



md5-030320bbb7466925d3f2dabd74dc02d1



{%- extends 'null.j2' -%}

## set to python3
{%- block header -%}
#!/usr/bin/env python3
# coding: utf-8
{% endblock header %}

## remove cell counts entirely
{% block in_prompt %}
{% if resources.global_content_filter.include_input_prompt -%}
{% endif %}
{% endblock in_prompt %}

## remove markdown cells entirely
{% block markdowncell %}
{% endblock markdowncell %}

{% block input %}
{{ cell.source | ipython2python }}
{% endblock input %}


## remove magic statement completely
{% block codecell %}
{{'' if "get_ipython" in super() else super() }}
{% endblock codecell%}

[txoof]

Is #1394 related to this?

[MSeal]

The template paths didn't change substantially since 5.x, though the way templates are defined was a major change with 6.0. We updated https://nbconvert.readthedocs.io/en/latest/customizing.html to reflect customization changes that were introduced and where nbconvert looks for paths to template directories. If you are using template_path then you are using the old 5.6.1 code, as we changed the name to template_paths in 6.0 (noted in the change logs). However we didn't capture how to manipulate that field in the customization docs though -- I'll look at adding that with this upcoming patch release.

And yes https://github.com/jupyter/nbconvert/pull/1429 introduces a fix for the underlying issue you hit with the null.j2 import error when you're operating with 5.x template patterns, though for your import I don't think you need that fix regardless. Instead you need to reference the new template path to the underlying template file you were extending. In this case if you change {%- extends 'null.j2' -%} to {%- extends 'base/null.j2' -%} or if you have a 6.x template (preferred and more supported path) add a conf.json with:

{
    "base_template": "base",
    "mimetypes": {
        "text/x-python": true
    }
}

in your template directory to make sure the base template files are available at runtime.

The fix for backwards compatibility to 5.x template inheritance patterns should be released in the next couple days once we get docs and two more minor fixes added.

[MSeal]

In terms of default locations that should be included I had thought that '/path/to/.jupyter/templates should have been omni-present. @maartenbreddels do you know if we purposefully removed that or if it was unintentionally removed in one of the later 6.0 prep refactors?

[maartenbreddels]

I think only /home/pi/.local/share/jupyter/nbconvert/templates or '/Users/ac/Library/Jupyter/nbconvert/templates' are supported, I don't know if this was removed, or ever present, but given the new system, I think it should be these locations following 'jupyter standards'

[txoof]

I would greatly appreciate the documentation on this. Thanks for taking up the question.

I can confirm that /home/pi/.local/share/jupyter/nbconvert/templates and /Users/ac/Library/Jupyter/nbconvert/templates are supported. Dropping custom templates into those directories works as expected.

In the meantime, I think I sorted out a solution for specifying a custom location based on this stack overflow answer for version 5.

Is this a logical way to specify a custom template location in the configuration file?

I was mistaken
This solution does not work properly, I had left a copy of the template in one of the known paths and that was giving me a false-positive on my tests.

~/.jupyter/jupyter_nbconvert_config.py

import os
c = get_config()
c.TemplateExporter.template_paths = ['.', os.path.expanduser('~/my/path/to/custom_templates)']

I should have mentioned that I do have a conf.json that matches the suggestion above. For completeness the files for the template in question are included below:

conf.json

{
    "base_template": "base",
    "mimetypes": {
        "text/x-python": true
    }
}

index.py.j2

{%- extends 'null.j2' -%}

## set to python3
{%- block header -%}
#!/usr/bin/env python3
# coding: utf-8
{% endblock header %}

## remove cell counts entirely
{% block in_prompt %}
{% if resources.global_content_filter.include_input_prompt -%}
{% endif %}
{% endblock in_prompt %}

## remove markdown cells entirely
{% block markdowncell %}
{% endblock markdowncell %}

{% block input %}
{{ cell.source | ipython2python }}
{% endblock input %}


## remove magic statement completely
{% block codecell %}
{{'' if "get_ipython" in super() else super() }}
{% endblock codecell%}

[MSeal]

So the issue is that by setting template_paths you're replacing ALL the template paths with just the ones you specify so it's eliminating the paths that include the base template you have installed on your system. To fix this you should instead be setting extra_template_basedirs so it's additive instead of exclusive.

$ ls /home/mseal/Workspace/tmp/tester
total 16
drwxrwxr-x 2 mseal mseal 4096 Oct  1 00:37 .
drwxrwxr-x 5 mseal mseal 4096 Oct  1 00:38 ..
-rw-rw-r-- 1 mseal mseal   88 Oct  1 00:36 conf.json
-rw-rw-r-- 1 mseal mseal  575 Oct  1 00:37 index.py.j2

$ jupyter nbconvert --to script --template tester --TemplateExporter.extra_template_basedirs="/home/mseal/Workspace/tmp" simple_execute.ipynb
[NbConvertApp] Converting notebook simple_execute.ipynb to script
[NbConvertApp] Writing 74 bytes to simple_execute.py

[txoof]

@MSeal -- Thank you! That's exactly what I was looking for.

I hope my PR goes through appropriately. I'm quite novice with this.

[MSeal]

I updated the customization docs to better indicate how to add custom template paths. Going to close this issue as I believe it's resolved.

[txoof]

@MSeal Thank you for the updates and the guidance for updating the documentation.

[mfouesneau]

I am trying to use the new nbconvert template structure, but I cannot make it recognize custom templates:

> nbconvert  examples/example.ipynb --template report --to pdf --TemplateExporter.extra_template_basedirs="./templates/"
[NbConvertApp] WARNING | Config option `extra_template_basedirs` not recognized by `PDFExporter`.