Notebook: File save hooks—docs & question

[khaeru]

notebook/docs/source/extending/savehooks.rst gives an example of a post-save hook for exporting a script, one that I find very useful! Two issues and a question about this:

1. As far as I can tell, this is code from elsewhere in the code base that could be just imported, e.g. a user can set the hook in jupyter_notebook_config.py with:

python from notebook.services.contents.filemanager import _post_save_script c.FileContentsManager.post_save_hook = _post_save_script

1. The example in savehooks.rst is out of date—flake8 warns that py_fname is defined but not used.
2. In using this code, the ScriptExporter that's created seems to eventually create a PythonExporter (at least, for IPython notebooks). It seems not possible to configure this Exporter through the normal configuration mechanism. For instance adding any of the following lines to jupyter_notebook_config.py seems to have no effect:

python c.Exporter.template_file = 'custom.tpl' c.TemplateExporter.template_file = 'custom.tpl' c.PythonExporter.template_file = 'custom.tpl'

Some of these appear in the nbconvert docs, but that is for jupyter_nbconvert_config.py, not loaded by the notebook. I've also tried hacking _post_save_script to pass or set the template_file or template_path options, and looked in vain for documentation about configuring custom exporters for ScriptExporter. (My ultimate goal here is to use a custom .tpl file for this export so that I can suppress prompt numbers.)

[Carreau]

the post_save_hook is any import string or callable, which mean that in your notebook_config.py you can import and instantiate the ScriptExporter yourself, as well as pass a config object (for custom templates) yourself:

from nbconvert.exporters.script import ScriptExporter
from traitlets.config import Config
c = Config()
ScriptExporter(config=c)
...

The post_save_hook is not passed any config instance, though it is passed contents_manager=self so you should be able to use c.FileContentsManager.(Script)Exporter.template_file = 'custom.tpl' (Script beeing optional) in jupyter_notebook_config.py (note the extra FileContentsManager.(Script)Exporter nesting. But we should check that.

[michaelaye]

Could someone confirm/comment if point 1. made above is a safe way to implement the post_save_hook?

[Carreau]

No it's not safe, the _post_save_script has a leading underscore it's hence private.
And some of the code around that should likely warn using a DeprecationWarning so will likely be removed in future version.

Though I don't see any reason for this to be dangerous.

[khaeru]

@Carreau — the suggestion doesn't seem to work.

My knowledge of traitlets is about zero, but I continued the clumsy hacking I mentioned in my original point 3. With the following in jupyter_notebook_config.py:

import io
import os

from nbconvert.exporters import TemplateExporter
from notebook.utils import to_api_path

from traitlets import Dict
from traitlets.config import Config
from traitlets.utils.importstring import import_item


# From nbconvert/exporters/script.py
class ScriptExporter(TemplateExporter):

    _exporters = Dict()

    def _template_file_default(self):
        return 'script'

    def from_notebook_node(self, nb, resources=None, **kw):
        langinfo = nb.metadata.get('language_info', {})

        # delegate to custom exporter, if specified
        exporter_name = langinfo.get('nbconvert_exporter')
        if exporter_name and exporter_name != 'script':
            self.log.debug("Loading script exporter: %s", exporter_name)
            from nbconvert.exporters.export import exporter_map
            if exporter_name not in self._exporters:
                if exporter_name in exporter_map:
                    Exporter = exporter_map[exporter_name]
                else:
                    self.log.debug("Importing custom Exporter: %s",
                                   exporter_name)
                    Exporter = import_item(exporter_name)
                self.log.debug("ScriptExporter.template_file: %s",
                               self.template_file)
                c = Config()
                # @khaeru: the following works!
                # c.PythonExporter.template_file = 'custom.tpl' # **********
                self._exporters[exporter_name] = Exporter(parent=self,
                                                          config=c)
            exporter = self._exporters[exporter_name]
            self.log.debug("ScriptExporter._exporters['%s'].template_file: %s",
                           exporter_name, exporter.template_file)
            return exporter.from_notebook_node(nb, resources, **kw)

        self.file_extension = langinfo.get('file_extension', '.txt')
        self.output_mimetype = langinfo.get('mimetype', 'text/plain')
        return super(ScriptExporter, self).from_notebook_node(nb, resources,
                                                              **kw)


# From notebook/services/contents/filemanager.py
_script_exporter = None


def post_save_script(model, os_path, contents_manager, **kwargs):
    """convert notebooks to Python script after save with nbconvert
    replaces `ipython notebook --script`
    """

    if model['type'] != 'notebook':
        return

    global _script_exporter
    if _script_exporter is None:
        c = Config()
        # @khaeru: none of these work
        c.template_file = 'custom.tpl'
        c.Exporter.template_file = 'custom.tpl'
        c.TemplateExporter.template_file = 'custom.tpl'
        c.ScriptExporter.template_file = 'custom.tpl'
        c.PythonExporter.template_file = 'custom.tpl'
        c.FileContentsManager.Exporter.template_file = 'custom.tpl'
        c.FileContentsManager.TemplateExporter.template_file = 'custom.tpl'
        c.FileContentsManager.ScriptExporter.template_file = 'custom.tpl'
        c.FileContentsManager.PythonExporter.template_file = 'custom.tpl'
        # @khaeru: since ScriptExporter gives 'parent=self' when instantiating
        #          PythonExporter, tried these. They also don't work:
        c.ScriptExporter.PythonExporter.template_file = 'custom.tpl'
        c.FileContentsManager.ScriptExporter.PythonExporter.template_file = \
            'custom.tpl'
        _script_exporter = ScriptExporter(config=c, parent=contents_manager)
    log = contents_manager.log

    base, ext = os.path.splitext(os_path)
    script, resources = _script_exporter.from_filename(os_path)
    script_fname = base + resources.get('output_extension', '.txt')
    log.info("Saving script /%s", to_api_path(script_fname,
                                              contents_manager.root_dir))
    with io.open(script_fname, 'w', encoding='utf-8') as f:
        f.write(script)

# @khaeru: also don't work
c.Exporter.template_file = 'custom.tpl'
c.TemplateExporter.template_file = 'custom.tpl'
c.ScriptExporter.template_file = 'custom.tpl'
c.PythonExporter.template_file = 'custom.tpl'
c.FileContentsManager.Exporter.template_file = 'custom.tpl'
c.FileContentsManager.TemplateExporter.template_file = 'custom.tpl'
c.FileContentsManager.ScriptExporter.template_file = 'custom.tpl'
c.FileContentsManager.PythonExporter.template_file = 'custom.tpl'
c.ScriptExporter.PythonExporter.template_file = 'custom.tpl'
c.FileContentsManager.ScriptExporter.PythonExporter.template_file = \
    'custom.tpl'

c.FileContentsManager.post_save_hook = post_save_script

Then running jupyter notebook --debug and creating and checkpointing a new notebook to trigger the save hook:

[I 12:58:00.237 NotebookApp] Saving file at /Untitled.ipynb
[D 12:58:00.237 NotebookApp] Saving /home/khaeru/Untitled.ipynb
[D 12:58:00.304 NotebookApp] Running post-save hook on /home/khaeru/Untitled.ipynb
[D 12:58:00.317 NotebookApp] Loading script exporter: python
[D 12:58:00.317 NotebookApp] ScriptExporter.template_file: custom.tpl
[D 12:58:00.322 NotebookApp] ScriptExporter._exporters['python'].template_file: python
[D 12:58:00.322 NotebookApp] Applying preprocessor: coalesce_streams
[D 12:58:00.323 NotebookApp] Attempting to load template python.tpl
[D 12:58:00.330 NotebookApp] Loaded template python.tpl
[D 12:58:00.331 NotebookApp] Attempting to load template python.tpl
[D 12:58:00.331 NotebookApp] Loaded template python.tpl
[I 12:58:00.430 NotebookApp] Saving script /Untitled.py

But if I uncomment the line with **********:

[I 13:07:02.806 NotebookApp] Saving file at /Untitled.ipynb
[D 13:07:02.806 NotebookApp] Saving /home/khaeru/Untitled.ipynb
[D 13:07:02.830 NotebookApp] Running post-save hook on /home/khaeru/Untitled.ipynb
[D 13:07:02.842 NotebookApp] Loading script exporter: python
[D 13:07:02.842 NotebookApp] ScriptExporter.template_file: custom.tpl
[D 13:07:02.846 NotebookApp] ScriptExporter._exporters['python'].template_file: custom.tpl
[D 13:07:02.848 NotebookApp] Applying preprocessor: coalesce_streams
[D 13:07:02.848 NotebookApp] Attempting to load template custom.tpl.tpl
[D 13:07:02.853 NotebookApp] Attempting to load template custom.tpl
[D 13:07:02.854 NotebookApp] Loaded template custom.tpl
[D 13:07:02.854 NotebookApp] Attempting to load template custom.tpl.tpl
[D 13:07:02.854 NotebookApp] Attempting to load template custom.tpl
[D 13:07:02.854 NotebookApp] Loaded template custom.tpl
[I 13:07:02.855 NotebookApp] Saving script /Untitled.py

I believe this indicates that PythonExporter is not receiving the configuration options, but I still don't know how to make it work (short of having those 100+ lines of code in my config file).

[khaeru]

As a temporary workaround, for anyone who's curious: at the line marked ********** above, I put:

                from jupyter_core.paths import jupyter_config_dir
                c.PythonExporter.template_path = [
                    os.path.join(jupyter_config_dir(), 'templates')] + \
                    self.template_path  # i.e., of ScriptExporter, usually '.'

Then in my $JUPYTER_CONFIG_DIR/templates/python.tpl the following, modified from the stock python.tpl:

{%- extends 'null.tpl' -%}

{% block header %}
# coding: utf-8
{% endblock header %}

{% block in_prompt %}
# Cell:
{% endblock in_prompt %}

{% block input %}
{{ cell.source | ipython2python }}
{% endblock input %}

{% block markdowncell scoped %}
{{ cell.source | comment_lines }}
{% endblock markdowncell %}

This is helpful to me because I tend to keep the exported .py under version control, and this way if only the prompt numbers change, the file does not change.

[JamiesHQ]

@Carreau & @minrk : can this issue be closed based on other work that has been completed?

[minrk]

I think we can still take some of what's discussed here and expand the examples we have in the documentation.

[urbandw]

The bigger issue here is that there seems to be no way for a user-set value of include_input_prompt to make its way to https://github.com/jupyter/nbconvert/blob/e59f4ceaf6e67931f178498dc8a0d91bad7571bc/nbconvert/templates/python.tpl#L9.

I've tried instantiating an exporter with _exporter = ScriptExporter(config=c) where c = {"ScriptExporter": {"include_input_prompt": False}}, and also passing a resources dictionary to _exporter.from_filename(resources=resources), or _exporter.from_notebook_node(resources=resources) where resources.global_content_filter.include_input_prompt == True. None of the above works.

I would guess that the overriding of user inputs with default settings is a bigger problem than just include_input_prompt. Are there any other options?

[bashtage]

@urbandw you can set these values using:

_exporter = ScriptExporter()
_exporter.exclude_input_prompt = True # Opposite since excluded