Nbconvert: Markdown to RST converts all Headers to level 1

[mscuthbert]

Hello! A bit of a regression in the new v4 header types. Prior to nbconvert 4, running this code converted a python notebook into something that was able to be read by Sphinx properly:

    app = nb.NbConvertApp.instance() # @UndefinedVariable
    app.initialize(argv=['--to', 'rst', ipythonNotebookFilePath])
    app.writer.build_directory = os.path.dirname(ipythonNotebookFilePath)
    app.start()

In the newest version, however, all headings (which are now just markdown with # prefixes) get converted to top level ============ rst headings, rather than ======== or --------- or ~~~ according to the context. Generating the markdown directly via nbconvert and then converting via pandoc to rst does not show this problem whether or not --atx-headers is selected.

Thanks for great work!

[minrk]

I think this is a recent change in behavior in pandoc, where when you pass snippets, if there's only one header it uses ===, regardless of the originating format. What version of pandoc are you using?

[mscuthbert]

pandoc 1.15.0.6
Compiled with texmath 0.8.2.2, highlighting-kate 0.6.
(OS X)

All of the files I'm using have more than one header level, including a top-level ====/# near the beginning of the file. Thanks for looking into it.

[mscuthbert]

Indeed it is from a change in pandoc. v.1.13.2 works great. v.1.14 (next version; May 28, 2015) breaks. It's probably this change:

Normalize headings to sequential levels (Nikolay Yakimov). This is pretty much required by docutils.

Ironically, this is what is killing docutils for me in my Jupyter/Notebook to Sphinx workflow.

So it seems to be a pandoc problem in some sense rather than a nbconvert problem, but I wonder if there's a hint that nbconvert could be making to not have this behavior function.

[minrk]

Yup, I think we just need to ask pandoc what the right thing to do is.

[mscuthbert]

I made a note there. Thanks!

[minrk]

Pandoc has slightly improved the situation, but there doesn't appear to be anything we can do in the meantime on the nbconvert side.

[jhamrick]

Ugh, @willingc and I just came across this in the nbgrader documentation. A workaround for nbconvert could potentially be to create a special preprocessor just for the rst exporter that does the following:

1. Change the type of all markdown cells to raw nbconvert, and convert their source to rst
2. Parse out the heading levels from the original markdown cells and make sure they are the right level in the rst output

It's pretty hacky, but I think it would mostly work, at least. I may write something like this for the nbgrader docs, since I really need the heading levels there to be correct, but maybe it's too hacky for nbconvert proper...

[minrk]

@jhamrick are you running on a machine you control? Pinning pandoc to 1.13 might be the easiest temporary solution.

[minrk]

The other workaround is going nbconvert → md; pandoc → rst. I'm not sure how much is lost there, though.

[minrk]

A fix has been merged into pandoc that should improve things once released. I'm guessing it will first appear in 1.16, which means we should consider blacklisting (or warning, at least) 1.14-1.15 for rst export.

[mscuthbert]

Supposedly 1.15.1 (15 October 2015) fixed this. I upgraded on one server and it appears to be functional. It's not enough for a close yet, but it's enough that I'll be willing to try out on a production machine soon.

[mpacer]

Closing for now — talked with @jhamrick and it's been long enough since they've had to do something like that for nbgrader that it's unlikely that this is still a bug.

[mscuthbert]

(and pandoc 1.19 is working great with nbconvert)