Jupyter-book: Table header issue of R dataframe from Jupyter R notebook

[gaow]

I'm currently developing a Jupter book using this wonderful project. However I'm running into one issue with table rendering in Jupyter R notebook.

Problem

The table header are messed up in the HTML output:

Reproducing it

To save you the efforts reproducing the issue please find attached the relevant section from the intermediate markdown file exported by the build script into the _build folder:

R_dataframe_output.md.txt

The <div markdown="0" class="output output_html"> in question was automatically generated from Jupyter notebook.

Software versions

1.

> jupyter-notebook --version
5.6.0

1. jupyter-book version: the current master.

Please let me know if you need more information. Looking forward to hearing from you!

[choldgraf]

hmmm - it looks like the actual generated HTML isn't using the < and > characters, instead it's coding them with &lt and &gt so they're being rendered as characters and not as HTML syntax. Is there a way to check on your end to see if that's what the R table renderer is doing (as opposed to this happening within Jupyter book?)

[gaow]

Thanks @choldgraf for your prompt feedback. I've created a minimal working example here

mwe-issue72.tar.gz

The Untitled.ipynb is a notebook using R kernel. If you open it with plain text editor you see the HTML table rendering seems fine. The Untitled.md is the markdown file generated by script generate_book.py which also looks good. I suspect it has something to do with the jekyll rendering step.

For my example in the original post, you are right, the coding is indeed &lt and &gt in the HTML:

</code></pre></div></div>

<div class="output output_html">
<table>
<thead><tr>&lt;th scope=col&gt;DSC&lt;/th&gt;&lt;th scope=col&gt;simulate&lt;/th&gt;&lt;th scope=col&gt;simulate.true_mean&lt;/th&gt;&lt;th scope=col&gt;analyze&lt;/th&gt;&lt;th scope=col&gt;analyze.est_mean&lt;/th&gt;&lt;th scope=col&gt;score&lt;/th&gt;&lt;th scope=col&gt;score.error&lt;/th&gt;</tr></thead>

[choldgraf]

huh, that is really weird, I agree that it seems the markdown produced by generate_book is fine, but I have no idea why jekyll would only encode just this one line. I'll try looking into it

[choldgraf]

ah - I think I see what the problem is - the thead section is actually not valid HTML because of the scope=col sections. I think it should be scope="col" and I tried adding quotes that it started working once I did this. Maybe Jekyll sees that it isn't valid HTML and so tries to encode it as "regular" text instead?

[gaow]

ahh indeed. Thanks for identifying the problem so quickly. It looks like a Jupyter Notebook problem with R kernel then, because the syntax in the ipynb file is already wrong ... maybe an upstream fix at Jupyter's end is the best fix?

[choldgraf]

I'm guessing it's a problem with the R kernel that you're using, is it IRKernel?

[gaow]

Yes it is IRKernel. I'm on the current cran version 0.8.15. Perhaps we should open a ticket at their repo? For the time being do you suggest parsing all HTML fixing this particular problem (or would like to do it on your end)?

[choldgraf]

Yeah I'd open an issue about it there. I don't think there's anything we can do on the jupyter book side, unfortunately. Let me know if you can think of something!

[gaow]

Thank you I've opened a ticket (see above). For the time being I'd just run an additional perl command to replace those patterns in the md file under _build, until there is an upstream fix. I'm cool if you want to close the ticket.

[choldgraf]

sounds good! let's keep it open until we hear something back from that issue, because if a fix doesn't happen then we should probably document this in the book guide

[flying-sheep]

ah - I think I see what the problem is - the thead section is actually not valid HTML because of the scope=col sections. I think it should be scope="col" and I tried adding quotes that it started working once I did this. Maybe Jekyll sees that it isn't valid HTML and so tries to encode it as "regular" text instead?

Hi! I’m the author of repr (the package used by IRkernel rendering that HTML). Unquoted attribute values have been valid in HTML4 and continue to be so in HTML5. Whatever HTML parser is used here has a bug that needs to be fixed.

That bug is probably either in nbcleaner or something else active before the creation of the Jekyll Markdown, or it’s in the Markdown+HTML parser used by Jekyll.

/edit: I just checked: the emitted markdown contains the HTML unchanged, so it’s Jekyll’s default markdown parser, kramdown.

PS: I’m not obnoxiously against changing the HTML I emit, but I want to see an honest effort to fix the actual bug before I do that.

[flying-sheep]

I did a bit of research and… this will be fixed in the next version of kramdown: https://github.com/gettalong/kramdown/pull/523

So you just need to wait, ask @gettalong for a new kramdown release, or use the git version of kramdown and everything is peachy.

[choldgraf]

ah beautiful - thanks so much for following up on this @flying-sheep ! I didn't realize that this was now valid HTML, so I've learned something new today :-)

Let's keep this issue open so that we can follow up w/ kramdown later. @gaow does this work OK for you?

[gaow]

Sure thank you @choldgraf for reporting it. Thank you @flying-sheep for your input!

[flying-sheep]

I didn't realize that this was now valid HTML, so I've learned something new today :-)

great! i’m happy that i could be of value!

however I can’t resist: now? :laughing: it has been valid since at least 21 years (HTML 4.0, 1998)

[choldgraf]

LOL that is slightly embarassing

[flying-sheep]

don’t worry! there’s so many concepts and specs we’re required to know that we can hardly be expected to keep all their details in our minds.

[choldgraf]

Closing as this should be superceded by beta.jupyterbook.org!

[flying-sheep]

Well, it’s been fixed anyway. In your words:

Let's keep this issue open so that we can follow up w/ kramdown later.

[choldgraf]

yeah - the reason it's superceded is because we're not going to use kramdown in the new build system 👍

[flying-sheep]

Ah, I see! I’d assumed that kramdown had a new release since 2018 :smile:

[gettalong]

@flying-sheep It does :wink: