Almanac.httparchive.org: MD tables not rendering correctly

[mikegeyser]

The markdown tables don't appear to be rendering correctly, as @rviscomi mentions in the PR #215. This is just a bug to track the investigation / resolution.

[mikegeyser]

Apologies for being so quiet, it's been heavy at work. Will try and look into the generate script and see what's wrong.

[rviscomi]

Thanks for filing this issue @mikegeyser. I totally understand that other commitments take precedence and greatly appreciate any time you can lend to this. 🙏

[mikegeyser]

@rviscomi I spent most of yesterday trying to debug the issue and I need some advice. The version of mistune published to pip is 0.8.4, and seems to have some endemic bugs - the regex that's responsible for tables just doesn't seem to work (looks like a problem with newlines). Additionally, the repo appears to be currently working on a v2 release that bears little resemblance to the version that we're using - leaving us in a bit of a bind. I believe that we have the following options available to us:

• Use a stable repo version, instead of the pip version: As far as I understand, we can reference a stable branch as a part instead of the pip repo. We can do that and fix the (hopefully minimal) breaking changes, and re-evaluate later.
• Use another markdown renderer: We can simply move to another python-based markdown renderer, such as python-markdown. It seems mature, but you don't know what you don't know. If we do this, we will need to rewrite the visualisation lexer.
• Use something else, like showdown: While it may be my inexperience in the python ecosystem, but there don't seem to be as many featureful markdown parsers as I expected. I have used showdown extensively, and trust it. I know it would mean relying on node for tooling, but I assume we are going to need some bundling/minification at some stage, so this may or may not be a big deal.
• Don't use md tables: I have to mention this as an option. Given that we're close to CDS, we can avoid the problem by writing our tables in html. I don't believe that this is the right option, as there may be a bunch of other similar pitfalls that chapter authors will experience.

Which option do you think is the right one to pursue?

I'm really sorry about the problems, I thought mistune was the best choice, but I'm committed to fixing it asap (whichever way we go).

[rviscomi]

Thanks for digging into this. I'm leaning towards your showdown suggestion. Very few people actually need to run the chapter generation script so I'm not too worried about adding a new requirement to the toolchain. Let's just go with the most tried and true solution to keep things moving smoothly.

[mikegeyser]

So I think I've found the root cause, it's the tables embedded inside <figure> elements (so that they have a <figcaption>. Most markdown renderers don't support parsing markdown inside html, but usually have a setting to enable it. In mistune it's parse_block_html=True but it doesn't appear to work...

In showdown it means adding an extra attribute in the md file, as follows:

<figure markdown>

| Fast FCP | Average FCP | Slow FCP |
| -------- | ----------- | -------- |
| 2.17%    | 37.55%      | 60.28%   |

<figcaption>Figure 3. Table of the percent of websites labelled as having fast, average, or slow FCP.</figcaption>
</figure>

I've tested it, and it works - so I'm going to go ahead with the showdown implementation.

[rviscomi]

Great, thanks for working on this!

[rviscomi]

Was about to complain that it wasn't working until I reread your comment. I just needed the markdown attribute. Carry on! 😄