Ex_doc: Rendering regression in 0.22

[imanhodjaev]

Hey,

In version 0.21 it was rendering markdown with HTML blocks as expected for example the following block

<h1 align="center">
Bla bla
</h1>
<p align="center">
  <img width="150" height="150" src="https://image.svg"/>
</p>

In version 0.22 ex_doc renders tags as plain text.

Also parser is unable to properly parse text blocks for example if you put

<h1 align="center">Title</h1>

It will render

Title</h1>

Elixir version: 1.10.3
earmark: 1.4.5
makeup: 1.0.3
makeup_elixir: 0.14.1

Thanks.

[josevalim]

Can you please provide a sample project that reproduces the error? Thanks.

[imanhodjaev]

Hi Jose,

Project documentation is here https://hexdocs.pm/dust/readme.html and GitHub link is https://github.com/imanhodjaev/dust

[eksperimental]

The generated HTML is fine and it looks fine in my browser.
Did you try with a different browser or with no extensions installed?

[imanhodjaev]

@eksperimental yes I tried on different browsers also hex docs https://hexdocs.pm/dust/readme.html show the same thing

[eksperimental]

@imanhodjaev I see what you mean now.
I went head and wrote a test and ran git bisect and the commit that introduced the bug is:

f553cc0c406ab0b589be652bd4148893c785bd8b is the first bad commit
commit f553cc0c406ab0b589be652bd4148893c785bd8b
Author: Wojtek Mach <[email protected]>
Date:   Thu Mar 5 15:53:42 2020 +0100

    Use new autolinker

    * Replace `ExDoc.Markdown.to_html/2` with `to_ast/2`
    * Change the flow from `markdown->autolink->html` to `markdown->ast->autolink->html`
    * Add `:app` to ExDoc configuration (for generating project-local links)

/cc @wojtekmach
I can share the test and the shell script if needed

[imanhodjaev]

@eksperimental awesome! Thank you very much for your investigation. ✨🚀✨

[eksperimental]

Here's a reduced case:
it extracts the title from <h1>Title</h1> to the following string Title</h1> and this is inserted in the <title> and <h1> elements

Also <p><img src="image.svg"/></p> to
to <p>&lt;img src=&quot;image.svg&quot;/&gt;&lt;/p&gt;</p>

[wojtekmach]

Thanks for the report and confirmation. The issue is similar to https://github.com/elixir-lang/ex_doc/issues/1168, we shouldn't escape.

[eksperimental]

PR submitted #1190

[josevalim]

Closing in favor of #1190.

[lexmag]

The regression happens from the EarmarkParser usage:
ExDoc 0.21 started to use Earmark 1.4 (shipped with EarmarkParser),
and now ExDoc directly depends on EarmarkParser.
EarmarkParser has incomplete support for HTML blocks: (https://github.com/RobertDober/earmark_parser/issues/30), and sometimes it could discard content (https://github.com/RobertDober/earmark_parser/issues/31).

The following is a workaround for now:

{:earmark, "~> 0.13.0", only: :dev, runtime: false},
{:ex_doc, "~> 0.20.0", only: :dev, runtime: false},

[wojtekmach]

@RobertDober sorry for some of the earmark reports that stemmed from this and similar issues, we were incorrectly handling the Earmark AST. Now we're properly differentiating between verbatim and non-verbatim nodes which pretty much solves the problem!

[RobertDober]

again no need to apologise I am not really up to speed with these issues, we had been planning to make EarmarkParser more GFM like, as it is becoming quite clear that for many, nowadays Markdown == GFM, unfortunately I do not have time, energy and inspiration right now, sorry.

I will of course at least tend to my social duties.

[josevalim]

Hi @RobertDober!

FWIW, EarmarkParser works great for us! And to be clear, I don't think anyone should expect you to make Earmark behave more like GFM. If you would welcome contributions that add those features then you can always say so and hope people will contribute, but you shouldn't really sweat or work on it unless you actually want to. :)

Have a great week!

[RobertDober]

Oh I will certainly accept any PR in that direction, but it is a delicate matter ...

Thanx for your kind encourgament and (IIANM) Prospero Ano Novo