Lsp: Single newlines are removed when rendering markdown

[rchl]

As reported by @TheSecEng on Discord, there are problems with rendering hover info from some servers:

For a payload like:

{
    "contents": [
        {
            "language": "python",
            "value": "b64decode(s: _decodable, altchars: bytes=..., validate: bool=...) -> bytes"
        },
        "Decode the Base64 encoded bytes-like object or ASCII string s.\n\nOptional altchars must be a bytes-like object or ASCII string of length 2\nwhich specifies the alternative alphabet used instead of the '+' and '/'\ncharacters.\n\nThe result is returned as a bytes object.\xa0\xa0A binascii.Error is raised if\ns is incorrectly padded.\n\nIf validate is False (the default), characters that are neither in the\nnormal base-64 alphabet nor the alternative alphabet are discarded prior\nto the padding check.\xa0\xa0If validate is True, these non-alphabet characters\nin the input result in a binascii.Error."]
}

the output is missing some spaces between words:

This is because single newlines are removed when parsing markdown.

This can be fixed by adding markdown.extensions.nl2br extension to our frontmatter config (the extension is normally enabled but we are not using it currently). I'm not sure if we are on purpose not using that extension or what's the reason.

I have enabled it locally and have quickly noticed a small issue that is more generic to markdown parsing but it doesn't happen in VSCode. There is now extra padding between the code block and the description:

The payload for that hover content is:

{
  "contents": {
    "kind": "markdown",
    "value": "```python\n(function) cache_path: () -> Unknown\n```\nReturns the path where Sublime Text stores cache files"
  }
}

and it's related to there being a newline just after the code block ends:

```\nReturns

[facelessuser]

Stripped newlines are no longer an issue with mdpopups.

With that said, the issue with there not being two newlines between fenced content and other content is still an issue:

````markdown

fenced content

paragraph
````

Per the SuperFences instructions, there should be two new lines (a blank line) separating content:

````markdown

fenced content

paragraph
````

This becomes particularly noticeable when nl2br is enabled.

The Markdown parser will create a placeholder when in parses fenced content in the preprocessor step:

{placeholder}
paragraph

This will treat the entire content as a paragraph and convert the newline to a <br>.

<p>{placeholder}<br>
paragraph</p>

Later the placeholder is replaced with actual fenced content. This is why two new lines are needed. If the server returned content as such, there would be no problem here.

It may be possible for SuperFences, the extension used in mdpopups, to work around this and maybe provide a buffer of an additional newline. I'm not sure if this could introduce unwanted side effects when paired in other Markdown constructs, so we would have to experiment.

Just thought I'd provide an explanation.