Rust: Tracking issue for hoedown -> pulldown regressions

[steveklabnik]

In https://github.com/rust-lang/rust/pull/40338, we landed pulldown-cmark. 🎊

But, given that it's a different renderer, there are bound to be differences. This comment pointed out some obvious problems. These bugs are going to be much easier to clean up than the initial PR was to land, though 👍

To help with this work, I generated docs for both the commit before and the commit of, ran all the HTML files through tidy-html5, and put it up here: https://github.com/steveklabnik/docdiff/commit/ddda1fe51588755874bbfd331ecb134e79922596 the ' -> " changes are expected, but that tool doesn't have the ability to re-write those, so I left them in for now, which, frankly, makes the diff kinda large. working on it.

Here's the stuff we've found so far:

• [x] horizontal rules
• [x] footnotes (waiting for pulldown update https://github.com/google/pulldown-cmark/pull/21)
• [x] images
• [x] <br /> for HardBreaks
• [x] the title part of links being rendered as visible text rather than as an attribute on the <a> tag (imperio update: I'll need an example to be able to fix this one)
• [x] MarkdownHtml now doing the same as Markdown when it should be escaping raw HTML.
• [x] [some strange rendering](https://github.com/steveklabnik/docdiff/commit/ddda1fe51588755874bbfd331ecb134e79922596#diff-73ae1ced9c2ca5ba756a60f7b63b3e9dL785) around methods at times? rendered: https://doc.rust-lang.org/nightly/std/collections/struct.LinkedList.html#method.front_place source https://github.com/rust-lang/rust/blob/master/src/libcollections/linked_list.rs#L700 fix: https://github.com/rust-lang/rust/pull/41066
• [x] In src/libstd/collections/hash/map.rs under "Relevant papers/articles:", it appears the ordered list has been rendered as an unordered list instead. PR r+'d
• [x] ^ now needs to be an html <sup>. (this is not a regression in the code, but something that needs to be adapted in the docs) https://github.com/rust-lang/rust/pull/41043
• [x] https://github.com/rust-lang/rust/issues/40918 (this one was due to markdown spec switch)
• [x] Footnotes should have a link back to their referent, see https://github.com/rust-lang/rust/pull/40919#discussion_r109126016
• [x] words getting smooshed together (I think the hard/soft break fixed this, I'm not observing it now)
• [x] https://github.com/rust-lang/rust/issues/40978 (this one was due to markdown spec switch)

This change will land in tonight (3/29)'s nightly, so we can also poke at them then. I plan on making a users post tomorrow to advertise this bug.

Tagging as a regression so we make sure to take care of it. Marking as P-high and assigning @GuillaumeGomez and @frewsxcv who are both already working at knocking some of this out.

[ScottAbbey]

In core/intrinsics/fn.overflowing_sub.html.txt (and elsewhere?), hoedown was rendering 2^N as 2<sup>N</sup>, the new version just leaves it as 2^N.

This isn't recognized in the CommonMark spec, probably need to change the source to 2<sup>N</sup> ?

[steveklabnik]

@ScottAbbey agree 100%; we'll have to send in a PR to fix. I wonder what the easiest way of finding all of them is.....

[ScottAbbey]

It looks like markdown.rs is squishing things together when a list item spans multiple lines. In pulldown-cmark's html.rs, this would have been a SoftBreak that got turned into a \n, it appears this rendering is just throwing those away. Maybe not all of them? But some?

[steveklabnik]

For the ^ issue; rg -g !*.cpp -g !*.h \^ | rg "///" is giving me pretty reasonable results. Still lots of false positives, but manageable.

[ScottAbbey]

In src/libstd/collections/hash/map.rs under "Relevant papers/articles:", it appears the ordered list has been rendered as an unordered list instead.

[SergioBenitez]

I'm getting an unexpected panic from rustc that appears to be caused by the transition:

---- src/request/form/from_form_value.rs - request::form::from_form_value::FromFormValue (line 59) stdout ----
    error: unknown start of token: `
 --> <anon>:2:42
  |
2 | A value is validated successfully if the `from_str` method for the given
  |                                          ^

thread 'rustc' panicked at 'Box<Any>', /checkout/src/libsyntax/parse/lexer/mod.rs:81
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'rustc' panicked at 'couldn't compile the test', /checkout/src/librustdoc/test.rs:270

The markdown being referenced in the error looks like this:

/// Rocket implements `FromFormValue` for many standard library types. Their
/// behavior is documented here.
///
///   * **f32, f64, isize, i8, i16, i32, i64, usize, u8, u16, u32, u64**
///
///   **IpAddr, Ipv4Addr, Ipv6Addr, SocketAddrV4, SocketAddrV6, SocketAddr**
///
///     A value is validated successfully if the `from_str` method for the given
///     type returns successfully. Otherwise, the raw form value is returned as
///     the `Err` value.

Edit: I'll open a separate issue for this.

[bluss]

Here's one I found. (The diff is the fix needed, the extra line, to have the link render as a link).

diff --git a/src/impl_views.rs b/src/impl_views.rs
index c26a4ef1..ac2f787c 100644
--- a/src/impl_views.rs
+++ b/src/impl_views.rs
@@ -19,6 +19,7 @@ use StrideShape;
 /// Methods for read-only array views `ArrayView<'a, A, D>`
 ///
 /// Note that array views implement traits like [`From`][f] and `IntoIterator` too.
+///
 /// [f]: #method.from
 impl<'a, A, D> ArrayBase<ViewRepr<&'a A>, D>
     where D: Dimension,

[GuillaumeGomez]

@bluss: A commonmark thing once again. Get a look here. When you add a newline, it works as expected.

[ollie27]

Another bit of expected breakage is that autolinks now require <> around them. This can be seen in the error index: E0033.

If anyone is interested, I've started implementing what I suggested here: https://github.com/rust-lang/rust/pull/40338#issuecomment-285918453, using push_html to render the HTML and iterator adapters to modify to our needs. This will simplify rustdoc and fix most of the issues with the current implementation. Unfortunately it will involve removing most of the rendering code that's there at the moment.

[GuillaumeGomez]

@ollie27: Fine by me. But please wait for #40919 to get merged before please so that we'll avoid regressions.

[bluss]

Linebreak issue in bullet points:

Input:

//! - Still iterating on and evolving the crate this is a longer sentence that
//!   is going to wrap

Output includes the two words pulled together across the linebreak: “thatis”

“• Still iterating on and evolving the crate this is a longer sentence thatis going to wrap”

[steveklabnik]

Output includes the two words pulled together across the linebreak: “thatis”

I thought I made a bullet for this, but did not. However, I think this is related to the hard/soft break stuff. Making a bullet now though.

[GuillaumeGomez]

It is. I'll handle when the current fix PR is merged.

[Marwes]

28712 seems to have been fixed causing this code block to be treated as a doc test https://travis-ci.org/gluon-lang/gluon/jobs/217399595#L668.

Fixed in https://github.com/gluon-lang/gluon/pull/268/commits/b77232930d2cb51d814bc30a0ec55b345c5ebc98

[GuillaumeGomez]

@bluss: HardBreak issue has been fixed.

[bluss]

Great!

[steveklabnik]

I built the docs again with the latest stuff, including #41049. Here's a diff https://github.com/steveklabnik/docdiff/commit/9d9e78be4f93d70622ac7f8cc8add66222002604

Lots of noise now...

[tshepang]

wow, that's a massive diff!

[steveklabnik]

Okay so, I found one of the issues: https://github.com/rust-lang/rust/blob/master/src/libcollections/linked_list.rs#L700

this doesn't render as a link anymore, where it did previously.

[steveklabnik]

Fix in: https://github.com/rust-lang/rust/pull/41066

[steveklabnik]

@tshepang yeah I don't think my normalization did as good of a job this time :/

[steveklabnik]

I just reviewed this again. oustanding things:

footnotes (waiting for pulldown update)

raph is busy, and this is a tiny thing. We should still see if we can get it landed. I think it can be worked around with an extra newline? This is still an under-spec'd thing.

the title part of links being rendered as visible text rather than as an attribute on the tag

@ollie27 you had said this here, but I can't see what you're referring to. Any pointers?

MarkdownHtml now doing the same as Markdown when it should be escaping raw HTML.

@ollie27 same here, is this still a thing or not?

[GuillaumeGomez]

MarkdownHtml now doing the same as Markdown when it should be escaping raw HTML.

I didn't fix this at all. I actually don't even know where this is generated.

[bluss]

rustc 1.18.0-nightly (91ae22a01 2017-04-05) still shows a hard break issue. Not in regular paragraphs anymore, but in bullets.

Repro:

//! * Test line
//!   break

Expected Output: • Test line break

Actual output: • Test linebreak

[steveklabnik]

@bluss specifically what happens and what do you expect?

[bluss]

Updated the comment with that. It pulls together the word across the line break. Still https://github.com/rust-lang/rust/issues/40912#issuecomment-290825795 in fact (it's also a bullet)

[GuillaumeGomez]

Ah nice! However, this isn't a "hard" break issue but a soft one.

[ollie27]

the title part of links being rendered as visible text rather than as an attribute on the tag

@ollie27 you had said this here, but I can't see what you're referring to. Any pointers?

This was fixed for HTML by #40919 but is still broken for plain_summary_line. I've fixed it in #41112.

MarkdownHtml now doing the same as Markdown when it should be escaping raw HTML.

@ollie27 same here, is this still a thing or not?

MarkdownHtml is supposed to treat raw HTML as plain text so for example Struct<br> in Markdown would result in Struct<br> HTML. I've fixed this in #41112.

I've gone through a diff of the std docs and found two issues caused by differences between hoedown and pulldown-cmark not listed in the above checklist:

1. pulldown-cmark now needs a blank line before reference link URLs. I've fixed all example I could find of this in #41111.
2. autolinks now require <> around them. Affected pages:

error-index.html
std/process/struct.Command.html
*/hash/struct.SipHasher*.html
std/net/struct.UdpSocket.html

[steveklabnik]

@ollie27 thanks so much!

[steveklabnik]

It looks like this is complete, modulo some small possible changes upstream with regards to footnotes.

Great work everyone! Thank you so much ❤️

[bluss]

Thanks :heart: