Pkgdown: pandoc's markdown_github does not respect user <div>

Created on 18 Aug 2020  ·  14Comments  ·  Source: r-lib/pkgdown

I was having the same problem as #404 and I found that it is due to a bug in pandoc's deprecated markdown_github when --section-divs are used, which I illustrated in this gist.

In short, I found a solution and would like to submit a PR. Read on for a few reprexes that illustrate the problem and solution.

Problem

With the current version of {pkgdown}, divs are collapsed:

remotes::install_github("r-lib/pkgdown@eae56f0")
library("pkgdown")
library("usethis")
library("xml2")

tmp <- tempfile()
create_package(path = tmp, check_name = FALSE, roxygen = FALSE, open = FALSE)
cat("# Test", file = file.path(tmp, "README.md"))
cat('# Test
### level 3
<div class="boo">

## level2

</div>
', file = file.path(tmp, "classy.md"))
pkgdown::init_site(tmp)
x <- read_html(file.path(tmp, "docs", "classy.html"))
xml_find_all(x, ".//div[contains(@class, 'boo')]")
#> {xml_nodeset (1)}
#> [1] <div class="boo">\n</div>

It's also worth noting that this engine does not appear to process the fenced divs either:


Unprocessed Fenced Divs

remotes::install_github("r-lib/pkgdown@eae56f0")
library("pkgdown")
library("usethis")
library("xml2")

tmp <- tempfile()
create_package(path = tmp, check_name = FALSE, roxygen = FALSE, open = FALSE)
cat("# Test", file = file.path(tmp, "README.md"))
cat('# Test
### level 3

::: boo

## level2

:::
', file = file.path(tmp, "classy.md"))
pkgdown::init_site(tmp)
x <- read_html(file.path(tmp, "docs", "classy.html"))
xml_find_all(x, ".//div[contains(@class, 'boo')]")
#> {xml_nodeset (0)}
xml_text(x) # fenced divs unprocessed :'(
#> [1] "Test • file79b649068d68\n    \n      \n  \n    \n      \n        Toggle navigation\n        \n        \n        \n      \n      \n        file79b649068d68\n        0.0.0.9000\n      \n    \n\n    \n      \n  \n    \n     \n  \n\n\n  Reference\n\n      \n  \n\n\n      \n\n      \n  \n    \n      Test\n    \n\n\n\n\n\nlevel 3\n::: boo\n\n\n\nlevel2\n:::\n\n\n\n  \n\n  \n    Contents\n    \n\n\n\n\n\n      \n  Developed by First Last.\n\n\n\n  Site built with pkgdown 1.5.1.9000.\n\n\n      \n\n  \n\n\n  "

Created on 2020-08-18 by the reprex package (v0.3.0)

Solution: use pandoc's markdown engine

I would like to submit a PR for this change if one of the maintainers agrees that it is cromulent.

I've created a fork of the repo and was able to fix this issue using markdown with the markdown_github options. There was initially the same issues that #1195 ran into, but i was able to fix them using xpath manipulation. The branch is here: https://github.com/zkamvar/pkgdown/tree/05175508ac8cef1b9d20755497bc1ce718aa2160 with a reprex:

remotes::install_github("zkamvar/pkgdown@05175508")
library("pkgdown")
library("usethis")
library("xml2")

tmp <- tempfile()
create_package(path = tmp, check_name = FALSE, roxygen = FALSE, open = FALSE)
cat("# Test", file = file.path(tmp, "README.md"))
cat('# Test
### level 3
<div class="boo">

## level2

</div>
', file = file.path(tmp, "classy.md"))
pkgdown::init_site(tmp)
pkgdown::build_home(tmp, preview = FALSE)
x <- read_html(file.path(tmp, "docs", "classy.html"))
xml_find_all(x, ".//div[contains(@class, 'boo')]")
#> {xml_nodeset (1)}
#> [1] <div id="level2" class="section level2 boo">\n<h2 class="hasAnchor">\n<a  ...

Created on 2020-08-18 by the reprex package (v0.3.0)


Fenced Divs are processed :)

# remotes::install_github("r-lib/pkgdown@eae56f0")
remotes::install_github("zkamvar/pkgdown@05175508")
library("pkgdown")
library("usethis")
library("xml2")

tmp <- tempfile()
create_package(path = tmp, check_name = FALSE, roxygen = FALSE, open = FALSE)
cat("# Test", file = file.path(tmp, "README.md"))
cat('# Test
### level 3

::: boo

## level2

:::
', file = file.path(tmp, "classy.md"))
pkgdown::init_site(tmp)
pkgdown::build_home(tmp, preview = FALSE, quiet = TRUE)
x <- read_html(file.path(tmp, "docs", "classy.html"))
xml_find_all(x, ".//div[contains(@class, 'boo')]") # Fenced divs processed :)
#> {xml_nodeset (1)}
#> [1] <div id="level2" class="section level2 boo">\n<h2 class="hasAnchor">\n<a  ...

Created on 2020-08-18 by the reprex package (v0.3.0)

_Originally posted by @zkamvar in https://github.com/r-lib/pkgdown/issues/404#issuecomment-675637822_

reprex
Source
picture zkamvar

All 14 comments

Can you please provide a reprex using just markdown()?

hadley picture hadley on 20 Aug 2020
👍1

Old version

remotes::install_github("r-lib/pkgdown@1990820")
# remotes::install_github("zkamvar/pkgdown@c36b884")
library("pkgdown")
library("xml2")

tmp <- file.path(tempdir(), "test")
cat('# Test
### level 3

<div class="boo">

## level2

</div>
', file = tmp)

cat(pkgdown:::markdown(file.path(tmp)))
#> <div id="test" class="section level1">
#> <h1 class="hasAnchor">
#> <a href="#test" class="anchor"></a>Test</h1>
#> <div id="level-3" class="section level3">
#> <h3 class="hasAnchor">
#> <a href="#level-3" class="anchor"></a>level 3</h3>
#> <div class="boo">
#> </div>
#> <div id="level2" class="section level2">
#> <h2 class="hasAnchor">
#> <a href="#level2" class="anchor"></a>level2</h2>
#> </div>
#> </div>
#> </div>
x <- read_html(pkgdown:::markdown(file.path(tmp)))
xml_find_all(x, ".//div[contains(@class, 'boo')]")
#> {xml_nodeset (1)}
#> [1] <div class="boo">\n</div>

Created on 2020-08-20 by the reprex package (v0.3.0)


Fenced divs are also unprocessed 

remotes::install_github("r-lib/pkgdown@1990820")
# remotes::install_github("zkamvar/pkgdown@c36b884")
library("pkgdown")
library("xml2")

tmp <- file.path(tempdir(), "test")
cat('# Test
### level 3

::: boo

## level2

:::
', file = tmp)

cat(pkgdown:::markdown(file.path(tmp)))
#> <div id="test" class="section level1">
#> <h1 class="hasAnchor">
#> <a href="#test" class="anchor"></a>Test</h1>
#> <div id="level-3" class="section level3">
#> <h3 class="hasAnchor">
#> <a href="#level-3" class="anchor"></a>level 3</h3>
#> <p>::: boo</p>
#> </div>
#> <div id="level2" class="section level2">
#> <h2 class="hasAnchor">
#> <a href="#level2" class="anchor"></a>level2</h2>
#> <p>:::</p>
#> </div>
#> </div>
x <- read_html(pkgdown:::markdown(file.path(tmp)))
xml_find_all(x, ".//div[contains(@class, 'boo')]")
#> {xml_nodeset (0)}
xml_text(x) # fenced divs unprocessed :'(
#> [1] "\n\nTest\n\n\nlevel 3\n::: boo\n\n\n\nlevel2\n:::\n\n"

Created on 2020-08-20 by the reprex package (v0.3.0)

New version

# remotes::install_github("r-lib/pkgdown@1990820")
remotes::install_github("zkamvar/pkgdown@c36b884")
#> Using github PAT from envvar GITHUB_PAT
#> Downloading GitHub repo zkamvar/pkgdown@c36b884
#> 
#>      checking for file ‘/tmp/RtmpzHjXzm/remotes3baf1f04e3d/zkamvar-pkgdown-c36b884/DESCRIPTION’ ...  ✔  checking for file ‘/tmp/RtmpzHjXzm/remotes3baf1f04e3d/zkamvar-pkgdown-c36b884/DESCRIPTION’
#>   ─  preparing ‘pkgdown’:
#>      checking DESCRIPTION meta-information ...  ✔  checking DESCRIPTION meta-information
#>   ─  installing the package to process help pages
#>   ─  checking for LF line-endings in source and make files and shell scripts (2.3s)
#>   ─  checking for empty or unneeded directories
#>   ─  building ‘pkgdown_1.5.1.9000.tar.gz’
#>      
#> 
#> Installing package into '/home/zhian/R/library'
#> (as 'lib' is unspecified)
library("pkgdown")
library("xml2")

tmp <- file.path(tempdir(), "test")
cat('# Test
### level 3

<div class="boo">

## level2

</div>
', file = tmp)

cat(pkgdown:::markdown(file.path(tmp)))
#> <div id="test" class="section level1">
#> <h1 class="hasAnchor">
#> <a href="#test" class="anchor"></a>Test</h1>
#> <div id="level-3" class="section level3">
#> <h3 class="hasAnchor">
#> <a href="#level-3" class="anchor"></a>level 3</h3>
#> </div>
#> <div id="level2" class="section level2 boo">
#> <h2 class="hasAnchor">
#> <a href="#level2" class="anchor"></a>level2</h2>
#> </div>
#> </div>
x <- read_html(pkgdown:::markdown(file.path(tmp)))
xml_find_all(x, ".//div[contains(@class, 'boo')]")
#> {xml_nodeset (1)}
#> [1] <div id="level2" class="section level2 boo">\n<h2 class="hasAnchor">\n<a  ...

Created on 2020-08-20 by the reprex package (v0.3.0)

Fenced divs are also processed

# remotes::install_github("r-lib/pkgdown@eae56f0")
remotes::install_github("zkamvar/pkgdown@c36b884")
library("pkgdown")
library("xml2")

tmp <- file.path(tempdir(), "test")
cat('# Test
### level 3

::: boo

## level2

:::
', file = tmp)

cat(pkgdown:::markdown(file.path(tmp)))
#> <div id="test" class="section level1">
#> <h1 class="hasAnchor">
#> <a href="#test" class="anchor"></a>Test</h1>
#> <div id="level-3" class="section level3">
#> <h3 class="hasAnchor">
#> <a href="#level-3" class="anchor"></a>level 3</h3>
#> </div>
#> <div id="level2" class="section level2 boo">
#> <h2 class="hasAnchor">
#> <a href="#level2" class="anchor"></a>level2</h2>
#> </div>
#> </div>
x <- read_html(pkgdown:::markdown(file.path(tmp)))
xml_find_all(x, ".//div[contains(@class, 'boo')]")
#> {xml_nodeset (1)}
#> [1] <div id="level2" class="section level2 boo">\n<h2 class="hasAnchor">\n<a  ...
xml_text(x) # fenced divs work :)
#> [1] "\n\nTest\n\n\nlevel 3\n\n\n\nlevel2\n\n"

Created on 2020-08-20 by the reprex package (v0.3.0)

zkamvar picture zkamvar on 20 Aug 2020

Please start by explaining what the problem is. i.e. what is the problem with this?

tmp <- tempfile()
cat('# h1
### h3
<div class="boo">
## h2
</div>
', file = tmp)

cat(pkgdown:::markdown(file.path(tmp)))
#> <div id="h1" class="section level1">
#> <h1 class="hasAnchor">
#> <a href="#h1" class="anchor"></a>h1</h1>
#> <div id="h3" class="section level3">
#> <h3 class="hasAnchor">
#> <a href="#h3" class="anchor"></a>h3</h3>
#> </div>
#> <div id="h2" class="section level2 boo">
#> <h2 class="hasAnchor">
#> <a href="#h2" class="anchor"></a>h2</h2>
#> </div>
#> </div>

Created on 2020-08-20 by the reprex package (v0.3.0.9001)

hadley picture hadley on 21 Aug 2020

The issue is that the div contents are being stripped and placed below the container:

  tmp <- tempfile()
cat('# h1
### h3
<div class="boo">
## h2
</div>
', file = tmp)

cat(pkgdown:::markdown(file.path(tmp)))
#> <div id="h1" class="section level1">
#> <h1 class="hasAnchor">
#> <a href="#h1" class="anchor"></a>h1</h1>
#> <div id="h3" class="section level3">
#> <h3 class="hasAnchor">
#> <a href="#h3" class="anchor"></a>h3</h3>
#> <div class="boo">
#> </div>
#> <div id="h2" class="section level2">
#> <h2 class="hasAnchor">
#> <a href="#h2" class="anchor"></a>h2</h2>
#> </div>
#> </div>
#> </div>

Created on 2020-08-20 by the reprex package (v0.3.0)

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS Catalina 10.15.6      
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/Los_Angeles         
#>  date     2020-08-20                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version  date       lib source        
#>  assertthat    0.2.1    2019-03-21 [1] CRAN (R 4.0.0)
#>  backports     1.1.8    2020-06-17 [1] CRAN (R 4.0.0)
#>  callr         3.4.3    2020-03-28 [1] CRAN (R 4.0.0)
#>  cli           2.0.2    2020-02-28 [1] CRAN (R 4.0.0)
#>  crayon        1.3.4    2017-09-16 [1] CRAN (R 4.0.0)
#>  desc          1.2.0    2018-05-01 [1] CRAN (R 4.0.0)
#>  devtools      2.3.1    2020-07-21 [1] CRAN (R 4.0.2)
#>  digest        0.6.25   2020-02-23 [1] CRAN (R 4.0.0)
#>  ellipsis      0.3.1    2020-05-15 [1] CRAN (R 4.0.0)
#>  evaluate      0.14     2019-05-28 [1] CRAN (R 4.0.0)
#>  fansi         0.4.1    2020-01-08 [1] CRAN (R 4.0.0)
#>  fs            1.5.0    2020-07-31 [1] CRAN (R 4.0.2)
#>  glue          1.4.1    2020-05-13 [1] CRAN (R 4.0.0)
#>  highr         0.8      2019-03-20 [1] CRAN (R 4.0.0)
#>  htmltools     0.5.0    2020-06-16 [1] CRAN (R 4.0.0)
#>  knitr         1.29     2020-06-23 [1] CRAN (R 4.0.0)
#>  lifecycle     0.2.0    2020-03-06 [1] CRAN (R 4.0.0)
#>  magrittr      1.5      2014-11-22 [1] CRAN (R 4.0.0)
#>  MASS          7.3-51.6 2020-04-26 [2] CRAN (R 4.0.2)
#>  memoise       1.1.0    2017-04-21 [1] CRAN (R 4.0.0)
#>  pillar        1.4.6    2020-07-10 [1] CRAN (R 4.0.2)
#>  pkgbuild      1.1.0    2020-07-13 [1] CRAN (R 4.0.2)
#>  pkgconfig     2.0.3    2019-09-22 [1] CRAN (R 4.0.0)
#>  pkgdown       1.5.1    2020-04-09 [1] CRAN (R 4.0.0)
#>  pkgload       1.1.0    2020-05-29 [1] CRAN (R 4.0.0)
#>  prettyunits   1.1.1    2020-01-24 [1] CRAN (R 4.0.0)
#>  processx      3.4.3    2020-07-05 [1] CRAN (R 4.0.2)
#>  ps            1.3.3    2020-05-08 [1] CRAN (R 4.0.0)
#>  purrr         0.3.4    2020-04-17 [1] CRAN (R 4.0.0)
#>  R6            2.4.1    2019-11-12 [1] CRAN (R 4.0.0)
#>  remotes       2.2.0    2020-07-21 [1] CRAN (R 4.0.2)
#>  rlang         0.4.7    2020-07-09 [1] CRAN (R 4.0.2)
#>  rmarkdown     2.3      2020-06-18 [1] CRAN (R 4.0.0)
#>  rprojroot     1.3-2    2018-01-03 [1] CRAN (R 4.0.0)
#>  sessioninfo   1.1.1    2018-11-05 [1] CRAN (R 4.0.0)
#>  stringi       1.4.6    2020-02-17 [1] CRAN (R 4.0.0)
#>  stringr       1.4.0    2019-02-10 [1] CRAN (R 4.0.0)
#>  testthat      2.3.2    2020-03-02 [1] CRAN (R 4.0.0)
#>  tibble        3.0.3    2020-07-10 [1] CRAN (R 4.0.2)
#>  usethis       1.6.1    2020-04-29 [1] CRAN (R 4.0.0)
#>  vctrs         0.3.2    2020-07-15 [1] CRAN (R 4.0.2)
#>  withr         2.2.0    2020-04-20 [1] CRAN (R 4.0.0)
#>  xfun          0.16     2020-07-24 [1] CRAN (R 4.0.2)
#>  xml2          1.3.2    2020-04-23 [1] CRAN (R 4.0.0)
#>  yaml          2.2.1    2020-02-01 [1] CRAN (R 4.0.0)
#> 
#> [1] /Users/zhian/R
#> [2] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

Specifically this part:

#> <div class="boo">
#> </div>
#> <div id="h2" class="section level2">
#> <h2 class="hasAnchor">
#> <a href="#h2" class="anchor"></a>h2</h2>
#> </div>
zkamvar picture zkamvar on 21 Aug 2020

But that is fine in my example, right?

#> <div id="h2" class="section level2 boo">
#> <h2 class="hasAnchor">
#> <a href="#h2" class="anchor"></a>h2</h2>
#> </div>
hadley picture hadley on 21 Aug 2020

Are the other parts related to the problem? Does this still give the incorrect output for you?

tmp <- tempfile()
cat('<div class="boo">\n## h2\n</div>', file = tmp)
cat(pkgdown:::markdown(file.path(tmp)))
#> <div id="h2" class="section level2 boo">
#> <h2 class="hasAnchor">
#> <a href="#h2" class="anchor"></a>h2</h2>
#> </div>

Created on 2020-08-21 by the reprex package (v0.3.0.9001)

hadley picture hadley on 21 Aug 2020

Yes, the other parts are related to the problem. FWIW, I can submit a PR that will fix this issue.

I get the correct result with your minimal reprex:

tmp <- tempfile()
cat('<div class="boo">\n## h2\n</div>', file = tmp)
cat(pkgdown:::markdown(file.path(tmp)))
#> <div class="boo">
#> <div id="h2" class="section level2">
#> <h2 class="hasAnchor">
#> <a href="#h2" class="anchor"></a>h2</h2>
#> </div>
#> </div>

Created on 2020-08-21 by the reprex package (v0.3.0)

Session info

devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       Ubuntu 18.04.5 LTS          
#>  system   x86_64, linux-gnu           
#>  ui       X11                         
#>  language en_US:en                    
#>  collate  en_US.UTF-8                 
#>  ctype    en_US.UTF-8                 
#>  tz       America/Los_Angeles         
#>  date     2020-08-21                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version     date       lib source                         
#>  assertthat    0.2.1       2019-03-21 [1] CRAN (R 4.0.0)                 
#>  backports     1.1.8       2020-06-17 [1] CRAN (R 4.0.0)                 
#>  callr         3.4.3       2020-03-28 [1] CRAN (R 4.0.0)                 
#>  cli           2.0.2       2020-02-28 [1] CRAN (R 4.0.0)                 
#>  crayon        1.3.4.9000  2020-08-18 [1] Github (r-lib/crayon@6b3f0c6)  
#>  desc          1.2.0       2018-05-01 [1] CRAN (R 4.0.0)                 
#>  devtools      2.3.1       2020-07-21 [1] CRAN (R 4.0.2)                 
#>  digest        0.6.25      2020-02-23 [1] CRAN (R 4.0.0)                 
#>  downlit       0.0.0.9000  2020-08-17 [1] Github (r-lib/downlit@ed969d0) 
#>  ellipsis      0.3.1       2020-05-15 [1] CRAN (R 4.0.0)                 
#>  evaluate      0.14        2019-05-28 [1] CRAN (R 4.0.0)                 
#>  fansi         0.4.1       2020-01-08 [1] RSPM (R 4.0.0)                 
#>  fs            1.5.0       2020-07-31 [1] CRAN (R 4.0.2)                 
#>  glue          1.4.1       2020-05-13 [1] CRAN (R 4.0.0)                 
#>  highr         0.8         2019-03-20 [1] CRAN (R 4.0.0)                 
#>  htmltools     0.5.0       2020-06-16 [1] CRAN (R 4.0.0)                 
#>  knitr         1.29        2020-06-23 [1] CRAN (R 4.0.0)                 
#>  lifecycle     0.2.0       2020-03-06 [1] CRAN (R 4.0.0)                 
#>  magrittr      1.5         2014-11-22 [1] CRAN (R 4.0.0)                 
#>  memoise       1.1.0       2017-04-21 [1] CRAN (R 4.0.0)                 
#>  pillar        1.4.6       2020-07-10 [1] CRAN (R 4.0.2)                 
#>  pkgbuild      1.1.0       2020-07-13 [1] CRAN (R 4.0.2)                 
#>  pkgconfig     2.0.3       2019-09-22 [1] CRAN (R 4.0.0)                 
#>  pkgdown       1.5.1.9000  2020-08-21 [1] Github (r-lib/pkgdown@add5b7d) 
#>  pkgload       1.1.0       2020-05-29 [1] CRAN (R 4.0.0)                 
#>  prettyunits   1.1.1       2020-01-24 [1] CRAN (R 4.0.0)                 
#>  processx      3.4.3       2020-07-05 [1] CRAN (R 4.0.2)                 
#>  ps            1.3.4       2020-08-11 [1] CRAN (R 4.0.2)                 
#>  purrr         0.3.4       2020-04-17 [1] CRAN (R 4.0.0)                 
#>  R6            2.4.1       2019-11-12 [1] CRAN (R 4.0.0)                 
#>  remotes       2.2.0       2020-07-21 [1] CRAN (R 4.0.2)                 
#>  rlang         0.4.7       2020-07-09 [1] CRAN (R 4.0.2)                 
#>  rmarkdown     2.3         2020-06-18 [1] CRAN (R 4.0.1)                 
#>  rprojroot     1.3-2       2018-01-03 [1] CRAN (R 4.0.0)                 
#>  sessioninfo   1.1.1       2018-11-05 [1] CRAN (R 4.0.0)                 
#>  stringi       1.4.6       2020-02-17 [1] CRAN (R 4.0.0)                 
#>  stringr       1.4.0       2019-02-10 [1] CRAN (R 4.0.0)                 
#>  testthat      2.99.0.9000 2020-08-14 [1] Github (r-lib/testthat@9e643d8)
#>  tibble        3.0.3       2020-07-10 [1] CRAN (R 4.0.2)                 
#>  usethis       1.6.1       2020-04-29 [1] CRAN (R 4.0.0)                 
#>  vctrs         0.3.2       2020-07-15 [1] CRAN (R 4.0.2)                 
#>  withr         2.2.0       2020-04-20 [1] CRAN (R 4.0.0)                 
#>  xfun          0.16        2020-07-24 [1] CRAN (R 4.0.2)                 
#>  xml2          1.3.2       2020-04-23 [1] CRAN (R 4.0.0)                 
#>  yaml          2.2.1       2020-02-01 [1] CRAN (R 4.0.0)                 
#> 
#> [1] /home/zhian/R/library
#> [2] /usr/local/lib/R/site-library
#> [3] /usr/lib/R/site-library
#> [4] /usr/lib/R/library

but if I add ### h3\n before the div tag, it fails:

tmp <- tempfile()
cat('### h3\n<div class="boo">\n## h2\n</div>', file = tmp)
cat(pkgdown:::markdown(file.path(tmp)))
#> <div id="h3" class="section level3">
#> <h3 class="hasAnchor">
#> <a href="#h3" class="anchor"></a>h3</h3>
#> <div class="boo">
#> </div>
#> <div id="h2" class="section level2">
#> <h2 class="hasAnchor">
#> <a href="#h2" class="anchor"></a>h2</h2>
#> </div>
#> </div>

Created on 2020-08-21 by the reprex package (v0.3.0)

As I detailed in https://github.com/r-lib/pkgdown/issues/1381#issue-681296198, it also affects pandoc's fenced divs:

tmp <- tempfile()
cat('::: boo\n## h2\n:::', file = tmp)
cat(pkgdown:::markdown(file.path(tmp)))
#> 
#> <p>::: boo</p>
#> <div id="h2" class="section level2">
#> <h2 class="hasAnchor">
#> <a href="#h2" class="anchor"></a>h2</h2>
#> <p>:::</p>
#> </div>

Created on 2020-08-21 by the reprex package (v0.3.0)

zkamvar picture zkamvar on 21 Aug 2020

Essentially, I would like to replace the deprecated markdown_github with markdown. I have a fork to show that this works with tests.

zkamvar picture zkamvar on 25 Aug 2020

You have told me multiple times that you have a fork that works but I want to understand the underlying problem and I am having a great deal of difficulty.

hadley picture hadley on 25 Aug 2020

Thank you for being patient with me. I hope this succinctly states the problem:

What I expect

If I have markdown like this:

### h3
<div class="a">
## h2
</div>

I expect the HTML output to have a similar structure (contents of the ## h2 section inherit the "a" class, lines 4-8):

<div id="h3" class="section level3">
<h3 class="hasAnchor">
<a href="#h3" class="anchor"></a>h3</h3>
</div>
<div id="h2" class="section level2 a">
<h2 class="hasAnchor">
<a href="#h2" class="anchor"></a>h2</h2>
</div>

What I actually see

With the current HEAD of pkgdown (0f8b6b2d9b5c0bac7d2c990fd4ad34e8d8d19471), I see the following issues (which I've highlighted with HTML comments)

  1. my "a" class div tag has its contents removed (lines 4-5 of the output below)
  2. the ## h2 section is now a child of the ### h3 section
tmp <- tempfile()
cat('### h3\n<div class="a">\n## h2\n</div>', file = tmp)
cat(pkgdown:::markdown(file.path(tmp)))

md5-535c8a1ede44c794dffb5d7591b84e19




md5-d574bb6678a0f8e1fa04b23fcf1828f1


</div>          <!-- 2. h3 ends here -->

Created on 2020-08-25 by the reprex package (v0.3.0), with modifications for clarity

Thank you again for taking the time to consider this. I apologize for complicating things earlier.

zkamvar picture zkamvar on 25 Aug 2020
👍1

Just chiming in to share my thoughts on this one.

I think this is due to some pandoc extension usage. Here my try on this using your example.

# temp files to write into
tmp_in <- tempfile(fileext = ".md")
tmp_out <- tempfile(fileext = ".html")
cat('### h3\n<div class="a">\n## h2\n</div>', file = tmp_in)

# insuring pandoc 2.7.3 version
rmarkdown::find_pandoc(version = "2.7.3")
#> $version
#> [1] '2.7.3'
#> 
#> $dir
#> [1] "C:/Program Files/RStudio/bin/pandoc"

With extension markdown_in_html_blocks, divs are seen as raw HTML. We see that in the native output.

rmarkdown::pandoc_convert(tmp_in, from = "markdown_github+markdown_in_html_blocks", 
                          to = 'native', output = tmp_out, options = "--section-divs")
xfun::file_string(tmp_out)
#> [Header 3 ("h3",[],[]) [Str "h3"]
#> ,RawBlock (Format "html") "<div class=\"a\">"
#> ,Header 2 ("h2",[],[]) [Str "h2"]
#> ,RawBlock (Format "html") "</div>"]

That causes the rendering you see. I believe the raw html is associated with the h3 header part. I think the first </div> is just the h3 part closing. The raw html </div> on last line is put in last line too here.

rmarkdown::pandoc_convert(tmp_in, from = "markdown_github+markdown_in_html_blocks", 
                          to = 'html', output = tmp_out, options = "--section-divs")
xfun::file_string(tmp_out)
#> <div id="h3" class="section level3">
#> <h3>h3</h3>
#> <div class="a">
#> </div>
#> <div id="h2" class="section level2">
#> <h2>h2</h2>
#> </div>
#> </div>

If you use extension native_divs instead (which is almost the same), html divs are seen as Pandoc Divs, and that allows pandoc to better identify the div parts. See in native output, the div of class a is no more associated with header 3.

rmarkdown::pandoc_convert(tmp_in, from = "markdown_github+native_divs", 
                          to = 'native', output = tmp_out, options = "--section-divs")
xfun::file_string(tmp_out)
#> [Header 3 ("h3",[],[]) [Str "h3"]
#> ,Div ("",["a"],[])
#>  [Header 2 ("h2",[],[]) [Str "h2"]]]

That got us the expected formating because Pandoc Divs are translated to HTML divs correctly opening and closing.

rmarkdown::pandoc_convert(tmp_in, from = "markdown_github+native_divs", 
                          to = 'html', output = tmp_out, options = "--section-divs")
xfun::file_string(tmp_out)
#> <div id="h3" class="section level3">
#> <h3>h3</h3>
#> <div class="a">
#> <h2 id="h2">h2</h2>
#> </div>
#> </div>

markdown format you suggest include native_divs by default in 2.7.3 so that works. Also let’s note that this seems to have change in Pandoc 2.10.1, because we get the correct output with small change as the class a is added to the header div. (Change in pandoc 2.8 I think).

# insuring pandoc 2.7.3 version
rmarkdown::find_pandoc(version = "2.10.1")
#> $version
#> [1] '2.10.1'
#> 
#> $dir
#> [1] "C:/Users/chris/scoop/shims"
cat(pkgdown:::markdown(tmp_in))
#> 
#> <div id="h3" class="section level3">
#> <h3 class="hasAnchor">
#> <a href="#h3" class="anchor"></a>h3</h3>
#> </div>
#> <div id="h2" class="section level2 a">
#> <h2 class="hasAnchor">
#> <a href="#h2" class="anchor"></a>h2</h2>
#> </div>

Created on 2020-08-25 by the reprex package (v0.3.0.9001)

Hope it helps understand what is going on.

cderv picture cderv on 25 Aug 2020
👍1

markdown format you suggest include native_divs by default in 2.7.3 so that works. Also let’s note that this seems to have change in Pandoc 2.10.1, because we get the correct output with small change as the class a is added to the header div. (Change in pandoc 2.8 I think).

The change was implemented after 2.9.2.1 as that's the version I'm using.

zkamvar picture zkamvar on 26 Aug 2020

@cderv so do you think we just need to add native_divs to https://github.com/r-lib/pkgdown/blob/master/R/markdown.R#L6-L7? Or should we also remove markdown_in_html_blocks?

hadley picture hadley on 26 Aug 2020
👍1

TL;DR

native_divs shoud be added in addtion of markdown_in_html_blocks.

Details

We still need markdown_in_html_blocks so that cases like https://github.com/r-lib/pkgdown/issues/1220 works correclty. (i.e markdown syntax in any html tags). With only +native_divs enable, only the markdown between html <div> would be parsed. So #1220 wouldn't work.

So I would add +native_divs if we want to support cases like the one in this current issue.

It is really --sections-divs here which causes the interaction and requires +native_divs. We have to remember that the behavior changed between pandoc 2.7.3 and pandoc 2.9.2.1 (I think 2.8 that rewrote MakeSections)

In pandoc 2.7.3, the h2 header is correctly parsed but no section div is created for h2 header. With pandoc 2.9.2.1 and new --sections_divs behavior, the div of class a is correctly seen as a pandoc div and merge with the h2 section divs.
Example:

md <- tempfile(fileext = ".md")
html_out <- tempfile(fileext = ".html")
brio::write_lines(
  glue::glue("
  ### h3
  <div class=\"a\">
  ## h2
  </div>
  "), md)
xfun::file_string(md)
#> ### h3
#> <div class="a">
#> ## h2
#> </div>

convert <- function(file, from_extensions = NULL, output = NULL) {
  from <- paste(c("markdown_github",from_extensions), collapse = "") 
  rmarkdown::pandoc_convert(file, from = from, to = "html", 
                            options = "--section-divs",
                            output = output)
}

rmarkdown::find_pandoc(version = "2.7.3")
#> $version
#> [1] '2.7.3'
#> 
#> $dir
#> [1] "C:/Program Files/RStudio/bin/pandoc"
convert(md, "+markdown_in_html_blocks+native_divs", output = html_out)
xfun::file_string(html_out)
#> <div id="h3" class="section level3">
#> <h3>h3</h3>
#> <div class="a">
#> <h2 id="h2">h2</h2>
#> </div>
#> </div>
rmarkdown::find_pandoc(version = "2.9.2.1")
#> $version
#> [1] '2.9.2.1'
#> 
#> $dir
#> [1] "C:/Users/chris/scoop/shims"
convert(md, "+markdown_in_html_blocks+native_divs", output= html_out)
xfun::file_string(html_out)
#> <div id="h3" class="section level3">
#> <h3>h3</h3>
#> </div>
#> <div id="h2" class="section level2 a">
#> <h2>h2</h2>
#> </div>

Created on 2020-08-27 by the reprex package (v0.3.0)

Other evolution with Pandoc 2.10.1: it works ok out of the box with markdown_github because native_divs has been added by default in the format. (Part of new rendering for gfm and commonmark in https://github.com/jgm/pandoc/commit/d6b7b1dc772249e9a1b56bcdd4ae816cc54edb51#diff-12aef9fe883f972b3123118db76fe871R248)

We may need to start testing several pandoc versions to check everything is fine, and decide which pandoc version pkgdown really work well with; The [expected behavior described in https://github.com/r-lib/pkgdown/issues/1381#issuecomment-680076124 is not for pandoc 2.7.3 for example I think

cderv picture cderv on 27 Aug 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

seanhardison1 picture seanhardison1  ·  5Comments
ycphs picture ycphs  ·  6Comments
mjsteinbaugh picture mjsteinbaugh  ·  7Comments
thierrygosselin picture thierrygosselin  ·  9Comments
Anirban166 picture Anirban166  ·  8Comments