Pkgdown: Reduce run-time dependencies

[MichaelChirico]

It seems to me the DESCRIPTION is currently quite bloated relative to actual use in the package. Is it OK to move packages to Suggests as much as possible?

According to NAMESPACE dependencies are the following:

import(fs)
import(rlang)
importFrom(MASS,addterm)
importFrom(R6,R6Class)
importFrom(magrittr,"%>%")
importFrom(memoise,memoise)
importFrom(roxygen2,roxygenise)
importFrom(utils,installed.packages)

Which means the following Imports can be moved to Suggests:

• callr (>= 2.0.2)
• cli
• crayon
• desc
• digest
• evaluate
• fansi
• highlight
• httr
• openssl
• pkgload (>= 1.0.2)
• purrr
• processx
• rematch2
• rmarkdown (>= 1.1.9007)
• rsconnect
• rstudioapi
• tibble
• tools
• whisker
• withr
• xml2 (>= 1.1.1) # <-- also comes with a system dependency: libxml
• yaml

It also seems like roxygen2::roxygenise is imported without being used:

grep -r "roxygenise" .
./R/pkgdown.R:#' @importFrom roxygen2 roxygenise
./NAMESPACE:importFrom(roxygen2,roxygenise)

So that roxygen2 could be but in Suggests as well.

memoise::memoise is used twice; I don't know the internals well but at a glance it looks like crucial functionality so I'd keep in Imports.

Am happy to file a PR; filing the issue here first to discuss.

[hadley]

Could you explain a bit more as to why this would help you? All of those conditional imports are probably used on a critical path for some users, so we'll need to carefully analyse to check that they aren't actually needed in common cases.

I also don't see the problem with the MASS dependency, since every R install will already have that?

[MichaelChirico]

Retracted about MASS, I didn't see it in the core library so I assumed it's add-on.

The concern is about how to get to a minimal CI build of a pkgdown site, e.g. in case of data.table with no dependencies (and knitr in Suggests), the current list means 60 new packages and a new system dependency. It's possible to separate this to its own stage of CI, so it's not sine qua non.

Could you elaborate on "probably on a critical path"? The move to Suggests shouldn't have any immediate impact here? Since Imports->Suggests is a "downgrade".

[hadley]

Re-reading your comment, I think your analysis of what packages are required is based on a pattern that we don't follow — I think you are assuming that anything not in the NAMESPACE (i.e. referred to with :: in code) can be moved to suggests. But that's not something we do — we use namespace imports only to reduce the use of ::, not as a signal of whether or not a package must be installed in order for the package to work. That means we can't just move them all into Suggests — we need to analyse each to figure out where it's used, and whether or not common usage of the package assumes that it's installed. I think that analysis will be straightforward for many packages (e.g. I just noticed that rsconnect is in imports, where it definitely should be in suggests, and the code is already written in such a way to support that), but it will take some time.

The high number of dependencies does not currently cause us (or many others) any pain since we cache package installs on our CI system, so while I agree that we should work to reduce them, and I'll take look this week, it's not a high priority issue for us. (i.e. PRs would be appreciated/welcomed)

[hadley]

# devtools::install_github("jimhester/itdepends")
library(itdepends)
dep_plot_time("pkgdown")

dep_plot_size("pkgdown")

^{Created on 2019-08-27 by the reprex package (v0.3.0)}

So the worst offenders are stringi (not sure what's pulling that in), fs (which we rely on for safe file operations), xml2 (needed for article transformation), highlight (used for syntax highlighting in examples), Rcpp (used by multiple dependencies), roxygen2 (which we can probably make optional after #1079), then there's a lot of lower-cost packages.

[MichaelChirico]

we use namespace imports only to reduce the use of ::, not as a signal of whether or not a package must be installed in order for the package to work.

I see... agree it's much more burdensome then to figure out what's actually Imports v Suggests then. And indeed rsconnect was the major one I saw that seems like a clear Suggests, I don't see requireNamespace being conditioned for any of the others (which doesn't necessarily say that none of the other packages _couldn't_ be moved to Suggests, just that new branches would be required to move any of the others).

The plots provided are excellent! Will take another brief look today but I'm afraid it will take someone much more familiar with package internals to separate true-Imports from convenience-Imports.

[MichaelChirico]

Hopefully this can help you out -- here's something like a dependency tree for pkgdown:

https://gist.github.com/MichaelChirico/bbe82e7ef6e5e0898a1061127fc6f11b

From script here:

https://gist.github.com/MichaelChirico/b43c43b012f0a90fac2a2925a69bb555

library(data.table)
tree = fread('https://gist.githubusercontent.com/MichaelChirico/bbe82e7ef6e5e0898a1061127fc6f11b/raw/602b270f23389f5d9b3de20d6e53b1540b0f8d92/pkgdown_function_tree')
tree[sample(.N, 6L)]
#          parent_function child_function child_package
# 1:   as_data.tag_section        as_html       pkgdown
# 2:     navbar_components     print_yaml       pkgdown
# 3: topic_index_installed        as.list          base
# 4:           build_cname        is.null          base
# 5:          method_usage        as_html       pkgdown
# 6:   is_non_pkgdown_site              c          base

parent_function are functions in asNamespace('pkgdown'). child_function is roughly all the functions called within parent_function's body(). child_package is the package of child_function. Done with rough regex so not exact.

Here's the split by child_package for the packages listed as dependencies:

$rlang
            parent_function          child_function
 1:     article_index_local            is_installed
 2:     article_index_local               set_names
 3:    article_index_remote               set_names
 4:    as_data.tag_examples               child_env
 5:         build_reference            is_installed
 6:             context_get                 env_has
 7:             context_get                 env_get
 8:             context_get                   abort
 9:             context_set                 env_has
10:             context_set                 env_get
11:             context_set                 env_has
12:             context_set              env_unbind
13:             context_set                env_bind
14:             context_set                    `:=`
15:                   defer                   enquo
16:                   defer                    expr
17:                   defer        rlang::eval_tidy
18:                   defer               eval_bare
19:         devtools_loaded                 env_has
20:  extract_package_attach                 is_call
21:  extract_package_attach                 is_call
22:  extract_package_attach rlang::call_standardise
23:  extract_package_attach                 is_true
24: extract_package_attach_                  enexpr
25:             find_rdname                    warn
26:    find_reexport_source             ns_env_name
27:    find_reexport_source                 get_env
28:                fun_info                 is_call
29:                fun_info               call_name
30:                fun_name              parse_expr
31:                fun_name               is_symbol
32:                fun_name                 is_call
33:               href_expr               is_symbol
34:               href_expr                 is_call
35:               href_expr                 is_call
36:               href_expr               is_symbol
37:               href_expr        call_standardise
38:               href_expr                 is_call
39:               href_expr               is_symbol
40:               href_expr               is_string
41:             href_tokens               rep_along
42:        href_topic_local                  ns_env
43:        href_topic_local                 env_has
44:        href_topic_local                 env_get
45:               match_fun              parse_expr
46:              package_rd               set_names
47:              parse_opts               child_env
48:              parse_opts               child_env
49:             parse_usage             parse_exprs
50:      rmarkdown_template                     env
51:          tweak_pre_node               rep_along
52:              usage_type               is_symbol
53:              usage_type                 is_call
            parent_function          child_function

$fs
          parent_function   child_function
  1:  article_index_local             path
  2:  article_index_local       dir_exists
  3:  article_index_local           dir_ls
  4:  article_index_local         path_rel
  5:  article_index_local        path_real
 ---                                      
196:        tutorial_info  path_ext_remove
197:        tutorial_info        path_file
198: tweak_rmarkdown_html is_absolute_path
199: tweak_rmarkdown_html         path_rel
200:   write_if_different         path_abs

$tools
        parent_function            child_function
1: article_index_remote tools::file_path_sans_ext
2: data_reference_topic tools::file_path_sans_ext
3:              rd_file    tools::loadPkgRdMacros
4:              rd_file           tools::parse_Rd
5:              rd_file    tools::loadPkgRdMacros
6:              rd_file           tools::parse_Rd
7:              rd_file           tools::parse_Rd
8:              rd_text           tools::parse_Rd

$purrr
           parent_function     child_function
  1: as_data.tag_arguments        purrr::keep
  2: as_data.tag_arguments         purrr::map
  3:  as_data.tag_examples     purrr::map_lgl
  4:  as_data.tag_examples     purrr::map_lgl
  5:  as_data.tag_examples         purrr::map
 ---                                         
151:         usage_code.Rd      purrr::detect
152:        usage_code.tag purrr::flatten_chr
153:        usage_code.tag         purrr::map
154:  usage_code.tag_usage purrr::flatten_chr
155:  usage_code.tag_usage         purrr::map

$xml2
         parent_function        child_function
 1:        autolink_html       xml2::read_html
 2:        autolink_html      xml2::write_html
 3:       badges_extract  xml2::xml_find_first
 4:       badges_extract    xml2::xml_find_all
 5:       badges_extract  xml2::xml_find_first
 6:       badges_extract        xml2::xml_text
 7:       badges_extract    xml2::xml_children
 8:       badges_extract        xml2::xml_name
 9:       badges_extract      xml2::xml_remove
10:  badges_extract_text       xml2::read_html
11:        build_sitemap        xml2::read_xml
12:        build_sitemap   xml2::xml_add_child
13:        build_sitemap       xml2::write_xml
14:            cname_url       xml2::url_parse
15:            data_news       xml2::read_html
16:            data_news    xml2::xml_find_all
17:            data_news  xml2::xml_find_first
18:            data_news        xml2::xml_text
19:            data_news        xml2::xml_attr
20:       find_qualifier    xml2::xml_find_all
21:       find_qualifier        xml2::xml_name
22:       find_qualifier        xml2::xml_text
23:       find_qualifier        xml2::xml_name
24:       find_qualifier        xml2::xml_text
25:             markdown       xml2::read_html
26:             markdown  xml2::xml_find_first
27:             markdown        xml2::xml_text
28:             markdown      xml2::xml_remove
29:             markdown  xml2::xml_find_first
30:             markdown      xml2::write_html
31:        tweak_anchors    xml2::xml_find_all
32:        tweak_anchors    xml2::xml_find_all
33:        tweak_anchors        xml2::xml_attr
34:        tweak_anchors        xml2::xml_attr
35:        tweak_anchors    xml2::xml_find_all
36:        tweak_anchors        xml2::xml_attr
37:        tweak_anchors        xml2::xml_attr
38:        tweak_anchors  xml2::xml_find_first
39:        tweak_anchors        xml2::xml_name
40:        tweak_anchors    xml2::xml_contents
41:        tweak_anchors        xml2::xml_attr
42:        tweak_anchors xml2::xml_add_sibling
43:        tweak_anchors    xml2::xml_contents
44:           tweak_code    xml2::xml_find_all
45:           tweak_code    xml2::xml_find_all
46:   tweak_code_nodeset        xml2::xml_text
47:   tweak_code_nodeset    xml2::xml_contents
48:   tweak_code_nodeset     xml2::xml_replace
49:  tweak_homepage_html       xml2::read_html
50:  tweak_homepage_html  xml2::xml_find_first
51:  tweak_homepage_html  xml2::xml_find_first
52:  tweak_homepage_html    xml2::xml_children
53:  tweak_homepage_html   xml2::xml_add_child
54:  tweak_homepage_html    xml2::xml_find_all
55:  tweak_homepage_html      xml2::xml_remove
56:  tweak_homepage_html  xml2::xml_find_first
57:  tweak_homepage_html      xml2::xml_remove
58:  tweak_homepage_html       xml2::read_html
59:  tweak_homepage_html  xml2::xml_find_first
60:  tweak_homepage_html     xml2::xml_replace
61:  tweak_homepage_html    xml2::xml_find_all
62:  tweak_homepage_html        xml2::xml_attr
63:  tweak_homepage_html        xml2::xml_attr
64:       tweak_md_links    xml2::xml_find_all
65:       tweak_md_links        xml2::xml_attr
66:       tweak_md_links       xml2::url_parse
67:   tweak_news_heading    xml2::xml_find_all
68:   tweak_news_heading    xml2::xml_set_attr
69:   tweak_news_heading       xml2::read_html
70:   tweak_news_heading    xml2::xml_find_all
71:   tweak_news_heading    xml2::xml_find_all
72:   tweak_news_heading   xml2::xml_add_child
73:       tweak_pre_node        xml2::xml_text
74:       tweak_pre_node    xml2::xml_find_all
75:       tweak_pre_node        xml2::xml_text
76:       tweak_pre_node    xml2::xml_contents
77:       tweak_pre_node     xml2::xml_replace
78: tweak_rmarkdown_html    xml2::xml_find_all
79: tweak_rmarkdown_html        xml2::xml_attr
80: tweak_rmarkdown_html    xml2::xml_find_all
81: tweak_rmarkdown_html        xml2::xml_attr
82:         tweak_tables    xml2::xml_find_all
83:         tweak_tables       xml2::xml_attrs
84:         tweak_tables       xml2::xml_attrs
85:          update_html       xml2::read_html
86:          update_html      xml2::write_html
87:             url_node        xml2::read_xml
         parent_function        child_function

$rmarkdown
           parent_function                     child_function
 1:          build_article       rmarkdown::yaml_front_matter
 2: build_rmarkdown_format           rmarkdown::html_document
 3:        build_site_meta          rmarkdown::pandoc_version
 4:               markdown        rmarkdown::pandoc_available
 5:               markdown        rmarkdown::pandoc_available
 6:               markdown          rmarkdown::pandoc_convert
 7:               markdown        rmarkdown::pandoc_available
 8:               markdown        rmarkdown::pandoc_available
 9:    render_navbar_links       rmarkdown::navbar_links_html
10:       render_rmarkdown                  rmarkdown::render
11:       render_rmarkdown rmarkdown::find_external_resources
12:      skip_if_no_pandoc        rmarkdown::pandoc_available
13:          tutorial_info       rmarkdown::yaml_front_matter

$cli
          parent_function child_function
 1:        build_articles           rule
 2:        build_favicons           rule
 3:            build_home           rule
 4:            build_news           rule
 5:       build_reference           rule
 6: build_reference_topic       cat_line
 7:      build_site_local           rule
 8:      build_site_local       cat_line
 9:      build_site_local       cat_line
10:      build_site_local           rule
11:         build_sitemap       cat_line
12:       build_tutorials           rule
13:    deploy_site_github           rule
14:    deploy_site_github           rule
15:    deploy_site_github           rule
16:    deploy_site_github       cat_line
17:    deploy_site_github       cat_line
18:    deploy_site_github           rule
19:          file_copy_to       cat_line
20:          github_clone           rule
21:           github_push           rule
22:           github_push           rule
23:             init_site           rule
24:          preview_site           rule
25:             render_md       cat_line
26:      render_rmarkdown       cat_line
27:      render_rmarkdown       cat_line
28:                  rule  cli::cat_rule
29:    write_if_different       cat_line
          parent_function child_function

$openssl
      parent_function         child_function
1:     build_favicons openssl::base64_encode
2: deploy_site_github openssl::base64_decode

$httr
    parent_function        child_function
 1:  build_favicons            httr::POST
 2:  build_favicons      httr::http_error
 3:  build_favicons         httr::content
 4:  build_favicons             httr::GET
 5:  build_favicons      httr::write_disk
 6:      fetch_yaml             httr::GET
 7:      fetch_yaml         httr::timeout
 8:      fetch_yaml httr::stop_for_status
 9:      fetch_yaml         httr::content
10:    pkg_timeline             httr::GET
11:    pkg_timeline      httr::http_error
12:    pkg_timeline         httr::content

$tibble
                 parent_function    child_function
 1:             build_news_multi    tibble::tibble
 2:  data_articles_index_section    tibble::tibble
 3:                    data_news    tibble::tibble
 4: data_reference_index_section    tibble::tibble
 5:                 find_article          has_name
 6:                  find_rdname          has_name
 7:                  licenses_db tibble::as_tibble
 8:                  licenses_db    tibble::tibble
 9:               package_topics    tibble::tibble
10:            package_tutorials    tibble::tibble
11:            package_vignettes    tibble::tibble
12:                  remote_urls          has_name
13:             select_vignettes    tibble::tibble
14:                  set_classes         set_class

$pkgload
   parent_function    child_function
1: build_reference pkgload::load_all

$crayon
        parent_function     child_function
1:  build_site_external  crayon::has_color
2:  build_site_external crayon::num_colors
3:             dst_path       crayon::blue
4: format_example_chunk  crayon::has_color
5: format_example_chunk crayon::num_colors
6:                 rule       crayon::bold
7:             src_path      crayon::green

$callr
       parent_function       child_function
1: build_site_external             callr::r
2:  deploy_site_github          callr::rcmd
3:    render_rmarkdown        callr::r_safe
4:    render_rmarkdown callr::rcmd_safe_env

$yaml
    parent_function       child_function
1:  build_site_meta           write_yaml
2:       fetch_yaml      yaml::yaml.load
3: print.print_yaml        yaml::as.yaml
4:        read_meta yaml::yaml.load_file
5:       write_yaml        yaml::as.yaml

$withr
        parent_function      child_function
1:   context_set_scoped               defer
2: format_example_chunk withr::with_options
3:          github_push            with_dir
4:    scoped_in_pkgdown               defer

$rematch2
     parent_function     child_function
1:      deploy_local rematch2::re_match
2:    find_qualifier rematch2::re_match
3:      news_version rematch2::re_match
4: parse_github_link rematch2::re_match

$digest
          parent_function child_function
 1:           digest_plot digest::digest
 2:           file_digest digest::digest
 3:            file_equal digest::digest
 4:            file_equal digest::digest
 5:       remote_metadata         digest
 6:             repo_link         digest
 7:         same_contents digest::digest
 8:         same_contents digest::digest
 9: topic_index_installed         digest
10:     topic_index_local         digest

$evaluate
        parent_function     child_function
1: format_example_chunk evaluate::evaluate

$processx
   parent_function child_function
1:             git  processx::run

$highlight
    parent_function           child_function
1:   highlight_text     highlight::highlight
2: pkgdown_renderer highlight::renderer_html

$whisker
   parent_function          child_function
1: render_template whisker::whisker.render

$fansi
    parent_function     child_function
1: replay_html.list fansi::sgr_to_html

$rstudioapi
    parent_function              child_function
1: rstudio_save_all          rstudioapi::hasFun
2: rstudio_save_all rstudioapi::documentSaveAll

$rsconnect
   parent_function         child_function
1:   tutorial_info rsconnect::deployments

$magrittr
   parent_function child_function
1:      tweak_code            not
2:      tweak_code            not

Hopefully that makes it easier for you to go through each dependency and glance at the parent_functions and see if all are critical.

[hadley]

BTW there's also itdepends::dep_locate()

[jayhesselberth]

roxygen2 imports stringi, so I think that dependency was eliminated by 5fdb2ba43d90f5daab5b51ce3efe985af1368db2.

[jangorecki]

@jayhesselberth probably not only roxygen2 was using stringi, installing pkgdown still installs stringi

Installing 51 packages: callr, cli, crayon, desc, digest, evaluate, fs, fansi, 
highlight, httr, magrittr, memoise, openssl, purrr, processx, R6, rematch2, 
rlang, rmarkdown, rstudioapi, tibble, whisker, withr, xml2, yaml, assertthat, 
rprojroot, Rcpp, curl, jsonlite, mime, askpass, ps, knitr, htmltools, base64enc, 
tinytex, xfun, stringr, pillar, pkgconfig, backports, sys, utf8, vctrs, ellipsis, 
glue, zeallot, highr, markdown, stringi

it was installing 59 packages before, so at least it got reduced by 8.

[Bisaloo]

pkgdown -> rmarkdown -> stringr -> stringi

(-> = imports)

[hadley]

Looks like rmarkdown only uses it in one place but it's also needed for knitr.

[hadley]

I'm happy to continue to work to reduce dependencies, but installation takes <1 min in our recommended workflow (https://github.com/r-lib/pkgdown/runs/506468693?check_suite_focus=true) so this isn't high priority. Please feel free to file individual issues if you spot packages that we could easily drop.

[jangorecki]

Worth to note that not only R dependencies could be reduced.
At least on ubuntu, pkgdown always required libxml2-dev, and recently it also requires libfontconfig1-dev (from systemfonts package). This is pretty unfortunately because any OS dep, comparing to R dependencies, is much more difficult to handle in organisations.
If introducing this new OS dependency was not intentional, then I think it is worth to look into it. Otherwise many configuration has to be updated to include this new OS dependency.
Quite easily reproducible on r-base image

docker run -it rocker/r-base Rscript -e 'install.packages("systemfonts")'

adding that dependency there is not a big deal but I bet there are tons of other environments where it needs to be handled as well.

[hadley]

@jangorecki it was deliberate because we want people's plot to look as good as they possibly can.