devtools::check() running examples with --run-donttest when run_dont_test = FALSE

[HaydenMacDonald]

devtools::check() seems to be ignoring run_dont_test = FALSE resulting in this output:

√  checking examples (4.4s)
/   checking examples with --run-donttest ...

I'm unsure as to whether this has anything to do with the R 4.0.0 update, but rcmdcheck doesn't seem to have any open related issues.

Session info

```
Session info ------------------------------------------------------------------------------------------------
setting value
version R version 4.0.0 (2020-04-24)
os Windows 10 x64
system x86_64, mingw32
ui RStudio
language (EN)
collate English_Canada.1252
ctype English_Canada.1252
tz America/New_York
date 2020-04-26

Packages ---------------------------------------------------------------------------------------------------
! package * version date lib source
acepack 1.4.1 2016-10-29 [1] CRAN (R 4.0.0)
assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.0)
backports 1.1.6 2020-04-05 [1] CRAN (R 4.0.0)
base64enc 0.1-3 2015-07-28 [1] CRAN (R 4.0.0)
callr 3.4.3 2020-03-28 [1] CRAN (R 4.0.0)
checkmate 2.0.0 2020-02-06 [1] CRAN (R 4.0.0)
cli 2.0.2 2020-02-28 [1] CRAN (R 4.0.0)
clipr 0.7.0 2019-07-23 [1] CRAN (R 4.0.0)
cluster 2.1.0 2019-06-19 [2] CRAN (R 4.0.0)
colorspace 1.4-1 2019-03-18 [1] CRAN (R 4.0.0)
crayon 1.3.4 2017-09-16 [1] CRAN (R 4.0.0)
data.table 1.12.8 2019-12-09 [1] CRAN (R 4.0.0)
desc 1.2.0 2018-05-01 [1] CRAN (R 4.0.0)
devtools * 2.3.0 2020-04-10 [1] CRAN (R 4.0.0)
digest 0.6.25 2020-02-23 [1] CRAN (R 4.0.0)
dplyr 0.8.5 2020-03-07 [1] CRAN (R 4.0.0)
ellipsis 0.3.0 2019-09-20 [1] CRAN (R 4.0.0)
evaluate 0.14 2019-05-28 [1] CRAN (R 4.0.0)
fansi 0.4.1 2020-01-08 [1] CRAN (R 4.0.0)
foreign 0.8-78 2020-04-13 [2] CRAN (R 4.0.0)
Formula 1.2-3 2018-05-03 [1] CRAN (R 4.0.0)
fs 1.4.1 2020-04-04 [1] CRAN (R 4.0.0)
generics 0.0.2 2018-11-29 [1] CRAN (R 4.0.0)
ggplot2 3.3.0 2020-03-05 [1] CRAN (R 4.0.0)
glue 1.4.0 2020-04-03 [1] CRAN (R 4.0.0)
gridExtra 2.3 2017-09-09 [1] CRAN (R 4.0.0)
gtable 0.3.0 2019-03-25 [1] CRAN (R 4.0.0)
here 0.1 2017-05-28 [1] CRAN (R 4.0.0)
Hmisc 4.4-0 2020-03-23 [1] CRAN (R 4.0.0)
htmlTable 1.13.3 2019-12-04 [1] CRAN (R 4.0.0)
htmltools 0.4.0 2019-10-04 [1] CRAN (R 4.0.0)
htmlwidgets 1.5.1 2019-10-08 [1] CRAN (R 4.0.0)
httr 1.4.1 2019-08-05 [1] CRAN (R 4.0.0)
janitor 2.0.1 2020-04-12 [1] CRAN (R 4.0.0)
jpeg 0.1-8.1 2019-10-24 [1] CRAN (R 4.0.0)
knitr 1.28 2020-02-06 [1] CRAN (R 4.0.0)
lattice 0.20-41 2020-04-02 [2] CRAN (R 4.0.0)
latticeExtra 0.6-29 2019-12-19 [1] CRAN (R 4.0.0)
lifecycle 0.2.0 2020-03-06 [1] CRAN (R 4.0.0)
lubridate 1.7.8 2020-04-06 [1] CRAN (R 4.0.0)
magrittr 1.5 2014-11-22 [1] CRAN (R 4.0.0)
Matrix 1.2-18 2019-11-27 [2] CRAN (R 4.0.0)
memoise 1.1.0 2017-04-21 [1] CRAN (R 4.0.0)
munsell 0.5.0 2018-06-12 [1] CRAN (R 4.0.0)
naniar 0.5.0 2020-02-28 [1] CRAN (R 4.0.0)
nnet 7.3-13 2020-02-25 [2] CRAN (R 4.0.0)
pillar 1.4.3 2019-12-20 [1] CRAN (R 4.0.0)
pkgbuild 1.0.7 2020-04-25 [1] CRAN (R 4.0.0)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.0)
pkgload 1.0.2 2018-10-29 [1] CRAN (R 4.0.0)
plyr 1.8.6 2020-03-03 [1] CRAN (R 4.0.0)
png 0.1-7 2013-12-03 [1] CRAN (R 4.0.0)
polite 0.1.1 2019-11-30 [1] CRAN (R 4.0.0)
prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.0.0)
processx 3.4.2 2020-02-09 [1] CRAN (R 4.0.0)
ps 1.3.2 2020-02-13 [1] CRAN (R 4.0.0)
purrr 0.3.4 2020-04-17 [1] CRAN (R 4.0.0)
R6 2.4.1 2019-11-12 [1] CRAN (R 4.0.0)
ratelimitr 0.4.1 2018-10-07 [1] CRAN (R 4.0.0)
rcmdcheck 1.3.3 2019-05-07 [1] CRAN (R 4.0.0)
RColorBrewer 1.1-2 2014-12-07 [1] CRAN (R 4.0.0)
Rcpp 1.0.4.6 2020-04-09 [1] CRAN (R 4.0.0)
remotes 2.1.1 2020-02-15 [1] CRAN (R 4.0.0)
reprex * 0.3.0 2019-05-16 [1] CRAN (R 4.0.0)
rlang 0.4.5 2020-03-01 [1] CRAN (R 4.0.0)
rmarkdown 2.1 2020-01-20 [1] CRAN (R 4.0.0)
robotstxt 0.6.2 2018-07-18 [1] CRAN (R 4.0.0)
roxygen2 7.1.0 2020-03-11 [1] CRAN (R 4.0.0)
rpart 4.1-15 2019-04-12 [2] CRAN (R 4.0.0)
rprojroot 1.3-2 2018-01-03 [1] CRAN (R 4.0.0)
rstudioapi 0.11 2020-02-07 [1] CRAN (R 4.0.0)
rvest 0.3.5 2019-11-08 [1] CRAN (R 4.0.0)
scales 1.1.0 2019-11-18 [1] CRAN (R 4.0.0)
sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.0.0)
snakecase 0.11.0 2019-05-25 [1] CRAN (R 4.0.0)
P squashinformr * 0.1.0 2020-04-26 [?] local
stringi 1.4.6 2020-02-17 [1] CRAN (R 4.0.0)
stringr 1.4.0 2019-02-10 [1] CRAN (R 4.0.0)
survival 3.1-12 2020-04-10 [2] CRAN (R 4.0.0)
testthat 2.3.2 2020-03-02 [1] CRAN (R 4.0.0)
tibble 3.0.1 2020-04-20 [1] CRAN (R 4.0.0)
tidyr 1.0.2 2020-01-24 [1] CRAN (R 4.0.0)
tidyselect 1.0.0 2020-01-27 [1] CRAN (R 4.0.0)
usethis * 1.6.0 2020-04-09 [1] CRAN (R 4.0.0)
vctrs 0.2.4 2020-03-10 [1] CRAN (R 4.0.0)
visdat 0.5.3 2019-02-15 [1] CRAN (R 4.0.0)
whisker 0.4 2019-08-28 [1] CRAN (R 4.0.0)
withr 2.2.0 2020-04-20 [1] CRAN (R 4.0.0)
xfun 0.13 2020-04-13 [1] CRAN (R 4.0.0)
xml2 1.3.2 2020-04-23 [1] CRAN (R 4.0.0)
xopen 1.0.0 2018-09-17 [1] CRAN (R 4.0.0)

[1] C:/Users/Hayden/Documents/R/win-library/4.0
[2] C:/Program Files/R/R-4.0.0/library

P -- Loaded and on-disk path mismatch.

[HaydenMacDonald]

I've just noticed that R 4.0.0 has changed R CMD check --as-cran to run \donttest examples (see Utilities section here). "This can be temporarily circumvented during development by setting environment variable _R_CHECK_DONTTEST_EXAMPLES_ to a false value."

[Alternikaner]

In case it isn't obvious to anyone landing here how/where to set the environmental variable mentioned above (it wasn't clear to me) you can try the following:

1. Create a text file named check.Renviron containing the following line:
   _R_CHECK_DONTTEST_EXAMPLES_=false
2. Put this file in the (\~/.R/) directory [Can be identified with normalizePath("\~/.R/") if required]

[hadley]

I clarified the docs. Generally, now if you don't want to run tests on CRAN \dontrun{} is more likely to work, but using \dontrun{} may cause initial submission to fail.

[dankelley]

@hadley, I wonder whether you might say a bit more here about your statement "but using \dontrun{} may cause initial submission to fail" in the just-previous comment. Why would using \dontrun{} cause a problem?

This is a general question, but in case it's of any interest, I am asking because we have some examples in the oce package (https://github.com/dankelley/oce/issues/1798) that are display how to work with data that are not provided in the package. For years, we have used \donttest{} to wrap such examples.

I think the choices as

1. use \donttest{} ... this seems likely to cause CRAN rejection because the CRAN test machine will not be able to run the code.
2. use \dontrun{} ... this seems to be the way forward, but your note about submission failing worries me (hence this request)
3. move example code to \preformatted{} blocks, probably within the Details section of the documentation
4. drop the example code

As a user, I often jump to the Examples part of documentation, so I think option 3 is undesirable, although not as bad as option 4. Since option 1 seems destined to fail in the present-day R system, we are left with option 2.

Option 2 is what I infer as the best solution, based on https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Marking-text and the purpose of this comment is to ask whether this is correct (in light of your proviso about rejection).

My guess is that quite a few folks come upon this issue through web searches, and so it might be very helpful to get a bit of advice posted here. Even a single phrase like "Use \dontrun{} for examples using data not provided with your package" would be enormously helpful to those of us who fear CRAN rejection.

[hadley]

@dankelley on initial submission (only) CRAN tends to complain if you use \dontrun{}.

[dankelley]

@hadley thanks very much for the clarification. It helps a lot. Dan.