Testthat: tests run in R session, but "R CMD check" fails

[unDocUMeantIt]

tests for a package (see below) fail when run by "R CMD check", but if i run them from an R session, everything is fine. the package tested creates some files in a tempdir and i want to check for their existance.

i've noticed errors like
"Error in file(filename, \"r\", encoding = encoding) : cannot open the connection
Calls: local ... eval.parent -> eval -> eval -> eval -> eval -> source -> file
In addition: Warning message:
In file(filename, \"r\", encoding = encoding): cannot open file 'startup.Rs': No such file or directory"

my package functions use some setwd() calls. when i create a dummy "startup.Rs" file in the directory the functions jump to, these errors disappear. i have no clue what this file is supposed to do or why it's only missing when "R CMD check" is running the tests. anyway, the tests still fail after all.

in addition, testthat didn't load all dependencies, i had do manually add a require() call to the test suite. here's the package, if you want to try to replicate the problem: http://reaktanz.de/stuff/R/roxyPackage_0.03-11.tar.gz

[hadley]

Hmmm, calls to setwd() are usually a bad idea. The tests are probably being run in a different environment that what you expect - I've only ever been able to debug problems by painfully adding lots of print statements :(

[yoni]

I'm seeing similar issues, though I'm not calling setwd(). I'm able to run my test suite using testthat:::test_dir and devtools:::test(fresh=TRUE), but not devtools:::test() (i.e. with fresh=FALSE) or devtools:::check().

I'll dig deeper into the issue, but figured I'd share the strange behavior while I work through this.

[lhsego]

I'm finding the same issue, using R 3.2.2 on OSX Mavericks. Per the instructions in the README.md, I put the following into tests/testthat.R with test files in tests/testthat:

library(testthat)
library(qFeature)
test_check("qFeature")

Upon running R CMD CHECK qFeature, it hangs when it starts the tests. Breaking it with Ctrl-C gives the following:

    Running ‘testthat.R’^C
 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  > library(testthat)
  > library(qFeature)
  > 
  > test_check("qFeature")
  Error in file(filename, "r", encoding = encoding) : 
    cannot open the connection
  Calls: local ... eval.parent -> eval -> eval -> eval -> eval -> source -> file
  In addition: Warning message:
  In file(filename, "r", encoding = encoding) :
    cannot open file 'startup.Rs': No such file or directory
  Execution halted

However, all things being equal, testing in the R console works fine:

 > library(testthat)
 > test_package("qFeature")
ddply_getFeatures() - Output Values : .
discFeatures() - Output Values : .........
fitQ() - Summary Values : Loading required package: qFeature
................................
fitQ() - Linear Output Values : ....
fitQ() - Quadratic Output Values : ....
getFeatures() - Output Values : .

DONE 

[lhsego]

PS: devtools::check("~/path_to_package/qFeature") produces the same behavior as running R CMD CHECK: it hangs, and once broken with Cntrl-C, the same error appears.

[lhsego]

Found a workaround

This issue is related to these others: https://github.com/hadley/testthat/issues/144 and https://github.com/zoonproject/zoon/pull/150.

A workaround is here: https://github.com/luckyrandom/cmaker/commit/b85813ac2b7aef69932eca8fbb4fa0ec225e0af0

Simply add Sys.setenv("R_TESTS" = "") as the first line in tests/testthat.R

[tracyredcross]

This code keeps popping up in R.
Error in file(filename, "r", encoding = encoding) :
cannot open the connection
In addition: Warning message:
In file(filename, "r", encoding = encoding) :
cannot open file 'rankall.R': No such file or directory
, What can I do?

[gvegayon]

I see this error too in what I think is a more recent version of testthat (1.0.2). Once I include Sys.setenv("R_TESTS" = "") as the first line in tests/testthat.R as suggested here it works.

* checking tests ...
  Running ‘testthat.R’^C
 ERROR
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
  > 
  > test_check("phylogenetic")
  Error in file(filename, "r", encoding = encoding) : 
    cannot open the connection
  Calls: local ... eval.parent -> eval -> eval -> eval -> eval -> source -> file
  In addition: Warning message:
  In file(filename, "r", encoding = encoding) :
    cannot open file 'startup.Rs': No such file or directory
  Execution halted

  testthat results ================================================================
  OK: 6 SKIPPED: 0 FAILED: 0
  Execution halted

R version 3.3.2 (2016-10-31)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] testthat_1.0.2      phylogenetic_0.1.99

loaded via a namespace (and not attached):
 [1] magrittr_1.5      R6_2.2.0          Matrix_1.2-8      parallel_3.3.2    tools_3.3.2       coda_0.19-1      
 [7] crayon_1.3.2      Rcpp_0.12.9       ABCoptim_0.14.0   nlme_3.1-128      ape_4.0           grid_3.3.2       
[13] numDeriv_2016.8-1 lattice_0.20-34  

[jaredlander]

I am getting that same "RStudio not running" error even with Sys.setenv("R_TESTS" = "") in testthat.r.

Tests run fine with devtools::test() or running them manually. They fail with devtools::check().

> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.4.3  tools_3.4.3     withr_2.1.2     yaml_2.1.19     memoise_1.1.0  
[6] git2r_0.21.0    digest_0.6.15   packrat_0.4.8-1 devtools_1.13.5

[jsilve24]

I am also getting same error despite trying the Sys.setenv("R_TESTS" = "") workaround in testthat.R.

> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.4

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mongrel_0.1          BiocInstaller_1.30.0 devtools_1.13.6     

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.17              plyr_1.8.4                compiler_3.5.1            pillar_1.2.3             
 [5] git2r_0.21.0              bindr_0.1.1               forcats_0.3.0             tidybayes_0.12.1.9000    
 [9] tools_3.5.1               testthat_2.0.0            svUnit_0.7-12             digest_0.6.15            
[13] ggstance_0.3              jsonlite_1.5              memoise_1.1.0             tibble_1.4.2             
[17] gtable_0.2.0              lattice_0.20-35           pkgconfig_2.0.1           rlang_0.2.1              
[21] Matrix_1.2-14             commonmark_1.5            parallel_3.5.1            curl_3.2                 
[25] yaml_2.1.19               LaplacesDemon_16.1.1      bindrcpp_0.2.2            coda_0.19-1              
[29] withr_2.1.2               dplyr_0.7.6               httr_1.3.1                stringr_1.3.1            
[33] roxygen2_6.0.1            xml2_1.2.0                knitr_1.20                arrayhelpers_1.0-20160527
[37] grid_3.5.1                tidyselect_0.2.4          glue_1.2.0                R6_2.2.2                 
[41] purrr_0.2.5               tidyr_0.8.1               ggplot2_3.0.0             driver_0.1.0             
[45] magrittr_1.5              ggridges_0.5.0            MASS_7.3-50               scales_0.5.0             
[49] assertthat_0.2.0          colorspace_1.3-2          stringi_1.2.3             lazyeval_0.2.1           
[53] munsell_0.5.0             RcppEigen_0.3.3.4.0    

checking tests ... ERROR
  Running ‘testthat.R’
Running the tests in ‘tests/testthat.R’ failed.
Last 13 lines of output:
         if (logical.return) 
             message(paste("Error:", msg), domain = NA)
         else stop(msg, call. = FALSE, domain = NA)
     })
  3: tryCatchList(expr, classes, parentenv, handlers)
  4: tryCatchOne(expr, names, parentenv, handlers[[1L]])
  5: value[[3L]](cond)
  6: stop(msg, call. = FALSE, domain = NA)

  ══ testthat results  ════════════════════════════════════════════════════════════════════════════════════════════════
  OK: 13 SKIPPED: 0 FAILED: 1
  1. Error: (unknown) (@test-main.R#3) 

  Error: testthat unit tests failed
  Execution halted

Any advice?

[jimhester]

@jsilve24 Please open a new issue and if possible include the package you are seeing this error as a link, it won't be possible to debug this without seeing the package.

[espinielli]

In my particular case, the discrepancy I got between devtools::test() and devtools::check() was due to having the definition of a new unit in a .R file rather than inside .onLoad() (and removing it in .onUnload()).

Before

In R/unit_conversion.R:

units::install_conversion_constant("FL", "ft", 100)

fl_to_foot <- function(fl) {
  fl %>%
    units::set_units(FL) %>%
    units::set_units(ft) %>%
    units::drop_units()
}

in tests/testthat/test-conversion.R:

context("Units convertion")
library(trfcplx)

test_that("fl_to_foot converts flight levels (FL) to feet", {
  expect_equal(fl_to_foot(150), 15000)
})

• devtools::test(): OK
• devtools::check(): FAIL

After

In R/unit_conversion.R:

fl_to_foot <- function(fl) {
  fl %>%
    units::set_units(FL) %>%
    units::set_units(ft) %>%
    units::drop_units()
}

In zzz.R (a new files as per "When you do need side-effects" section):

# register flight levels (FL) as a unit when loading this package
.onLoad <- function(libname, pkgname) {
  # install user-define unit for flight level
  units::install_conversion_constant("FL", "ft", 100)

  invisible()
}

# register flight levels (FL) as a unit when loading this package
.onUnload <- function(libname, pkgname) {
  # uninstall user-define unit for flight level
  units::remove_symbolic_unit("FL")

  invisible()
}

• devtools::test(): OK
• devtools::check(): OK