Testthat: Confusing behavior of `test_package` and `test_check` at top level

[vspinu]

As recommended lubridate tests are in tests/testthat. From the documentation I infer that to manually run a test, I have to use test_check, but it fails:

> test_check("lubridate")
Error: No tests found for lubridate

Confusingly test_package("lubridate") works.

[jimhester]

test_check() should not be run interactively, it expects the working directory to be tests/ as it is when running R CMD check. test_package(X) should not really be run interactively either, as it only runs installed tests, so if you have modified them but have not installed them it will be out of date.

If you want to run your tests during development the suggested method is devtools::test() (or the corresponding Rstudio button).

[vspinu]

test_check() should not be run interactively, it expects the working directory to be tests/ as it is when running R CMD check. test_package(X) should not really be run interactively either, as it only runs installed tests, so if you have modified them but have not installed them it will be out of date

This is a good explanation to be put in the docs then. I will be forgetting this and opening the same bug in half a year from now.

If you want to run your tests during development the suggested method is devtools::test() (or the corresponding Rstudio button).

It doesn't make any sense to use another package to run testthat tests. If this is the "design" then could please add this to the documentation as well?

[hadley]

Running tests is easy; reliably loading R code without installing the package is hard. So either install the package and run tests with test_package(), or use devtools::test().

[vspinu]

loading R code without installing the package is hard

The reason for this issue is not whether I can run it without loading, but regarding the confusing and incomplete documentation. The current doc says:

     Use ‘test_package’ to test an installed package, or in
     ‘tests/test-all.R’ if you're using the older ‘inst/tests’
     convention.

     If your tests live in ‘tests/testthat’ (preferred) use
     ‘test_check’ in ‘tests/testthat.R’.  You still use ‘test_package’
     when testing the installed package.

What does that or in the first para mean? Last sentence of the second para suggests that I have to use test_check for non-installed packages.

I hope you will agree that those two paragraphs are confusing and most likely incorrect. As I said before, the only conclusion that I could vaguely infer from those lines is that (on an installed package) I run test_check on new style and test_package on old style of the directories.

[muschellij2]

For future people running into this that may/may not be relevant to above:

Ran into this problem when a pkgname.Rcheck still existed in a level above pkgname. I had checked the packages before without tests, but I guess I killed one of the checks, so pkgname.Rcheck still existed.

You may want to run this in your package directory:

pkg_name = read.dcf("DESCRIPTION")[, "Package"]
unlink(file.path("..", paste0(pkg_name, ".Rcheck")))

But you can also use here::here() for a more robust approach:

library(here)
pkg_name = read.dcf("DESCRIPTION")[, "Package"]
unlink(here("..", paste0(pkg_name, ".Rcheck")))

[dpc10ster]

I agree the documentation is ambiguous; I had to learn the hard way that the correct way to run all tests is to use devtools::tests(). The current manual still says

"Use test_dir() for a collection of tests in a directory; use test_package() interactively at the console, and test_check() inside of R CMD check."

There is no mention there of using devtools to run tests.