Roxygen2: FR: Literate testing

[krlmlr]

Often, the behavior of a function or object can be explicitly specified in a test. It would be great if roxygen somehow enabled tying the verbal and programmatical specification.

In #88, @dlebauer suggests a new "testthat" roclet. Another option would be to simply parse the files below the "tests" directory.

[leeper]

We are working on this at the rOpenSci unconf and have a mockup here: https://github.com/gmbecker/roxygen/tree/tests-tag Basic setup is to be able to do something like:

#' @tests
#' context("Example tests")
#' test_that("A test", {
#'   expect_true(TRUE)
#' })

and automatically generate a test file.

[krlmlr]

@leeper: Any updates?

[leeper]

Unfortunately, no. @jimhester? @gmbecker?

[gmbecker]

I'm buried entirely under conferences right now, but will try to put together a pull request in a couple weeks. The signals from @hadley were somewhat mixed about it's prospects though. At least that's the feeling I got at the time.

[hadley]

Yeah, it's unlikely that I'd accept in roxygen2. However, if roxygen2 needs extension points so it's easy to provide roclets in other packages, I'd be happy to add them.

[kindlychung]

I really like the way Rust handles this: Examples in the documentation are used as tests. The rationale is that tests are basically examples wrapped inside a layer of condition checking, so why not profit from them?

[jimhester]

@kindlychung R CMD check already runs all examples and fails if there are any errors in running the examples. There is also testthat::test_examples() if you want to explicitly run examples in your tests.

[krivit]

@kindlychung , this is less than ideal for R packages. Barring ugly workarounds, testing the output of a command consumes the result, so the user does not see what it actually looks like. For example,

will print the result (4) but will not test, whereas

stopifnot(2+2 == 4)

will test the result but won't print it.

Also, CRAN likes examples to be short (<5 seconds, IIRC), but they give more leeway to tests.

@hadley , if I may ask, why not? It looks like a feature that would be used quite heavily if available, and what it does seems relatively simple and low-maintenance.

[hadley]

@krivit (a) I don't think it would be used heavily and (b) every additional feature in roxygen2 makes it more complex and harder to maintain. I don't think the cost-benefit tradeoff favours implementation of this feature.

[krivit]

@hadley , thanks for the explanation. I know I would use it as much as I could: if already putting the documentation and the examples next to the source code, putting the unit tests there as well seems to me to be a no-brainer. Of course, I am a sample size of 1. I wonder if there is a way to poll potential users to gauge interest.

[hadley]

If you want it, you are very welcome to implement it in your own package.

[jimhester]

I am pretty sure someone recently started working on a package to do this, based off of our code from the unconf a few years ago, but I unfortunately cannot find it now.

[rmflight]

Is it this one @jimhester : https://github.com/rmflight/roxygentests

[jimhester]

Yeah, thanks for chiming in @rmflight!