Data.table: revdeps failed under R-devel

[mattdowle]

Most likely related to #3968

---

Dear maintainer,

package data.table_1.12.6.tar.gz has been auto-processed. The auto-check found problems when checking the first order strong reverse dependencies.
Please reply-all and explain: Is this expected or do you need to fix anything in your package? If expected, have all maintainers of affected packages been informed well in advance? Are there false positives in our results?

* Changes to worse in reverse dependencies *
Debian: https://win-builder.r-project.org/incoming_pretest/data.table_1.12.6_20191015_082015/reverseDependencies/summary.txt

Log dir: https://win-builder.r-project.org/incoming_pretest/data.table_1.12.6_20191015_082015/
The files will be removed after roughly 7 days.

Pretests:
Windows: https://win-builder.r-project.org/incoming_pretest/data.table_1.12.6_20191015_082015/Windows/00check.log
Debian: https://win-builder.r-project.org/incoming_pretest/data.table_1.12.6_20191015_082015/Debian/00check.log

Last published version on CRAN:

CRAN Web: https://cran.r-project.org/package=data.table

Best regards,
CRAN teams' auto-check service
Package check result: OK

Changes to worse in reverse depends:

Package: bigreadr
Check: tests
New result: ERROR
Running ‘spelling.R’ [0s/0s]
Running ‘testthat.R’ [2s/3s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:

library(testthat)
library(bigreadr)

test_check("bigreadr")
── 1. Failure: 'cbind_df' works (@test-bind.R#26) ─────────────────────────────
class(dt2) not identical to c("data.table", "data.frame").
Lengths differ: 1 is not 2

── 2. Failure: 'cbind_df' works (@test-bind.R#27)  ─────────────────────────────
`dt2` not identical to `dt`.
Attributes: < Names: 2 string mismatches >
Attributes: < Length mismatch: comparison on first 2 components >
Attributes: < Component 1: Modes: character, externalptr >
Attributes: < Component 1: target is character, current is externalptr >
Attributes: < Component 2: Modes: numeric, character >
Attributes: < Component 2: Lengths: 150, 2 >
Attributes: < Component 2: target is numeric, current is character >

... snip; many package fails ....

---

From Matt :
This is not expected. I had checked revdeps with R-release. But there is a change in this release related to the rbind/cbind S3 dispatch change in R-devel recently that Kurt asked me to cater for, and a new switch on getRversion>="4.0.0" which is in NAMESPACE too. I didn't check revdeps using R-devel as you do. I'll investigate tomorrow as it's 02:30 my time now.

---

From Uwe:
Thanks, I cancelled the submission for now.
Best,
Uwe

[mattdowle]

Smaller set than I feared :

$ grep "^Package: " summary.txt | uniq
Package: bigreadr
Package: distr6
Package: DTSg [Old version: 0.1.3, New version: 0.2.0]
Package: eplusr
Package: mlr3
Package: mlr3db
Package: mlr3filters
Package: mlr3learners
Package: mlr3pipelines
Package: mlr3tuning
Package: paradox
Package: rAmCharts
Package: scorecard
Package: staRdom
Package: text2vec

First one, bigreadr has:

cbind_df <- 
function (list_df) 
{
    do.call(cbind.data.frame, list_df)
}

and it tests that a data.table is returned when a data.table is passed in. Because data.table changed cbind.data.frame to do so. That needs a PR to bigreadr then to use the generic cbind from R 4.0.0. Will continue initial pass of these packages before deciding whether to wait for revdep updates or proceed without this change for now.

[mattdowle]

distr6 -- same 'not a function' error on CRAN in just one flavor (debian-gcc). May have been fleeting in R-devel which is clearing now. https://www.r-project.org/nosvn/R.check/r-devel-linux-x86_64-debian-gcc/distr6-00check.html

DTSg -- same 'not a function' in makeActiveBinding showing on CRAN too with R-devel : https://cran.r-project.org/web/checks/check_results_DTSg.html

eplusr -- same

mlr3* -- same

paradox -- same

text2vec -- same

staRdom -- Server error [504] for doi "10.1016/S0169-7439(97)00032-4", you may want to try again. Indeed running locally in Rdevel works ok for me.

scorecard
~A check in R-devel numbers of columns of arguments do not match appears to happen before dispatch. And data.table must be missing a test for that using the rbind generic.~
~rbind(data.table(A=1:2, B=4:5), data.table(A=3:4), fill=TRUE)
A B
1: 1 4
2: 2 5
3: 3 NA
4: 4 NA
rbind(data.table(A=1:2, B=4:5), data.table(A=3:4), fill=TRUE)
Error in rbind(deparse.level, ...) :
numbers of columns of arguments do not match
Will ask Kurt to move that check after dispatch. And needs a new test in data.table which will only pass when that's changed in R-devel.~
Nope. That error was because I installed 1.12.6 into my revdeplib using R-release, and then ran the test above using R-devel. data.table needs to be installed using R-devel, because it's install time that runs the if (getRversion()>"4.0.0") in data.table.R. To avoid that problem for users, that switch should be removed then, and I should cobble cc() to remove those in .GlobalEnv in dev until R-release (which I use for dev) becomes >=4.0.0.
So ... the real error for scorecard is that it does :

> dat = rbind(
+   germancredit[, c(sample(20,3),21)],
+   data.table(creditability=sample(c("good","bad"),10,replace=TRUE)),
+   fill=TRUE)
Error in rbind(deparse.level, ...) : 
  numbers of columns of arguments do not match

germancredit there is a data.frame. So the error is now correct, and scorecard needs to convert the first argument to data.table in order to dispatch to rbind.data.table. At least, that's my first thought at the best fix which will work in R before and after 4.0.0 with data.table before and after 1.12.6 (i.e. 4 cases).

rAmCharts
~same numbers of columns of arguments do not match with fill=TRUE. But looks like it's in the R level rbind.data.frame this time.~
It has an explicit call to rbind.data.frame, so like bigreadr, it will need a PR:
R/chart_amBoxplot.R: final.outliers <<- rbind.data.frame(final.outliers, inter, fill = TRUE)

[mattdowle]

Matt to CRAN:

---

I have investigated all the fails, here: https://github.com/Rdatatable/data.table/issues/3971
They were all one issue: 'not a function ... in makeActiveBinding'.   This was a recent problem in R-devel it seems, unrelated to the data.table update, and appears to be clearing. I can see the same errors on CRAN in many cases.
Other than 3: bigreadr, scorecard and rAmCharts.  All 3 fail for correct reasons now in R-devel and will need an update. For example 2 of them call rbind.data.frame directly with fill=TRUE, and so that should be fixed.  I will submit PRs to those 3 maintainers,  cc'd.
Those 3 will only fail in R-devel though.  They will pass on CRAN on R-release. For that reason, and also because there is some urgency to the MacOS error and binary unavailability in R-release causing users issues,  can I be excused the notice period with those 3 maintainers please?  If so, please publish 1.12.6 to CRAN. I will start submitting PRs to those 3 maintainers tomorrow so they can update to pass checks on R-devel.  And it will be helpful if this data.table 1.12.6 is on CRAN first, before they update.
Best, Matt

[mattdowle]

Matt to CRAN:

---

Thanks Florian for updating bigreadr to 0.1.11 so quickly! But that change is not quite best. It's much simpler and I've submitted a PR for bigreadr: https://github.com/privefl/bigreadr/pull/5

Uwe & Kurt,
I've added a check to data.table's .onLoad to ensure that data.table is reinstalled in R>4.0.0. This won't make a difference to CRAN checks but I think it would be tidier to get into this release.
So I will submit another 1.12.6 later tonight once I've submitted PRs for scorecard and rAmCharts.
Best, Matt

[mattdowle]

@privefl
@ShichenXie
@bthieurmel
Many thanks for updating your packages on CRAN so quickly to cater for this change in R-devel and data.table. CRAN's revdep autocheck service included your updates and tested them in latest R-devel with the updated data.table, and everything passed cleanly. data.table 1.12.6 has just been accepted on CRAN. Many thanks!

[privefl]

Many thanks to you for the great {data.table}!