Data.table: Nudge revdeps to pass on CRAN and Bioconductor

[mattdowle]

I reran revdep checks under R 3.5.0 and data.table 1.11.4. It would be great to encourage these maintainers to pass cleanly on CRAN / Bioconductor (no error or warning). I've resolved local problems first, so these are true fails, log attached. I haven't cross referenced to CRAN or Bioconductor status but they should be failing there too. Many of these seem unrelated to data.table but we need them to pass otherwise errors/warnings caused by future data.table changes may be missed when we run our revdep checks. If the revdep is already in error or warning status, it takes too long to inspect each one. Further, they could contain problems due to data.table but we're not seeing them because the other error happens first, or the warnings are to do with unexpected data/behaviour so the data.table problem isn't reached and therefore missed.

> status()
CRAN:
 ERROR   :   6 : lidR MultiABEL SpaDES.core SpaDES.tools splitstackshape tsbox 
 WARNING :   0 :  
 NOTE    : 159 
 OK      : 332 
 TOTAL   : 497 / 497 
 RUNNING :   0 :  

BIOC:
 ERROR   :   5 : BASiCS facopy ImmuneSpaceR TCGAbiolinks TPP 
 WARNING :  19 : biobroom CONFESS CRISPRseek crossmeta eQTL GenoGAM iCNV IrisSpatialFeatures MethylMix methylPipe MinimumDistance QUALIFIER RiboProfiling rTANDEM S4Vectors scater SISPA TitanCNA Uniquorn 
 NOTE    :  90 
 OK      :  21 
 TOTAL   : 135 / 135 
 RUNNING :   0 :  

fail.log

NB: box checked means now passing as of 12 June

CRAN:

• [x] lidR
• [x] MultiABEL -- https://github.com/xiashen/MultiABEL/issues/2 [ removed from CRAN ]
• [x] SpaDES.core & SpaDES.tools -- https://github.com/PredictiveEcology/SpaDES.core/issues/36#issuecomment-394060217
• [x] splitstackshape -- https://github.com/mrdwab/splitstackshape/issues/61 and https://github.com/mrdwab/splitstackshape/pull/62
• [x] tsbox -- https://github.com/christophsax/tsbox/issues/96

BIOC:
Matt emailed the maintainers of these 24 Bioc packages on 1 June asking them to look at this issue.

• [x] BASiCS
• [ ] facopy
• [ ] ImmuneSpaceR
• [x] TCGAbiolinks
• [x] TPP
• [ ] biobroom
• [ ] CONFESS
• [ ] CRISPRseek
• [ ] crossmeta
• [ ] eQTL
• [ ] GenoGAM
• [ ] iCNV
• [ ] IrisSpatialFeatures -- Maintainer's email address bounced.
• [ ] MethylMix
• [ ] methylPipe
• [ ] MinimumDistance
• [x] QUALIFIER
• [ ] RiboProfiling
• [ ] rTANDEM
• [ ] S4Vectors
• [ ] scater
• [ ] SISPA
• [ ] TitanCNA
• [ ] Uniquorn

[vlulla]

@mattdowle can you please include sessionInfo() or devtools::session_info() of your environment in your fail.log? I don't receive the "Cannot create RasterLayer object from this file. (file does not exist)" error (line 64 in fail.log) on my system! I suspect this error is for raster package and not data.table.

[mattdowle]

@vlulla Thanks for looking! Here's similar problem: https://github.com/loicdtx/bfastSpatial/issues/57, maybe it helps. I'm happy to try things as you suggest.

My sessionInfo() below and I'll add it to the log -- good idea. All packages are updated to latest CRAN and Bioconductor versions. My isolated revdeplib contains 2,829 packages of recursive dependencies.

> sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04 LTS

Matrix products: default
BLAS: /home/mdowle/build/R-3.5.0/lib/libRblas.so
LAPACK: /home/mdowle/build/R-3.5.0/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8      
 [8] LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] BiocInstaller_1.30.0

loaded via a namespace (and not attached):
[1] compiler_3.5.0 tools_3.5.0   
> 

[vlulla]

@mattdowle it appears that @Jean-Romain has rearranged the extdata folder which he uses in his tests. Maybe that is the problem? Upgrading the paths using something like sed -i -e 's@"extdata"@"/inst/extdata"@g' lidR/tests/*.R in the lidR package might do the trick? I have included Jean-Romain so he gets notified of this dicussion. Sorry, I have only ever used packages...never created any so I don't know how to do any of this testing and or other complicated orchestrating. HTH.

[mattdowle]

@vlulla @jean-romain but I see that lidR is pretty much passing on CRAN. Why might it fail for me with this error but work ok on CRAN? I'm pretty sure I'm using the latest version of lidR and all its dependencies.

[vlulla]

Sorry, I have no idea! It appears that something in your environment is not allowing you set path for the ctg (catalog) object using the vrt<- function.

[Jean-Romain]

@mattdowle Could you tell me what is going wrong for you with lidR. Currently lidR is ok on CRAN. There is an error on Solaris but it comes from the package rlas package and is not related to the data.table.

@vlulla what do you mean by I rearrange the extdata folder? My extdata folder is a regular folder that respects the CRAN requirements and only contains data to run the examples (and the tests).

Edit: Oups I missed the links to the log... I looking at it.

[Jean-Romain]

Ok I figured out the issue. Bad new, I don't know how to solve it. Let me explain the context.

Some functions write some files. For example the function catalog_reshape is dedicated to write files. Also the function grid_catalog may write some raster on the disk and load in R only a lightweight virtual raster. In the test suite I write in tempdir()or tempfile(). Everything was ok for a while. But during the last release I had to remove some tests because I got this error on CRAN.

The error you got is the same I got when testing on CRAN's win-builder. Because everything was fine on my computer I removed the incriminated tests. The functions just work and I guessed that something changed on CRAN with writing rights.

In your case, the incriminated code was ok on CRAN during the last release and I didn't remove the tests. I guess something changed with R 3.5.0 but this release is not available yet in the official repos and I'm still running R 3.4.4.

Anyway this issue is not related to data.table.

Edit: I just sent lidR 1.5.1 (equivalent to 1.5.0 with two minor bug fixed) to the CRAN's win builder and everything was OK. 0 ERROR, 0 WARNING, 0 NOTE. https://win-builder.r-project.org/uX18hs2XrfeK/00check.log

[vlulla]

@Jean-Romain the comment about extdata folder was a misunderstanding on my part. Sorry.

[mattdowle]

@Jean-Romain Since it was me-only I debugged it. In the process of debugging, it gave me this warning :

> debug(raster::stack)
> grid_canopy(ctg)
debugging in: raster::stack(save_in)
debug: standardGeneric("stack")
Browse[2]> ls()
[1] "x"
Warning messages:
1: In gdal_setInstallation(ignore.full_scan = ignore.full_scan, verbose = verbose) :
  No GDAL installation found. Please install 'gdal' before continuing:
        - www.gdal.org (no HDF4 support!)
        - www.trac.osgeo.org/osgeo4w/ (with HDF4 support RECOMMENDED)
        - www.fwtools.maptools.org (with HDF4 support)

2: In gdal_setInstallation(ignore.full_scan = ignore.full_scan, verbose = verbose) :
  If you think GDAL is installed, please run:
gdal_setInstallation(ignore.full_scan=FALSE)
Browse[2]>

After this point it seems to hang for about 1 minute before ending with the (file does not exist) message I saw from R CMD check.
How frustrating neither R CMD check nor testthat showed me this warning! I wonder what is hiding the warning. Anyway, rather than install GDAL from source as these links suggest as well as the rgdal package description, I searched around and found that gdal-bin is in standard Ubuntu repos which I didn't have, so installed. I also installed libbase1 which seems to be something to do with gdal. I already had libgdal-dev and libproj-dev which I thought was all that was needed since there were no warnings or errors to suggest any library was missing. I read somewhere that rgdal needs to be reinstalled to pick up the new configuration, so I reinstalled the rgdal R package.

Now lidR 1.5.0 is passing for me!! Thanks for looking.

[juyeongkim]

@mattdowle ImmuneSpaceR maintainer here. I just checked your fail.log, and the errors are not related to data.table.

[mattdowle]

@juyeongkim That's right - not due to data.table. But please still fix your package's errors asap, because they cause work for me to check every time if those errors are due to data.table or not.

Closing this issue now because all CRAN packages are now passing thanks to the maintainers of lidR, SpaDES.core, SpaDES.tools, splitstackshape and tsbox.