Data.table: oldtncol (0) < oldncol (5) but tl of class is marked.

[tyner]

Hello,

Any ideas? Whether I use save()/load() or saveRDS()/readRDS() the same error occurs. Thanks in advance!

> library(data.table)
data.table 1.10.5 IN DEVELOPMENT built 2017-11-13 22:20:11 UTC; travis
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
> DT <- as.data.table(iris)
> save(DT, file = "~/DT.RData")
> load("~/DT.RData")
> set(DT, j = "foo", value = rep(TRUE, nrow(DT)))
Error in set(DT, j = "foo", value = rep(TRUE, nrow(DT))) : 
  Internal error, please report (including result of sessionInfo()) to datatable-help: oldtncol (0) < oldncol (5) but tl of class is marked.
> sessionInfo()
R version 3.4.2 (2017-09-28)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.3 LTS

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.10.5

loaded via a namespace (and not attached):
[1] compiler_3.4.2 tools_3.4.2

[st-pasha]

I can confirm that with the latest dev version, after loading a DataTable the following works

DT[, foo:=T]

while the following doesn't:

set(DT, j="foo", value=T)

(however set works if the .[ form was called first).

[franknarf1]

Maybe I'm missing something here, but it seems to work if you call alloc.col(DT) first, mentioned in the FAQ:

Reading data.table from RDS or RData file

*.RDS and *.RData are file types which can store in-memory R objects on disk efficiently. However, storing data.table into the binary file loses its column over-allocation. This isn't a big deal – your data.table will be copied in memory on the next by reference operation and throw a warning. Therefore it is recommended to call alloc.col() on each data.table loaded with readRDS() or load() calls.
https://cran.r-project.org/web/packages/data.table/vignettes/datatable-faq.html

(I do setDT instead of alloc.col since I'm more familiar with it.)

[tyner]

Okay, thanks @franknarf1. Perhaps that "recommendation" should be made stronger in the FAQ? In any case, if this behavior is expected, why does the error message direct the user report it?

[st-pasha]

load() loads the data as a data.frame/data.table object:

> load("~/DT.RData")
> class(DT)
[1] "data.table" "data.frame"

Moreover, the documentation for set says that it is acceptable for the first argument to be a dataframe:

Usage:     
     set(x, i = NULL, j, value)
Arguments:
     x: A ‘data.table’. Or, ‘set()’ accepts ‘data.frame’, too.

So it appears that @tyner's code should have worked even without calling setDT. The fact that it doesn't is a bug.

[arunsrinivasan]

Thanks for the report. This is a duplicate of https://github.com/Rdatatable/data.table/issues/755.

[sanjmeh]

I faced this error today and found this issue log pertaining to the same error.

Error in set(sample_dty, j = "paras", value = x1) :
Internal error, please report (including result of sessionInfo()) to data.table issue tracker: oldtncol (0) < oldncol (14) but tl of class is marked.

The code that gives this error is:

cutparas <- function(sample_dty){
  cat("Input datatable has: ",nrow(sample_dty)," rows and ",length(sample_dty)," columns")
  if(! "paras" %in% names(sample_dty) )
    x1 <- 1:nrow(sample_dty) %>% map(~stuffNL(case = .x, dt = sample_dty))
  set(sample_dty,j="paras",value = x1)
}

stuffNL is my function which returns a character array. Therefore the value=x1 inside the set is basically binding (or embedding) a list of character arrays inside a data.table as a new column.

Here's my sessionInfo() output.

> sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS  10.14.2

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] dplyr_0.7.8           tidyr_0.8.2           jsonlite_1.5          readxl_1.1.0          openxlsx_4.1.0        googlesheets_0.3.0    rdrop2_0.8.1         
 [8] textreuse_0.1.4       wrapr_1.4.2           htmltools_0.3.6       htmlTable_1.11.2      assertthat_0.2.0      textshape_1.5.0       stringdist_0.9.5.1   
[15] splitstackshape_1.4.6 purrr_0.2.5           tm_0.7-3              NLP_0.1-11            textrank_0.2.0        udpipe_0.7            pdftools_1.8         
[22] readtext_0.71         magrittr_1.5          stringr_1.3.1         textclean_0.7.3       lubridate_1.7.4       data.table_1.11.8    

loaded via a namespace (and not attached):
 [1] tidyselect_0.2.5   slam_0.1-43        lattice_0.20-35    yaml_2.2.0         rlang_0.3.0.1      syuzhet_1.0.4      pillar_1.3.0       glue_1.3.0        
 [9] lexicon_1.0.0      bindrcpp_0.2.2     bindr_0.1.1        cellranger_1.1.0   zip_1.0.0          htmlwidgets_1.3    knitr_1.20         parallel_3.5.1    
[17] Rcpp_1.0.0         backports_1.1.2    checkmate_1.8.5    RcppProgress_0.4.1 digest_0.6.18      stringi_1.2.4      grid_3.5.1         tools_3.5.1       
[25] tibble_1.4.2       crayon_1.3.4       pkgconfig_2.0.2    Matrix_1.2-14      xml2_1.2.0         httr_1.3.1         rstudioapi_0.8     R6_2.3.0          
[33] igraph_1.2.2       compiler_3.5.1    

[jangorecki]

@sanjmeh please open new issue and provide complete and minimal reproducible example

[sanjmeh]

@sanjmeh please open new issue and provide complete and minimal reproducible example

Unable to replicate with a minimal example. But it is consistent in my example. Will keep trying to reach a minimal example that is replicable.