Data.table: segfaults on R3.4.4 when i is NULL

[shrektan]

The following code causes segfauls on R3.4.4.

1. Current it's reproducible on Windows. I haven't tested it on other platforms yet. UPDATE: I've reproduced this on Linux.
2. Both the CRAN version and the dev version have this issue.
3. Note, this happens only for R3.4.4 (or the previous version of R). On R3.5 or R3.6 it works fine So I doubt it relates to the code handles the ALTREP, which is introduced from R3.5. However, I'm not sure since I haven't digged into this.

library(data.table)
tbl <- data.table(A = "a", key = "A")
tbl[J(NULL)]

Update: the segfault is triggered by Cbmerge and I confirm data.table v1.12.3 also has this issue.

[shrektan]

It's cause by this line:

https://github.com/Rdatatable/data.table/blob/6f17d180ed52d517118b9df566bf7d4407754956/src/bmerge.c#L54

When i is NULL, it will be converted to zero column data.table() first. Since i is now a zero column data.table object, the VECTOR_ELT(i,0) doesn't exist and thus LENGTH(VECTOR_ELT(i,0)) returns garbage values.

I don't know why it can't happen in R3.5 or R3.6 but I will file a PR later anyway.

---

UPDATE: This is because the below code returns 0 on R3.5 or R3.6 but may be any garbage values on R3.4. I don't know the reason.

Rcpp::cppFunction("int getlength(SEXP x) { return LENGTH(VECTOR_ELT(x,0)); }")
getlength(list())