Data.table: Do not suggests fread fill=TRUE if already used

[jangorecki]

current fread behavior.

dt=fread("Rprofmem.out", header=FALSE)
#Warning message:
#In fread("Rprofmem.out", header = FALSE) :
#  Stopped early on line 2. Expected 3 fields but found 2. Consider fill=TRUE and comment.char=. First #discarded non-empty line: <<new page:"library" >>
dt=fread("Rprofmem.out", header=FALSE, fill=TRUE)
#Warning message:
#In fread("Rprofmem.out", header = FALSE, fill = TRUE) :
#  Stopped early on line 90. Expected 7 fields but found 8. Consider fill=TRUE and comment.char=. First #discarded non-empty line: <<new page:"close" "readRDS" "FUN" "lapply" "find.package" "system.file" #"library" >>

Before printing this warning we should check if fill=TRUE was used.

[st-pasha]

If fill=TRUE then fread should add new columns as necessary. And then no error will be emitted.

[jangorecki]

Rprofmem.out.zip
There are other issues in this file, so don't try hard to parse it, attaching so issue can be reproduced.
profmem package also fail to parse it.

[lotard]

One up this, which can be reproduced using this simple toy example:

> body = paste0(rep("1 2\n", 1000), collapse="")
> main = paste0(body, "1 2 3\n", body, collapse="")
> fread(main, fill=T)
Warning message:
In fread(main, fill=T) :
  Stopped early on line 1001. Expected 2 fields but found 3.
  Consider fill=TRUE and comment.char=. First discarded non-empty line: <<1 2 3>>

Issue seems to be incomplete sampling of possible number of columns. I don't necessarily think max(countfields(table)) is the right solution here (it is computationally expensive), but allowing the user to force the number of columns, if they know it (e.g. fill=3 in my example)

[randomgambit]

I have the same issue actually. Is there a workaround for now?
Thanks!

[lotard]

Find out the max number of fields, create a 'fake' line with that number of fields at the beginning of the file, read the file, scrap that line.

[Befrancesco]

I have the same issue, but my csv is 3.5 gb.
I have to modify it by powershell or is there an other way?

Thank you in advance.

[gleesonger]

Having the same issue, is there any work around which doesn't involve patching the start of the file contents?

[GabeAl]

Same problem.

[loukesio]

Hehehheh same for me