Data.table: fwrite ignores options(scipen = 999)

[florianWickler]

Hey,

I use options(scipen = 999) to avoid scientific notation. write.table() uses this option, but not fwrite() (data.table 1.10.0).

Example:

options(scipen = 999)
library(data.table)
dt <- data.table(a = 0.0001, b = 10^6)
write.table(dt, file = "wt.csv", row.names = FALSE)
fwrite(dt, file = "fw.csv")

[apprent]

library(data.table)

aaa <- data.table(c(10, 100, 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9, 1e10, 
                    1e11, 1e12, 1e13, 1e14, 1e15, 1e16))

options(scipen = 0)
write.csv(aaa, "scipen-0-writecsv.txt")
fwrite(aaa, "scipen-0-fwrite.txt")


options(scipen = 3)
write.csv(aaa, "scipen-3-writecsv.txt")
fwrite(aaa, "scipen-3-fwrite.txt")


options(scipen = 5)
write.csv(aaa, "scipen-5-writecsv.txt")
fwrite(aaa, "scipen-5-fwrite.txt")


options(scipen = 10)
write.csv(aaa, "scipen-10-writecsv.txt")
fwrite(aaa, "scipen-10-fwrite.txt")


options(scipen = 99)
write.csv(aaa, "scipen-99-writecsv.txt")
fwrite(aaa, "scipen-99-fwrite.txt")

[ajnisbet]

Looks like this is hardcoded in fwrite.c?

One workaround is to cast integer columns with bit64::as.integer64.

[mattdowle]

And fwrite() tests in test.data.table() are affected by user setting options(scipen=) reported here with screenshots : https://twitter.com/ilarischeinin/status/1085082345013739520

[sjain777]

Turning off scientific notation in fwrite not working is an issue. Any attempt to reformat the data using standard R functions add slowness to a big-data use-case. Hence, getting this addressed in 1.12.4 as indicated above would be awesome, thanks much!

[MichaelChirico]

I have a basic working version up in fwrite_scipen branch, see #3716. Raising here since there are so many interested parties on this one...

Also adding those who reacted since I'm not sure you'll be subscribed to notifications:

@charles-plessy @mike-wise @ndukler @AndroidOatmeal @AlexIzydorczyk @novakowd @gamestrRUS @simkoum @jon-nagra @apprent @tony-stone @SimonPBiggs @AdamSpannbauer @everron @juliebehr @mt1022 @AndrewsOR @jrhawley @OrdiNeu @joblolabinette @grighi @TalSa @jeparsons @mgahan @ajnisbet @LannyFox @johndrummond @RoyalTS @acronymhaterheidi @PoGibas @malcook @yilun11 @dzeltzer @logworthy @yaakovfeldman @mattmotoki @bendae19 @dbradnum @ejoranlienea @Cruiseee @gorcha @diegogruber @rikutakei @r2evans @PoisonAlien

code to get these since it was truncated above:

library(gh)
library(magrittr)
# reactions API is "in development" so requires special header:
react_header = 'application/vnd.github.squirrel-girl-preview+json'
reaction_url = "/repos/Rdatatable/data.table/issues/2020/reactions?per_page=100"
reaction_url %>% gh(.send_headers = c(Accept=react_header)) %>%
  sapply(`[[`, c('user', 'login')) %>% unique %>% paste0('@', . ) %>% 
  paste(collapse = ' ') %>% cat