Reticulate: Conversion from python to R: Performance

[remimarenco]

Hi!

Thank you for this package, it is pretty amazing!

We have two libraries for R and Python which are doing the same things. We are currently investigating if we could drop the R one to instead use reticulate + our python one.

We tried a few things and it works super well except that our users are dealing with pretty descent data size (in average around 400-500 MB).

Trying to measure the time it takes to py_to_r convert a pandas dataframe, we have something like 40 seconds for a 150MB file. Which makes it difficult to use for us :/.

Nobody seems to talk about performance issues in conversion, thus...I am missing something? Or most people are using way smaller data?

Thanks a lot!

[kevinushey]

Any chance you can provide a reproducible example, so that we could try benchmarking and learn where the slowness is coming from? 40 seconds definitely seems prohibitively long and we'd like to do better than that.

[remimarenco]

Thanks for being so quick to answer! I am uploading the file and will post it as soon as it is done.

I tried to do simply:

pd_df <- pd$read_csv('my_file')
r_df <- py_to_r(pd_df)

And it takes around 40 sec consistently. Might depends a lot on the machine though.

You don't see the same behavior with large files on your end?

Thanks!

[remimarenco]

The matrix!

[kevinushey]

Thanks! Ultimately it looks like the performance hit comes because we do the majority of the work here in R. We may want to move parts of this routine to the C++ (libpython) side:

https://github.com/rstudio/reticulate/blob/bfac80b72b8fd0bf88581a80fcaeb6449a77df89/R/conversion.R#L282-L289

[remimarenco]

Ok! Thanks for looking into that.

So basically we can't do anything to it right now, and maybe someday it will be improved.

[agsimeonov]

We are experiencing the exact same issue. The dataframe gets generated nearly instantly in python but takes 2-3 seconds to get translated to R which gets to be a bit much over the long haul. Would love to see the performance of this updated in reticulate.

[kevinushey]

I think this is something worth working on with the next version of reticulate -- I'll create a milestone.

[ryankarel]

Hi Kevin,

This is a fantastic package; thanks so much for all the work on it. I don't know if this is helpful for this issue, but I was running into some performance issues when converting from R data frames to pandas DataFrames. When I dug into the issue further, I learned that reticulate was converting all columns of the data frame quickly _except_ for date columns. Those caused even small data sets to struggle with conversion times.

Here's an example:

library(reticulate)
library(tidyverse)
library(lubridate)
library(tictoc)

ex <- 
  tibble(date_col = seq.Date(from = today() - 1e4 + 1, today(),length.out = 1e4),
         num_col = 1:1e4,
         chr_col = sample(letters, size = 1e4, replace = TRUE)) %>% 
  mutate(fct_col = as.factor(chr_col),
         ord_col = as.ordered(chr_col))

tic("All but the date col.")
a <- r_to_py(ex %>% select(-date_col), convert = TRUE)
toc()
# All but the date col.: 0.05 sec elapsed

tic("Only date col.")
b <- r_to_py(ex %>% select(date_col), convert = TRUE)
toc()
# Only date col.: 11.37 sec elapsed

tic("Everything")
whole_sample_py <- r_to_py(ex, convert = TRUE)
toc()
# Everything: 10.18 sec elapsed

[kvaranovich]

Hi Kevin,

This is a fantastic package; thanks so much for all the work on it. I don't know if this is helpful for this issue, but I was running into some performance issues when converting from R data frames to pandas DataFrames. When I dug into the issue further, I learned that reticulate was converting all columns of the data frame quickly _except_ for date columns. Those caused even small data sets to struggle with conversion times.

Here's an example:

library(reticulate)
library(tidyverse)
library(lubridate)
library(tictoc)

ex <- 
  tibble(date_col = seq.Date(from = today() - 1e4 + 1, today(),length.out = 1e4),
         num_col = 1:1e4,
         chr_col = sample(letters, size = 1e4, replace = TRUE)) %>% 
  mutate(fct_col = as.factor(chr_col),
         ord_col = as.ordered(chr_col))

tic("All but the date col.")
a <- r_to_py(ex %>% select(-date_col), convert = TRUE)
toc()
# All but the date col.: 0.05 sec elapsed

tic("Only date col.")
b <- r_to_py(ex %>% select(date_col), convert = TRUE)
toc()
# Only date col.: 11.37 sec elapsed

tic("Everything")
whole_sample_py <- r_to_py(ex, convert = TRUE)
toc()
# Everything: 10.18 sec elapsed

I have experienced exactly the same issue. I have noticed that if there is a date/datetime column present in data frame in R, converting it to pandas data frame takes a lot of time.

[skeydan]

If you install the development version of reticulate from github, R Dates should now get converted noticeably more quickly.