Reticulate: passing dates or datetime.date objects as parameters

[rafazaya]

Not sure if this is considered a good reprex example for reticulate, but let me try this.

I'm having issues using datetime.date objects as parameters into my python scripts, where they become floats in the process Per the below, a datetime.date object as a default parameter works, passing it in does not, and returning the value returns the integer equivalent. Thoughts on how to pass a date to a function?

test.py:

from datetime import datetime, timedelta, date

def prior_day(edate=date.today()):
    return edate - timedelta(days=1)

def return_param(edate):
    return edate

Then in R...

library(reticulate)
#> Warning: package 'reticulate' was built under R version 3.4.4
conda_path <- "my/path"
env_name <- "envir"
use_condaenv(condaenv = env_name, conda = conda_path)

dt <- import("datetime")
edate <- dt$date(2018L,3L,26L)
edate
#> [1] "2018-03-26"
source_python("test.py")
prior_day()
#> [1] "2018-03-25"
prior_day(edate)
#> Error in py_call_impl(callable, dots$args, dots$keywords): TypeError: unsupported operand type(s) for -: 'float' and 'datetime.timedelta'
#> 
#> Detailed traceback: 
#>   File "<string>", line 4, in prior_day
return_param(edate)
#> [1] 17616

Created on 2018-03-26 by the reprex package (v0.2.0).

[kevinushey]

Thanks for the bug report! This is primarily because we transform R Date objects into NumPy datetime64 arrays. See e.g.

> library(reticulate)
> r_to_py(Sys.Date())
['2018-03-26T00:00:00.000000000']

I believe this is typically the behaviour you want when manipulating e.g. Dates that live as part of a Pandas DataFrame, or as a vector / list of dates, but it does fail when one attempts to manipulate with plain old Python datetime objects.

I'm still not 100% sure whether we've made the right choice, though. Perhaps Date objects should be coerced into Python datetime objects? Maybe only times (e.g. POSIXt) should be converted into NumPy datetime64 objects?

[rafazaya]

I'm very biased, but it would be cool to have some way to get to python datetime objects. I haven't played around with reticulate enough to investigate if there's an awkward workaround, which would be sufficient. I could also just write wrappers which converts the strings to dates in python and then passes arguments along within python, which may be best if this is a small use case generally.

I'm thankful for this package and the comment!

[kevinushey]

For context, we make the conversion here:

https://github.com/rstudio/reticulate/blob/897c65dbf62fb570f86c9f6d54bc79a0fd98135a/R/conversion.R#L125-L146

Perhaps there's a better way we could decide whether we should produce a Python datetime vs. a NumPy datetime64?

Maybe we should only convert dates / times contained within a data frame to a NumPy datetime64, and standalone date vectors should use (lists of) Python datetime objects?

[kevinushey]

@terrytangyuan I'm curious what you think re: choosing Python datetime vs. NumPy datetime64 here.

[terrytangyuan]

I am all in for more performant conversion (I like the current default and fallback behavior) but I don't think this is 100% what the users might want. Perhaps we should expose this as an option?

[vqv]

I just ran into this issue. The current choice makes sense if the primary purpose is to convert R data frames into Pandas data frames. However, when calling Python functions it is necessary to have exact control over the resulting type produced by r_to_py().

The current documentation says that single element R vector will be converted to a Python scalar (https://rstudio.github.io/reticulate/articles/calling_python.html). This suggests that a single Date should be converted to a scalar datetime rather than an array (of a single element). However, I don't think it's a good idea for the resulting type to be fundamentally depend on the length of the input, i.e. datetime versus NumPy array of datetime64's. So I don't think it is a good idea for r_to_py() to guess, and there should be a way to get explicit control over the resulting type.

What about exposing explicit converters, e.g. r_to_py_datetime() with the same functionality as here:
https://github.com/rstudio/reticulate/blob/897c65dbf62fb570f86c9f6d54bc79a0fd98135a/R/conversion.R#L133-L145

[kevinushey]

Thanks for the comments, everyone. I think we should adopt the following convention:

• R Dates are converted to Python datetime objects and vice versa;
• R times are converted to NumPy datetime64 objects and vice versa.

Does this sound reasonable? Note that I'm exploring this as part of https://github.com/rstudio/reticulate/pull/259.

Note that doing this also somewhat relieves reticulate from having to provide conversion helpers -- if you want a Python datetime, then you can just convert your R time to a date object, and then request the conversion.

[eddelbuettel]

Just as an aside, we can't fully map to datetime64 as we only have 53 bits in POSIXct. That was one of the reasons I started the nanotime package which lives on top of bit64::integer64.

I think the route via Date is not good either as you'd loose all intra-day components (or maybe I am missing something). In that sense, better to 'just' loose the sub-microsecond part...

[kevinushey]

I think POSIXct <-> datetime64 as a default is at least sensible, but given that we do risk losing precision we may want to provide some APIs for using nanotime explicitly. Does that make sense?

Maybe we need some extensions to py_to_r() that allow users to define custom converters, that take column(s) of a Pandas DataFrame and produce R objects? Then in theory a user or a package could define a converter that goes from datetime64 <-> nanotime.

A similar argument could probably be made for r_to_py()...

[eddelbuettel]

Yes, a hook to permit extensions seems sensible, as does the suggested default.

The added converters always pull a tail of other packages in: bit64 for representation, RcppCCTZ to parse/convert and nanotime for the actual datetime objects. A handful. Probably not a useful default, but possible Suggests: and optional use.

[rafazaya]

Following up on this, it seems the (possibly new?) convert=FALSE parameters do the trick. For a scalar date, i can do the below:

library(reticulate)
dt <- import("datetime", convert = FALSE)
dt$date$today()
dt$date(2019L, 4L, 24L)

...and for the more general case, #259 is already merged.

I'll close this.