Cannot get to work rpy2 v 2.9.2 and reticulate v 1.6 with R v 3.4.3 and Python 3.5.5 (or 3.6.5) to share the same instance of R embedded process / engine instance.
context - reviewed these but still cannot make this work
https://github.com/rstudio/reticulate/issues/98
https://github.com/rstudio/reticulate/pull/104
https://github.com/rstudio/reticulate/issues/138
Environment
$ uname -a
Linux bigdata-vm 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Python and R setup with conda environment.yml
name: dev-ds
channels:
- r
- conda-forge
- anaconda
- defaults
dependencies:
- conda-forge::python=3.5.5
- conda-forge::numpy=1.14.2
- conda-forge::pandas=0.22.0
# ==============
# R core
# ==============
- r-base=3.4.3
- r-magrittr=1.5
- r-reticulate # is 1.3 but need 1.6* so re-installed manually from R
- r-dplyr=0.7.4
- r-dbi=0.7
- r-r6=2.2.2
- r-magrittr=1.5
- r-jsonlite=1.5
- r-dplyr=0.7.4
- r-purrr=0.2.4
- r-bindr=0.1
# ==============
# R application runtime
# ==============
- r-readr=1.1.1
- r-config=0.2
- r-xml
- r-reshape2
- r-data.table
- r-forecast=8.2
- r-tseries
- nlopt
- r-nloptr=1.0.4
# ==============
# R application dev
# ==============
- r-git2r # 0.20.0
- r-rcpp=0.12.14 # 0.12.16
- r-devtools=1.13.4
- r-testthat=2.0.0
- r-roxygen2
- r-sparklyr # 0.7.0
# ==============
# Python core
# ==============
- pip:
- rpy2==2.9.2
Using R from under the conda env:
library(reticulate)
reticulate::import("sys")
rpy2 <- import('rpy2')
robjects <- import('rpy2.robjects')
exists with _"R is already initialized"_ caused by last line.
From a Python script some.py
from rpy2.robjects import pandas2ri
from rpy2.robjects.packages import importr
import rpy2.robjects as robjects
def main():
r = robjects.r
x = robjects.IntVector(range(10))
y = r.rnorm(10)
import os, sys
os.environ['RETICULATE_PYTHON'] = sys.executable
robjects.r('library("reticulate")')
robjects.r('reticulate::import("sys")')
nested = robjects.r('''
nested <- function() {
library(reticulate)
use_condaenv("dev-ds")
reticulate::import("sys")
robjects <- import('rpy2.robjects')
robjects$r('R.version')
return (1)
}
nested
''')
n = nested()
print(n)
if __name__ == "__main__":
main()
getting this
.../bin/python/conda3/envs/dev-ds/lib/python3.5/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: 2: package ‘reticulate’ was built under R version 3.4.3
warnings.warn(x, RRuntimeWarning)
Traceback (most recent call last):
File ".../some.py", line 63, in <module>
main()
File ".../some.py", line 40, in main
robjects.r('reticulate::import("sys")')
File "/.../bin/python/conda3/envs/dev-ds/lib/python3.5/site-packages/rpy2/robjects/__init__.py", line 352, in __call__
res = self.eval(p)
File "/../bin/python/conda3/envs/dev-ds/lib/python3.5/site-packages/rpy2/robjects/functions.py", line 178, in __call__
return super(SignatureTranslatedFunction, self).__call__(*args, **kwargs)
File "/.../bin/python/conda3/envs/dev-ds/lib/python3.5/site-packages/rpy2/robjects/functions.py", line 106, in __call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in py_call_impl(callable, dots$args, dots$keywords) :
ImportError: 'rpycall' is not a built-in module
Detailed traceback:
File "/development/bin/python/conda3/envs/dev-ds/lib/R/library/reticulate/python/rpytools/call.py", line 2, in <module>
import rpycall
Segmentation fault (core dumped)
@randy3k kindly advise what is missing or wrong.
I am pretty sure that reticulate is ready to work in an R environment which is embedded in Python. But I am not a rpy user, I cannot say much about it.
thanks. will bug @lgautier next.
UPDATE: uninstalled rpy2 2.9.2 from pip and installed default (2.9.1 with conda) and got same results
```The following NEW packages will be INSTALLED:
jinja2: 2.10-py35h480ab6d_0
markupsafe: 1.0-py35h4f4fcf6_1
r-bit: 1.1_12-r343h086d26f_0
r-bit64: 0.9_7-r343h086d26f_0
r-blob: 1.1.0-r343h889e2dd_0
r-rsqlite: 2.0-r343h599a50d_0
rpy2: 2.9.1-py35r343h7a9f3d7_0
```
I don't use conda but plain Ubuntu packaging -- here rpy2 is the 2.8.6 version that still uses python2:
R> library(reticulate)
R> r <- import("rpy2")
R> ri <- import("rpy2.rinterface")
R> ri
Module(rpy2.rinterface)
R>
I believe the issue is that you are trying to use another virtual python env within python. Please note that it is impossible to switch to another python environment once you are in a specific python environment. You will need to make sure all the necessary packages (e.g., tensorflow and keras) are installed in the current python environment.
@eddelbuettel Forgive me if am missing something but that only proves reticulate works.
But not that it can work in the R engine started by Python. Please see the full scenario and the linked issues for context.
@randy3k you referring to "use_condaenv("dev-ds")" ?
This is same env. Plus same happens without that line.
I understand the same Python env must be used and do not need / mean to switch.
@SemanticBeeng If you insist on mixing multiple builds and it breaks, the pieces are all yours. I never needed conda. Coincidentally, I also never had issues.
I am sure there are limits to how much of a full circle we can build here:
so yes, I can also get the 'already initialized' errors you see but if you're more careful, at least some things clearly work:
R> library(reticulate)
R> ri <- import("rpy2.rinterface")
R> ri$R_VERSION_BUILD
[[1]]
[1] "3"
[[2]]
[1] "4.1"
[[3]]
[1] ""
[[4]]
[1] 72865
R>
R> R.version.string
[1] "R version 3.4.4 (2018-03-15)"
R>
I look after rpy2 for Debian/Ubuntu but don't use it much. It appears to load as promised; I can see that certain things that would work from a Python session could run into trouble when started from R via reticulate. But the onus is on you here to demonstrate a use case.
@SemanticBeeng The error is caused by the line nested(). It leads me to think that it is an issue of rpy2 of handling function callbacks.
Indeed nested() cases the exist/crash: the cause must be related to how the joint initialization between Python and embedded R happens.
The idea of recent joint development betweenrpy2 and reticulate was to make this work such that _same_ R instance is used. See https://github.com/rstudio/reticulate/issues/98#issuecomment-335030095.
Is there any chance am missing something?
Dirk: _"mixing multiple builds"_ same happens with default reticulate (2.9.1) and rpy2 (1.3) from conda, without me insisting on any specific versions.
_"demonstrate a use case"_ - may be missing what you mean.
Is the use case above supported?
If the same R instance is shared then control flow can go back and forth without problems for reasonable uses. See more here for reason behind wanting to share R instance: https://github.com/rstudio/reticulate/issues/98#issuecomment-374877363
If you have any versions that make the _above use case_ work, conda or no conda, then please share the exact setup to reproduce.
It is a very simple test and it is actually suggested by Laurent - is this supposed to work?
Thanks in advance.
To me it looks like this:
It all seems a wee bit contrived to me. Also the older (October etc) discussions seem to imply that this _may_ be fixed in current dev branches. Did you try those? I know I am behind with rpy2 2.8.6, but then I don't have the urgent need for r-py-r-py-r-py... nesting that you express.
I got this error with reticulate 1.6, python 3.5.3 and rpy2 2.9.2 while running nested().
RRuntimeError: Error in py_call_impl(callable, dots$args, dots$keywords) :
RuntimeError: Concurrent access to R is not allowed.
It suggests that it is an issue of rpy2 not handling function callbacks correctly.
Dirk: I "am" actually in Python to start with...
_Also the older (October etc) discussions seem to imply_ that the functionality may have been ported to 2.9* - hence me "insisting" on that version.
No, did not try the dev branches.
Will do if you say the functionality of the above use case, as per how I read Laurent, is supposed to work.
Yes, Randy: that is what I logged here as well: https://github.com/rstudio/reticulate/issues/98#issuecomment-374774512 a while back
But did not have the clean env I now have with conda.
I used conda to increase the chance that all binaries are aligned.
I meant you should report to https://bitbucket.org/rpy2/rpy2 It is not an issue of reticulate. The double initialization issue has been taken care of.
If you are in Python, what do you need reticulate for?
Need both Python and R for the functionality at hand.
I meant to say that execution of the functionality starts in Python and crosses into R.
Then the R pieces needs some big pandas data already loaded from Python and rpy2 does marshalling (data conversion).
With reticulate I could reach for it from R without conversion.
If you read through all this https://github.com/rstudio/reticulate/issues/98, Dirk, would you not say this kind of use case was the point of the work to share the embedded R instance? :)
_"double initialization issue has been taken care of."_ - is there an integration test case somewhere?
the fact that we see the error Concurrent access to R is not allowed means that it is not an initialization issue.
@eddelbuettel : import rpy2.rinterface does not initialize the embedded R, which I believe is the moment where a possible double initialization (reticulate and rpy2) would create problems. rpy2.rinterface.initr() will initialize the embedded R.
@randy3k : The double initialization issue has not been unilaterally "taken care of". R does not make its initialization status available in its C-API and what I proposed is a simple convention using environment variables as a workaround. While I manage to defend that it "seems reasonable" on the paper there is, to my knowledge, no demonstration that is it working in practice.
Consider the quick sanity check below.
Setup:
library("reticulate")
os <- import("os")
varname <- "R_SESSION_INITIALIZED"
Initialization environment variable found by R but not that the Python embedded through reticulate:
> os$environ$get(varname)
> Sys.getenv(varname)
[1] "PID=7354:NAME=\"reticulate\""
I see. @lgautier thank you for the clarification. An alternative is to check R_NilValue or another global constant in libR exists and is not equal to NULL.
While waiting for the matter of joint initialization to be sorted out...
Any insights on the cause of this
File "/.../bin/python/conda3/envs/dev-ds/lib/python3.5/site-packages/rpy2/robjects/functions.py", line 106, in __call__
res = super(Function, self).__call__(*new_args, **new_kwargs)
rpy2.rinterface.RRuntimeError: Error in py_call_impl(callable, dots$args, dots$keywords) :
ImportError: 'rpycall' is not a built-in module
Detailed traceback:
File "/development/bin/python/conda3/envs/dev-ds/lib/R/library/reticulate/python/rpytools/call.py", line 2, in <module>
import rpycall
Segmentation fault (core dumped)
for my specific environment above @lgautier ?
Anything wrong in the combination of packages (both Python and R) and libraries versions I am using?
Same error happens with latest github based version of reticulate. (https://github.com/rstudio/keras/issues/33#issuecomment-306986440)
Your last report was also about segmentation fault : https://github.com/rstudio/reticulate/issues/98#issuecomment-345480103
The segfault, which is the root issue here, can be triggered with a shorter example:
from rpy2 import robjects
robjects.r['library']('reticulate')
# Attempt at specifying the path of the Python binary, since `reticulate::use_virtualenv()`
# is broken with Python 3 (https://github.com/rstudio/reticulate/issues/194). The segfault
# occurs with or without this.
robjects.r['use_python']('/home/laurent/Desktop/software/python/py36_env/bin/python3', required=True)
robjects.r['import']('sys')
This with reticulate v1.6 (latest release).
Inspecting the stack of frames leading to the segfault indicate that this is happening in _reticulate_py_initialize(), suggesting an issue with reticulate.
Thanks for the test scenario, Laurent.
I realize the seg fault is separate from the failure to share the embedded R but this helps clarify.
I strictly need only rpy2 atm: please confirm it is supposed to work with Python 3.6.5.
On https://pypi.python.org/pypi/rpy2 say 3.5 but this suggest "yes"
https://bitbucket.org/rpy2/rpy2/src/5a9c102a47e4c20fb688f536a57e7be6a7d5330f/docker/base_rpy2/Dockerfile?at=default&fileviewer=file-view-default#Dockerfile-31
rpy2-2.9.0 was released with Python 3.5 as a target, but this is now Python 3.6 in dev (default) branch.
I have not noticed or heard about any issue with Python 3.6 and rpy2-2.9.x.
(note: this is more a rpy2 question than a reticulate question)
@lgautier , executed your test above in my conda env (above) + reticulate from github and did not get the seg fault but the same rpycall import error from rpy2
````
$ python
Python 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 18:21:58)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
from rpy2 import robjects
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: During startup -
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: Warning messages:
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: 1: package ‘methods’ was built under R version 3.4.3
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: 2: package ‘datasets’ was built under R version 3.4.3
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: 3: package ‘utils’ was built under R version 3.4.3
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: 4: package ‘grDevices’ was built under R version 3.4.3
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: 5: package ‘graphics’ was built under R version 3.4.3
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: 6: package ‘stats’ was built under R version 3.4.3
warnings.warn(x, RRuntimeWarning)
robjects.r'library'
R object with classes: ('character',) mapped to:
['reticula..., 'tools', 'stats', 'graphics', ..., 'utils', 'datasets', 'methods', 'base']
robjects.r'use_python'
R object with classes: ('character',) mapped to:
['/development/bin/python/conda3/envs/dev-d...]
robjects.r('reticulate::import("sys")')
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: Error in py_call_impl(callable, dots$args, dots$keywords) :
ImportError: 'rpycall' is not a built-in module
Detailed traceback:
File "/development/bin/python/conda3/envs/dev-ds/lib/R/library/reticulate/python/rpytools/call.py", line 2, in
import rpycall
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: In addition:
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: Warning message:
warnings.warn(x, RRuntimeWarning)
/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/rinterface/__init__.py:145: RRuntimeWarning: package ‘tools’ was built under R version 3.4.3
warnings.warn(x, RRuntimeWarning)
Traceback (most recent call last):
File "
File "/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/robjects/__init__.py", line 352, in __call__
res = self.eval(p)
File "/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/robjects/functions.py", line 178, in __call__
return super(SignatureTranslatedFunction, self).__call__(args, *kwargs)
File "/development/bin/python/conda3/envs/dev-ds/lib/python3.6/site-packages/rpy2/robjects/functions.py", line 106, in __call__
res = super(Function, self).__call__(new_args, *new_kwargs)
rpy2.rinterface.RRuntimeError: Error in py_call_impl(callable, dots$args, dots$keywords) :
ImportError: 'rpycall' is not a built-in module
Detailed traceback:
File "/development/bin/python/conda3/envs/dev-ds/lib/R/library/reticulate/python/rpytools/call.py", line 2, in
import rpycall
````
Any thoughts on what might be the cause, please?
It would be good to use same env to reproduce same issues.
Wondering if you would be kind to run with mine: take 2 mins to setup miniconda and then you have a self contained environment.
conda env create -f env.yml # from top of thread
source activate dev-ds
These kind of integration testing done without a contained environment is ... not reproducible.
I use conda because of the requirement to create _"relocatable environments"_. (http://quasiben.github.io/blog/2016/4/15/conda-spark/)
UPDATE:
>>> robjects.r['use_condaenv']('dev-ds')
rpy2.rinterface.NULL
So use_python returns a valid thing but use_condaenv returns NULL.
This seems to be a problem in reticulate.
Note that neither calls to use_* really mean to _change_ the Python environment: the Python binary used is already set to the right one.
Summary
rpy2 to reticulate causes _"ImportError: 'rpycall' is not a built-in module"_reticulate to rpy2 causes seg fault when executing in a Python process from rpy2 and _"R is already initialized"_ + exit when done from a standalone R shell.The "R is already initialized" is caused by Python unaware that environment variables are set after is has been initialized. Currently the workaround is to set the environment variable manually before.
library("reticulate")
## Set the environment variable **before** Python is initialized.
## Ideally this code from `reticulate` would be in its own `reticulate` function.
Sys.setenv(R_SESSION_INITIALIZED = sprintf('PID=%s:NAME="reticulate"', Sys.getpid()))
## Initialization only occurs when a module is imported
os <- import("os")
ri <- import("rpy2.rinterface")
ri$initr()
There are more issues though:
## Create an R vector using rpy2
v <- ri$FloatSexpVector(c(1,2,3))
## `reticulate` seems to be unsure about what to do:
v[[0]]
## Return the error message:
## Error in py_to_r.default(x) : Object to convert is not a Python object
When going the other way around (that is Python -> rpy2 -> reticulate -> Python), there is still a segfault but the issue seems to be with reticulate (or so I understand from reading quickly the code - I am not very familiar with this code).
The function initialize_python() is calling py_discover_config() (and that function seems to successfully retrieve the environment variable created by rpy2), but will attempt to initialize Python on line 118 even if it probably should be done if coming from rpy2.
In the C++ code called by line 118 we will not actually call Py_Initialize if it's already initialized (see the code here: https://github.com/rstudio/reticulate/blob/master/src/python.cpp#L1502-L1509).
In the C++ code called by line 118 we will not actually call Py_Initialize if it's already initialized (see the code here: https://github.com/rstudio/reticulate/blob/master/src/python.cpp#L1502-L1509).
OK. Thanks.Then the segfault comes from somewhere else.
@lgautier check also https://github.com/rstudio/reticulate/pull/279
The PR checks the existence of the python symbols directly rather than to use an environment variable.
In the C++ code called by line 118 we will not actually call Py_Initialize if it's already initialized (see the code here: https://github.com/rstudio/reticulate/blob/master/src/python.cpp#L1502-L1509).
OK. Thanks.Then the segfault comes from somewhere else.
Unlike R's initialization, calling Py_Initialize() a second time would be a no-op anyway, if I understand the doc: https://docs.python.org/3/c-api/init.html#c.Py_Initialize
The origin of the segfault is likely one of the other steps in initialize_python().
@lgautier check also #279
The PR checks the existence of the python symbols directly rather than to use an environment variable.
The environment variable on the Python side was, beside symmetry with the R side, also to track info to debug in case child processes get involved.
The issue here might not be with how Python is assessed to be initialized, but with what reticulate is doing around Python's initialization.
Hi! My usecase for this is that I have written a rpy2 converter module that handles converting between a pair of datastructures (one in R and one in python).
I’d now like to use that converter from an R host. Optimal would be integrating the converter system into py_to_r/r_to_py:
py_to_r.default <- function(x, ...) {
if (!inherits(x, "python.builtin.object"))
stop("Object to convert is not a Python object")
# Try using rpy2’s conversion system
try({
conv <- import('rpy2.robjects.conversion')
return(conv$py2rpy(x))
})
# get the default wrapper
x <- py_ref_to_r(x)
...
}
py_to_r.rpy2.rinterface_lib.sexp.Sexp <- function(x, ...) {
# This is the python base class for all R objects wrapped by rpy2
# It contains a pointer to an R object, just unwrap it
}
Then I could do:
# activate my converter
import('anndata2ri')$activate()
# now rpy2.robjects.conversion.py2rpy can handle AnnData objects
# call a function returning an AnnData object
sc <- import('scanpy')
paul15 <- sc$datasets$paul15()
# paul15 is now an R SingleCellExperiment
cc @LuckyMD
rpy2 seems to be handling reticulate well. The sequence R -> reticulate -> Python -> rpy2 -> R works.
The reverse (Python -> rpy2 -> R -> reticulate -> Python) is still broken (segfault). It seems to be caused by the way reticulate initializes Python. My understanding is that @randy3k has a patch to fix reticulate. Is there something that prevents its inclusion in the package?
I think we just lost track of this work. Here is the PR: https://github.com/rstudio/reticulate/pull/279. @kevinushey I think we should take a closer look at this now and merge if we don't seen anything problematic.
Sorry for the delay -- I've now merged the PR and it will be part of the upcoming reticulate release.
cc: @randy3k, in case merging of the associated PR affects Radian. Sorry for taking so long to get to it.
Thanks for merging the PR. I just checked that the current version of reticulate is working with radian.
Actually, there is still a little patch in order to make it work. It is needed to append the path system.file("python", package = "reticulate") to python's sys.path as reticulate no longer inserts it, instead, it uses PYTHONPATH to include that path.
Thanks -- do you plan to submit a PR, or should I take a look?
Is there any update on the progress here?
It seems that in the latest version it is still not available to call reticulate in rpy2.
Please forgive me if I've missed some features.
Thank you. Hope for your answer!
I just tried with a snapshot of the latest revision in the main branch and the segfault is still happening.
Most helpful comment
Hi! My usecase for this is that I have written a rpy2 converter module that handles converting between a pair of datastructures (one in R and one in python).
I’d now like to use that converter from an R host. Optimal would be integrating the converter system into
py_to_r/r_to_py:Then I could do:
cc @LuckyMD