Reticulate: use_condaenv() never uses the specified environment

[micdonato]

Hello,

I've been trying to use reticulate with my conda environments, but it seems that I am not able to load the right environment.

No matter what I put into use_condaenv() I always end up with the base conda python version.

This is what I am trying:

On the server:

[user@server-rstudio1 ~]$ which python
~/anaconda2/bin/python
[user@server-rstudio1 ~]$ python
Python 2.7.14 |Anaconda custom (64-bit)| (default, Mar 27 2018, 17:29:31) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import umap
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named umap
>>> 
[user@server-rstudio1 ~]$ source activate py27
(py27) [user@server-rstudio1 ~]$ which python
~/anaconda2/envs/py27/bin/python
(py27) [user@server-rstudio1 ~]$ python
Python 2.7.14 |Anaconda, Inc.| (default, Dec  7 2017, 17:05:42) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import umap
>>> 
(py27) [user@server-rstudio1 ~]$ 

As you can see, the base python (in ~/anaconda2/bin/python) doesn't have the umap package installed, while the python in ~/anaconda2/envs/py27/python does.

When I open RStudio, I try:

library(reticulate)
conda_list()
use_condaenv(condaenv = 'py27', required = TRUE)
# commented here because I tried a few options
# use_condaenv(condaenv = '/home/user/anaconda2/envs/py27', required = TRUE)
# use_condaenv(condaenv = '/home/user/anaconda2/envs/py36', required = TRUE)
# use_condaenv(condaenv = '/home/user/anaconda2/envs/py27')
# use_condaenv(condaenv = '/home/user/anaconda2/envs/py36')
numpy <- import('numpy')
umap <- import('umap')

py_config()

system('which python')

conda_list() correctly shows all my conda environments.

No matter what I put into use_condaenv() what I get from py_config is:

python:         /home/user/anaconda2/bin/python
libpython:      /home/user/anaconda2/lib/libpython2.7.so
pythonhome:     /home/user/anaconda2:/home/user/anaconda2
version:        2.7.14 |Anaconda custom (64-bit)| (default, Mar 27 2018, 17:29:31)  [GCC 7.2.0]
numpy:          /home/user/anaconda2/lib/python2.7/site-packages/numpy
numpy_version:  1.13.3
numpy:          /home/user/anaconda2/lib/python2.7/site-packages/numpy

python versions found: 
 /home/user/anaconda2/bin/python
 /usr/bin/python
 /home/user/anaconda2/envs/tensorflow/bin/python
 /home/user/anaconda2/envs/py27/bin/python
 /home/user/anaconda2/envs/py36/bin/python
 /home/user/anaconda2/envs/r_env/bin/python

And since umap is installed only on py27, I cannot load it with reticulate.
Any idea of what I am doing wrong?

[jjallaire]

I suspect that the environment you are trying to use doesn't have numpy in it (and we always prefer environments with NumPy to those without it). Is that the case?

[micdonato]

Hi, thanks for the reply.

No, that is not the case.

I can import numpy in that environment.

[user@server-rstudio1 ~]$ source activate py27
(py27) [user@server-rstudio1 ~]$ python
Python 2.7.14 |Anaconda, Inc.| (default, Dec  7 2017, 17:05:42) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> 
(py27) [user@server-rstudio1 ~]$  which python
~/anaconda2/envs/py27/bin/python

[jjallaire]

Then what's happening is that attempts to load the umap are failing within R for some reason (so we keep looking for a Python that we succeed with). This could be a result of a different set of environment variables in R?

To get a better error message try:

library(reticulate)
use_condaenv(condaenv = 'py27', required = TRUE)
py_run_string('import umap')

[micdonato]

Then what's happening is that attempts to load the umap are failing within R for some reason (so we >keep looking for a Python that we succeed with). This could be a result of a different set of >environment variables in R?

To get a better error message try:

library(reticulate)
use_condaenv(condaenv = 'py27', required = TRUE)
py_run_string('import umap')

returns

Error in py_run_string_impl(code, local, convert) : 
  ImportError: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/mdonato/anaconda2/envs/py27/lib/python2.7/site-packages/scipy/sparse/_sparsetools.so)

hmmmmm

[nbafrank]

Hi everyone,

I have been experiencing the same issue as @micdonato

Working on this, we found that this is related to some settings of RStudio server (see below post for explanations):

https://support.rstudio.com/hc/en-us/community/posts/200645248-Setting-up-LD-LIBRARY-PATH-for-a-rsession

Basically, the LD-LIBRARY-PATH variable which points to libstdc++.so.6 from Python does not get honored in RStudio server and Reticulate for some reason. I cannot tweak things on the Rstudio server because it would affect multiple users. Can you guys do something about this on Reticulate?

Thanks!

[jjallaire]

The LD_LIBRARY_PATH has to be set prior to the process loading so the only way to make this work is to change the configuration of the RStudio Server.

[nbafrank]

So, for us this is not a viable option, because it affects ALL users settings. Is there any way for Reticulate to solve this issue, for example through the RETICULATE_PYTHON variable? This should be pretty easy and fixable.

[jjallaire]

LD_LIBRARY_PATH has to be established before the process loads so there is nothing that can be done from within R. It has to be done by the process which actually loads R, which in this case is RStudio Server.

The RETICULATE_PYTHON variable just determines the version of Python to use, it can't force directories onto the LD_LIBRARY_PATH (again, nothing from within R can do this).

[nbafrank]

I see. It is problematic on our end, because the libstdc++.so.6 is also specified by the Python environment one uses. For example, Anaconda2 points to a version of libstdc++.so.6 that is different from the one present at a systems level, and different from the one available in another version of Python. We will try to find a workaround.

[nbafrank]

Side note: is there any way to enable user-specific settings for the LD_LIBRARY_PATH used in RStudio server? Based on the documentation, it is not intuitive that it is the case, and of course, we are worried about making changes in RStudio server that could/would affect all users. Thanks a lot for your help @jjallaire

[jjallaire]

No, it's a global setting. RStudio Server Pro has support for the PAM session API, which you could use to implement user-specific LD_LIBRARY_PATH. I noted that your GitHub affiliation lists Stanford University -- RStudio Server Pro is free for teaching and 1/2 price for academic research if that helps here.

[nbafrank]

We are in touch with customer support (we have a PRO license). Hopefully, we should be able to solve this. Thanks!

[jjallaire]

Okay, great. You should be able to define user-specific profiles files that set LD_LIBRARY_PATH or alternatively use PAM to provide user-specific environment variables (I don't know all of the details on that but I know it's possible). These docs are likely to be relevant: http://docs.rstudio.com/ide/server-pro/r-sessions.html#customizing-session-launches

[mjsteinbaugh]

I can confirm that I'm seeing the same issue with libstdc++.so.6 using r-base 3.5.1 from conda-forge

[bowowski]

On ubuntu 16.04
added directory containing libstdc++.so.6 to rserver.conf
use_condaenv now accepts the environment, but I get an ImportError later:
`> library(keras)

library(tensorflow)
library(reticulate)
conda_list()
name python
1 anaconda3 /home/rstudio/anaconda3/bin/python
2 r-tensorflow /home/rstudio/anaconda3/envs/r-tensorflow/bin/python
3 tensorflow_p36_cpu /home/rstudio/anaconda3/envs/tensorflow_p36_cpu/bin/python
use_condaenv("tensorflow_p36_cpu")
numpy=import("numpy")
umap=import("umap")
py_config()
python: /home/rstudio/anaconda3/envs/tensorflow_p36_cpu/bin/python
libpython: /home/rstudio/anaconda3/envs/tensorflow_p36_cpu/lib/libpython3.6m.so
pythonhome: /home/rstudio/anaconda3/envs/tensorflow_p36_cpu:/home/rstudio/anaconda3/envs/tensorflow_p36_cpu
version: 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51) [GCC 7.2.0]
numpy: /home/rstudio/anaconda3/envs/tensorflow_p36_cpu/lib/python3.6/site-packages/numpy
numpy_version: 1.15.1
keras: /home/rstudio/anaconda3/envs/tensorflow_p36_cpu/lib/python3.6/site-packages/keras

python versions found:
/home/rstudio/anaconda3/envs/tensorflow_p36_cpu/bin/python
/home/rstudio/anaconda3/envs/r-tensorflow/bin/python
/usr/bin/python
/usr/bin/python3
/home/rstudio/anaconda3/bin/python

foo=dataset_mnist()
Error: ImportError: Traceback (most recent call last):
File "/home/rstudio/anaconda3/envs/tensorflow_p36_cpu/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/rstudio/anaconda3/envs/tensorflow_p36_cpu/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/rstudio/anaconda3/envs/tensorflow_p36_cpu/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/home/rstudio/anaconda3/envs/tensorflow_p36_cpu/lib/python3.6/imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
File "/home/rstudio/anaconda3/envs/tensorflow_p36_cpu/lib/python3.6/imp.py", line 343, in load_dynamic
return _load(spe`

[jsta]

I was able to get use_condaenv() to use the specified environment by launching RStudio from a terminal with the specified environment already activated:

source activate something
rstudio something.Proj

Still can't get use_condaenv() to switch into a non-activated environment.

[kateruksha]

This thing worked:
By setting the value of the RETICULATE_PYTHON environment variable to a Python binary. Note that if you set this environment variable, then the specified version of Python will always be used (i.e. this is prescriptive rather than advisory). To set the value of RETICULATE_PYTHON, insert Sys.setenv(RETICULATE_PYTHON = PATH) into your project’s .Rprofile, where PATH is your preferred Python binary.

[f0nzie]

The solution by @kateruksha worked great in my Linux Debian 9.

You have to indicate the Python interpreter under the environment bin folder:
Sys.setenv(RETICULATE_PYTHON = "/home/superuser/anaconda3/envs/pyres/bin/python")

Then launch your reticulate call to use the conda environment:

library(reticulate)

# select environment pyres
reticulate::use_condaenv("pyres", required = TRUE)
reticulate::py_config()
py_run_string('import pandas')
py_run_string('import numpy')

WHich will give you this message:

python:         /home/superuser/anaconda3/envs/pyres/bin/python
libpython:      /home/superuser/anaconda3/envs/pyres/lib/libpython3.6m.so
pythonhome:     /home/superuser/anaconda3/envs/pyres:/home/superuser/anaconda3/envs/pyres
version:        3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)  [GCC 7.3.0]
numpy:          /home/superuser/anaconda3/envs/pyres/lib/python3.6/site-packages/numpy
numpy_version:  1.16.2

NOTE: Python version was forced by RETICULATE_PYTHON

[jikutlenova]

Hi,
while using reticulate. I am unable to load numpy. It always says numpy : [not found]. when i execute the following chunk
```{r}
Sys.setenv(RETICULATE_PYTHON="C:/Users/priya/Anaconda3/envs/pyres/bin/python")
library(reticulate)
reticulate::use_condaenv("pyres", required = TRUE)
reticulate::py_config()
py_run_string('import numpy')
py_run_string('import pandas')

reticulate::use_python("C:/Users/priya/Anaconda3/envs/pyres/python.exe", required = TRUE)

```
python: C:/Users/priya/Anaconda3/envs/pyres/python.exe
libpython: C:/Users/priya/Anaconda3/envs/pyres/python37.dll
pythonhome: C:\Users\priya\ANACON~1\envs\pyres
version: 3.7.2 (default, Feb 21 2019, 17:35:59) [MSC v.1915 64 bit (AMD64)]
Architecture: 64bit
numpy: [NOT FOUND]

NOTE: Python version was forced by use_python function

[mfansler]

I had a similar issue trying to work the Seurat package, which uses umap-learn via reticulate. I found loading reticulate and calling use_condaenv prior to loading Seurat gets around the issue. Namely,

library(reticulate)
use_condaenv('umap', required=TRUE)
library(Seurat)

# do a bunch of other stuff

seurat_obj <- RunUMAP(seurat_obj)
# works

[limpbizkit6]

I had a similar issue trying to work the Seurat package, which uses umap-learn via reticulate. I found loading reticulate and calling use_condaenv prior to loading Seurat gets around the issue. Namely,

library(reticulate)
use_condaenv('umap', required=TRUE)
library(Seurat)

# do a bunch of other stuff

seurat_obj <- RunUMAP(seurat_obj)
# works

After spending an embarrassingly long time trying to get umap-learn to work, this is literally the ONLY thing that worked for me and was only available since I decided to research the issue again and your reply from a day ago popped up. Thank you SO much!

[Ceazy-Gentleman]

Thank you! I had solved my problems by your ways.

[cjwoodfield]

Hi All.

I'm experiencing a similar problem but with the bcpandas package...

My colleague has written a great function using bcpandas - the problem is installing it so that it's recognized and can be used by the R Reticulate package

After a fresh Install of miniconda the packages we need to run his code arent available, so I install them...

conda_install(envname = "C:\\Users\\xxxxx\\AppData\\Local\\r-miniconda\\envs\\r-reticulate", packages="pandas")
conda_install(envname = "C:\\Users\\xxxxx\\AppData\\Local\\r-miniconda\\envs\\r-reticulate", packages="pyodbc")
conda_install(envname = "C:\\Users\\xxxxx\\AppData\\Local\\r-miniconda\\envs\\r-reticulate", packages="bcpandas")

After installation however - bcpandas is not available even though I can see it has been installed into the site packages folder,

Sys.setenv(RETICULATE_PYTHON = "C:\\Users\\xxxxxx\\AppData\\Local\\r-miniconda\\envs\\r-reticulate\\bin\\python")
use_condaenv("C:\\Users\\xxxxxx\\AppData\\Local\\r-miniconda\\envs\\r-reticulate",required = T)

> py_module_available("numpy")
[1] TRUE
> py_module_available("glob")
[1] TRUE
> py_module_available("os")
[1] TRUE
> py_module_available("numpy") 
[1] TRUE
> py_module_available("subprocess")
[1] TRUE
> py_module_available("pyodbc") 
[1] TRUE
> py_module_available("pandas")
[1] TRUE
> py_module_available("bcpandas")
[1] FALSE

Do you know why this might be the case?

[kevinushey]

Can you confirm that reticulate::py_config() shows you've successfully loaded the requested Python environment?

Can you also try running something like:

reticulate::py_run_string("import bcpandas")

as this may provide extra information as to why the attempt to import the module is failing.

[cjwoodfield]

Sure!

Im not sure actually I needed the \bin\python in my Sys.setenv() call.

Sys.setenv(RETICULATE_PYTHON = "C:\\Users\\xxxxxx\\AppData\\Local\\r-miniconda\\envs\\r-reticulate")
use_condaenv("C:\\Users\\xxxxxx\\AppData\\Local\\r-miniconda\\envs\\r-reticulate",required = T)

> reticulate::py_config()
python:         C:/Users/xxxxxx/AppData/Local/r-miniconda/envs/r-reticulate/python.exe
libpython:      C:/Users/xxxxxx/AppData/Local/r-miniconda/envs/r-reticulate/python36.dll
pythonhome:     C:/Users/xxxxxx/AppData/Local/r-miniconda/envs/r-reticulate
version:        3.6.11 (default, Aug  5 2020, 19:41:03) [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:/Users/xxxxxx/AppData/Local/r-miniconda/envs/r-reticulate/Lib/site-packages/numpy
numpy_version:  1.19.4

... I think this is loaded correctly. So here's the more detailed error message...

> reticulate::py_run_string("import bcpandas")
Error in py_run_string_impl(code, local, convert) : 
  OSError: [WinError 6] The handle is invalid

Detailed traceback: 
  File "<string>", line 1, in <module>
  File "C:\Users\xxxxxx\Documents\R\win-library\4.0\reticulate\python\rpytools\loader.py", line 31, in _import_hook
    level=level
  File "C:\Users\xxxxxx\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\bcpandas\__init__.py", line 14, in <module>
    run(["bcp", "-v"], stdout=DEVNULL, stderr=DEVNULL)
  File "C:\Users\xxxxxx\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\xxxxxx\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\subprocess.py", line 687, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "C:\Users\xxxxxx\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\subprocess.py", line 910, in _get_handles
    p2cread = _winapi.GetStdHandle(_winapi.STD_INPUT_HANDLE)

[kevinushey]

Thanks! This looks like it may be an issue in the bcpandas module. I found this note on Stackoverflow:

https://stackoverflow.com/questions/40108816/python-running-as-windows-service-oserror-winerror-6-the-handle-is-invalid

which basically suggests that one use stdin=DEVNULL to avoid attempting to resolve a (potentially missing) stdin handle.

[cjwoodfield]

Ok thanks - good to know as this problem was driving me a little mad!

In that case what would be the next steps to resolve? Would this require action by the developer of bcpandas?

[kevinushey]

@cjwoodfield: I believe this issue should now be fixed in version 1.0.1 of bcpandas. Can you try upgrading and let me know if that appears to be the case?

[cjwoodfield]

Yes I can import it now. Thanks very much for your help!

[cjwoodfield]

Hi Kevin - Thanks to your fix I can import bcpandas - although when running one of the functions - to_sql - I'm getting a similar handle is invalid error as last time.

Wondering whether this issue and the previous issue are linked?

To elaborate ….

Here is a simplified version of our R App using reticulate...

rm(list=ls())

library(reticulate)

Script_Dir = gsub("/","\\\\",dirname(rstudioapi::getActiveDocumentContext()$path))
setwd(Script_Dir)

py_run_file("Upload_to_SQL.py")

And Here is the Python file we're running...

import pandas as pd
import glob, os, pyodbc, numpy, subprocess
from bcpandas import SqlCreds, to_sql

sql_config  = SqlCreds('xxxxxx,
                        'xxxxx',
                        'xxxxx',
                        'xxxxx')
driver='{ODBC Driver 17 for SQL Server}'
server = 'xxxxx'    
dbase='xxxxx'
usern='xxxxx'
passw='xxxxx'

root=r'C:\Users\woodfieldc\Desktop\TEST BCP'

df = pd.read_csv(root + '\\FACTS.CSV' ,delimiter=';',header=None)

tablename="Temp_" +"_Facts"

to_sql(df, tablename, sql_config, index=True, if_exists='replace')

And this give a similar Handle is invalid error as before...

Error in py_run_file_impl(file, local, convert) : 
  OSError: [WinError 6] The handle is invalid

Detailed traceback: 
  File "<string>", line 29, in <module>
  File "C:\Users\WOODFI~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\bcpandas\main.py", line 348, in to_sql
    bcp_path=bcp_path,
  File "C:\Users\WOODFI~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\bcpandas\utils.py", line 114, in bcp
    ret_code = run_cmd(bcp_command)
  File "C:\Users\WOODFI~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\site-packages\bcpandas\utils.py", line 231, in run_cmd
    proc = Popen(cmd, stdout=PIPE, stderr=PIPE, encoding="utf-8", errors="utf-8", shell=with_shell,)
  File "C:\Users\WOODFI~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\subprocess.py", line 687, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "C:\Users\WOODFI~1\AppData\Local\R-MINI~1\envs\R-RETI~1\lib\subprocess.py", line 910, in _get_handles
    p2cread = _winapi.GetStdHandle(_winapi.STD_INPUT_HANDLE)

I have also notified yehoshuadimarsky on the bcpandas GitHub

[cjwoodfield]

As a further comment - The code above will create a Table on SQL - but not send data

[kevinushey]

This probably requires another similar use of stdin=DEVNULL in the use of Popen.