Reticulate: Py_install doesn't use python version set by use_python

[lazariv]

I have a problem with installing different modules using py_install() for tensorflow package. Probably because py_install() uses python version, found by

reticulate:::python_unix_binary("python")
[1] "/usr/bin/python"

It looks up only the following locations locations <- file.path(c("/usr/bin", "/usr/local/bin", path.expand("~/.local/bin")), bin) and doesn't take into account the locations, strictly set by use_python("/sw/installed/Python/3.6.4-foss-2018a/bin/python3", required = TRUE) or Sys.setenv(RETICULATE_PYTHON = "/sw/installed/Python/3.6.4-foss-2018a/bin/python3")

> py_config()
python:         /sw/installed/Python/3.6.4-foss-2018a/bin/python3
libpython:      /sw/installed/Python/3.6.4-foss-2018a/lib/libpython3.6m.so
pythonhome:     /sw/installed/Python/3.6.4-foss-2018a:/sw/installed/Python/3.6.4-foss-2018a
version:        3.6.4 (default, Jun  7 2018, 17:18:01)  [GCC 6.4.0]
numpy:          /sw/installed/TensorFlow/1.8.0-foss-2018a-Python-3.6.4-CUDA-9.2.88/lib/python3.6/site-packages/numpy
numpy_version:  1.14.4
keras:          /sw/installed/Keras/2.2.0-foss-2018a-Python-3.6.4/lib/python3.6/site-packages/Keras-2.2.0-py3.6.egg/keras

python versions found: 
 /sw/installed/Python/3.6.4-foss-2018a/bin/python3
 /sw/installed/Python/3.6.4-foss-2018a/bin/python
 /usr/bin/python

It would be great, if the same python version would be used across different reticulate-functions?!

````

sessionInfo()
R version 3.4.4 (2018-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server 7.4 (Maipo)

Matrix products: default
BLAS/LAPACK: /software/haswell/OpenBLAS/0.2.20-GCC-6.4.0-2.28/lib/libopenblas_haswellp-r0.2.20.so

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=de_DE.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=de_DE.UTF-8 LC_MESSAGES=en_US.UTF-8 LC_PAPER=de_DE.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] keras_2.2.0.9001 reticulate_1.10.0.9003 tensorflow_1.9

loaded via a namespace (and not attached):
[1] Rcpp_0.12.19 lattice_0.20-35 zeallot_0.1.0 grid_3.4.4 R6_2.3.0 jsonlite_1.5 magrittr_1.5 tfruns_1.4
[9] whisker_0.3-2 Matrix_1.2-12 generics_0.0.1 tools_3.4.4 yaml_2.2.0 compiler_3.4.4 base64enc_0.1-3
```

[cderv]

Hi,

I also encountered this issue .

py_install is the recommended way to install python package in a "r-reticulate" env

I am currently setting up a Rstudio cluster for use with python and RStudio 1.2. By default, the system has python 2 in /usr/bin/python and we want users to use python3 in /applis/python/bin/python3. Setting RETICULATE_PYTHON is a great way to do that, but as not taken into account by py_install, python2 is used instead. use_python does not help as explained above.

Is there any news on this ?
is this something that will be fixed or is it expected to work that way and we are missing something ?

Thanks a lot !

[kevinushey]

Could you share a reproducible example just so we have a common set of code to work off of? In theory py_install() should delegate to virtualenv_install(), which then uses the copy of pip within that virtual environment.

[cderv]

The theory is what I had in mind but the source code seems to say otherwise.

If on linux with no conda available (use_conda is FALSE), system python is used and if found, python3 is not even tried
https://github.com/rstudio/reticulate/blob/c664a6c5ae4e77d62fd5a115bb5505cbd9d57ac4/R/install.R#L66-L74
So here, /usr/bin/python from the system is used on the linux server

From there, pip and virtualenv I looked for to see if installed. However, on my system these are not available for /usr/bin/python. This means have_pip and have_virtualenv are FALSE. They are available for the python3 installed in /applis/python/bin/ and defined in RETICULATE_PYTHON or set with use_python()

Due to these tools not found, install_command is not NULL but equal to c("pip", "virtualenv") (I am on redhat).
So the code stop here throwing an error, just before delegation to virtualenv_install()
https://github.com/rstudio/reticulate/blob/c664a6c5ae4e77d62fd5a115bb5505cbd9d57ac4/R/install.R#L130-L147

All these check does not happen use the version of python indicated by py_config() and gets from use_python() or RETICULATE_PYTHON env var.

if pip and virtualenv exists on the system python, these check should pass but I wonder what those checks with python_unix_binary() if virtualenv_install does the job 🤔

I'll work on an example tomorrow to illustrate but you'll need reproduce on a similar environment 😉

[cderv]

Step to reproduce the environment

1. Linux machine with only python2, no pip or virtualenv available (centos 7)
2. Install python3 following RStudio Guidelines on support site
   
   
   
   • install to a custom location like /opt/python
   
   
   • install _pip_ and _virtualenv_ for python3
   
   

Reticulate find python2 by default and no pip or virtualenv is available

reticulate::py_discover_config()
#> python:         /usr/bin/python
#> libpython:      /usr/lib64/libpython2.7.so.1.0
#> pythonhome:     /usr:/usr
#> version:        2.7.5 (default, Aug  4 2017, 00:39:18)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
#> numpy:           [NOT FOUND]
reticulate::py_module_available("pip")
#> [1] FALSE
reticulate::py_module_available("virtualenv")
#> [1] FALSE

Restart R and now use python3 with with reticulate. Pip and virtualenv are available

reticulate::use_python("/applis/Python/bin/python3.6")
reticulate::py_discover_config()
#> python:         /applis/Python/bin/python3.6
#> libpython:      /applis/Python/lib/libpython3.6m.so
#> pythonhome:     /applis/Python:/applis/Python
#> version:        3.6.5 (default, Apr 17 2019, 09:41:01)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
#> numpy:           [NOT FOUND]
#> 
#> python versions found: 
#>  /applis/Python/bin/python3.6
#>  /usr/bin/python
reticulate::py_module_available("pip")
#> [1] TRUE
reticulate::py_module_available("virtualenv")
#> [1] TRUE

python::use_python() helps find the correct environment.

Try installing with py_install that should install within a

a common default environment (“r-reticulate”) across the installation of distinct Python packages.

as the doc says.

reticulate::use_python("/applis/Python/bin/python3.6")
reticulate::py_discover_config()
#> python:         /applis/Python/bin/python3.6
#> libpython:      /applis/Python/lib/libpython3.6m.so
#> pythonhome:     /applis/Python:/applis/Python
#> version:        3.6.5 (default, Apr 17 2019, 09:41:01)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
#> numpy:           [NOT FOUND]
#> 
#> python versions found: 
#>  /applis/Python/bin/python3.6
#>  /usr/bin/python
reticulate::py_config()
#> python:         /applis/Python/bin/python3.6
#> libpython:      /applis/Python/lib/libpython3.6m.so
#> pythonhome:     /applis/Python:/applis/Python
#> version:        3.6.5 (default, Apr 17 2019, 09:41:01)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
#> numpy:           [NOT FOUND]
#> 
#> python versions found: 
#>  /applis/Python/bin/python3.6
#>  /usr/bin/python
reticulate::py_module_available("pip")
#> [1] TRUE
reticulate::py_module_available("virtualenv")
#> [1] TRUE
reticulate::py_install("numpy")
#> Error: Prerequisites for installing Python packages not available.
#> 
#> Please install the following Python packages before proceeding: pip, virtualenv

We get the error of pip and virtualenv not being available. See my comment above on how it is related to python2 from the system.

I can create a virtualenv and install correctly with these commands.

reticulate::use_python("/applis/Python/bin/python3.6")
reticulate::virtualenv_create("r-reticulate")
reticulate::virtualenv_install("r-reticulate", "numpy")

But even with that I can't install with py_install for the same reason as comment above, because use_python is not used during the check phase of py_install()

Sys.setenv(RETICULATE_PYTHON_ENV = file.path(reticulate::virtualenv_root(),"r-reticulate"))
reticulate::use_virtualenv()
reticulate::py_discover_config()
#> python:         /home/bureau.si.interne/dervieuxchr/.virtualenvs/r-reticulate/bin/python
#> libpython:      /applis/Python/lib/libpython3.6m.so
#> pythonhome:     /applis/Python:/applis/Python
#> version:        3.6.5 (default, Apr 17 2019, 09:41:01)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
#> numpy:           [NOT FOUND]
#> 
#> NOTE: Python version was forced by RETICULATE_PYTHON_ENV
reticulate::py_module_available("numpy")
#> [1] FALSE
reticulate::py_install(packages = "numpy")
#> Error: Prerequisites for installing Python packages not available.
#> 
#> Please install the following Python packages before proceeding: pip, virtualenv
reticulate::virtualenv_install(packages = "numpy")
#> Using virtual environment '/home/bureau.si.interne/dervieuxchr/.virtualenvs/r-reticulate' ...
reticulate::py_module_available("numpy")
#> [1] TRUE

I hope it helps illustrate the issue, even if not so obvious to reproduce. The key is having a system python2 installation without pip or virtualenv available.

py_install checks for those, and I don't see why because virtualenv_install is working correctly without.

Thanks.

[kevinushey]

This is mainly because reticulate tries to find a pip or virtualenv binary in some default locations; e.g.

https://github.com/rstudio/reticulate/blob/2a2b7d14ad7538aeb9427f327fff3c21d1508aa2/R/install.R#L81

and this will of course fail if the default system Python does not have these utilities installed (or they have not otherwise been installed to /usr/local/bin).

I _think_ that py_install() should instead delegate to the versions of pip / virtualenv that would be installed in the requested environment if available, and fall back to the system versions otherwise.

[cderv]

I guess with the change from Python2 to Python3, this is something you'll be fixing ?

Currently, it is difficult to use reticulate on our RSP servers with both Python2 and Python3. In particular, user can't use install_keras or install_tensorflow because it uses py_install

I think that py_install() should instead delegate to the versions of pip / virtualenv that would be installed in the requested environment if available, and fall back to the system versions otherwise.

It is what I would expect. Use of pip and virtualenv from the specified python version.

[kevinushey]

We're planning to make this sort of change soon, as part of https://github.com/rstudio/reticulate/pull/598.

[kevinushey]

py_install() now uses the environment associated with the version of Python currently initialized, when no environment has been explicitly specified:

https://github.com/rstudio/reticulate/blob/6bbc34a0137c8f1032ead7011af18674588f7d72/R/install.R#L74-L94

I believe this resolves this issue.

[cderv]

I can confirm my example above is now working fine !
This works now nicely after a use_virtualenv(), even if my system python doesn't have pip and virtualenv. Thanks a lot !

[kevinushey]

Glad to hear it :-)