reticulate::py_install() not detecting pip and virtualenv when using python3 with venv?

[cboettig]

I have python3 set up with venv in /opt/venv, (python3 -m venv /opt/venv), and /opt/venv/bin at the top of my path. py_config() shows:

> py_config()
python:         /opt/venv/bin/python
libpython:      /usr/lib/python3.5/config-3.5m-x86_64-linux-gnu/libpython3.5.so
pythonhome:     /usr:/usr
version:        3.5.3 (default, Sep 27 2018, 17:25:39)  [GCC 6.3.0 20170516]
numpy:          /opt/venv/lib/python3.5/site-packages/numpy
numpy_version:  1.16.3

but py_install() complains it cannot find pip or virtualenv:

> library(reticulate)
> py_install("pandas")
Error: Prerequisites for installing Python packages not available.

Please install the following Python packages before proceeding: pip, virtualenv

even though pip is on the PATH, from bash:

$ which pip
/opt/venv/bin/pip
$ pip --version
pip 9.0.1 from /opt/venv/lib/python3.5/site-packages (python 3.5)

How do I get reticulate to find pip?

I also see that if I do virtualenv_create() I see:

virtualenv_create("venv")
Creating virtual environment 'venv' ...
Using python: /usr/bin/python3.5
Error: tools for managing Python virtual environments are not installed.

Install pip with:
$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ /usr/bin/python3.5 get-pip.py

Install virtualenv with:
$ /usr/bin/python3.5 -m pip install --upgrade --user virtualenv

This is strange as I already have pip on the path and have venv installed. If I install virtualenv with pip as instructed by reticulate, the second message goes away, but (re)installing pip as directed upgrades my pip:

Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-19.1

but still reticulate cannot find it. What am I missing to make reticulate find pip under python3?

---

sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)

Matrix products: default
BLAS/LAPACK: /usr/lib/libopenblasp-r0.2.19.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] reticulate_1.12

loaded via a namespace (and not attached):
[1] compiler_3.6.0  Matrix_1.2-17   Rcpp_1.0.1      grid_3.6.0     
[5] jsonlite_1.6    lattice_0.20-38
> 

[yuvipanda]

I don't know much R, but from reading the source at https://github.com/rstudio/reticulate/blob/master/R/virtualenv.R#L189, I wonder if we need to set RETICULATE_PYTHON_ENV to /opt/venv? I don't know if it looks in PATH to see if it's already in a virtualenv...

[cboettig]

@yuvipanda Thanks! I tried that too but no luck... (also tried setting use_virtualenv("opt/venv") but no luck there either)

From the docs it sounds like it looks for pip on the PATH, and /opt/venv/bin/ is already on the PATH so I'm not clear where it goes wrong. perhaps more splunking into py_install() sources is gonna be needed..

[yuvipanda]

@cboettig yeah, someone who actually knows R is gonna have more luck than me figuring this out :) Let me know how I can help!

[kevinushey]

I think reticulate needs to stop trying to find and use the command-line utilities pip and instead invoke it as a module (e.g. /path/to/python -m pip <...>).

[cboettig]

Thanks @kevinushey , I think that would be great (and would resolve https://github.com/rstudio/reticulate/issues/499 as well I think)

[jimhester]

I am running into a very similar issue to this, it can be reproduce easily with RStudio docker containers for ubuntu xenial. I am explicitly installing python3-venv, but reticulate does not find it.

docker run -it rstudio/r-base:3.6-xenial /bin/bash
apt update
apt install -y python3-venv python3-pip
R -e 'install.packages("reticulate", repo = "https://cloud.r-project.org")' \
  -e 'reticulate::virtualenv_create()'

# Error: Tools for managing Python virtual environments are not installed.
#
# Install virtualenv with:
# $ sudo apt-get install python3-venv
# 
# Execution halted

[kevinushey]

@jimhester: are you still seeing this issue? If so, do you also see it with the development version of reticulate?

[cderv]

I wonder if this is not related to #368 also where py_install seemed to use system's pip and virtualenv instead of the one from the current python inside the active virtualenv.

[kevinushey]

Possibly, but either way that should be fixed in the development version.

[cderv]

I am currently testing this! I'll tell you very soon.

EDIT: I can confirm that my previous examples is working fine . Thanks !

[jimhester]

It worked with

docker run -it rstudio/r-base:3.6-xenial /bin/bash
apt update
apt install -y python3-venv python3-pip python3-virualenv
R -e 'install.packages("remotes", repo = "https://cloud.r-project.org")' -e 'remotes::install_github("rstudio/reticulate")' \
  -e 'reticulate::virtualenv_create()'

Note I had to add python3-virtualenv as well as python3-venv, it did _not_ work with just python3-venv. Perhaps python3-virtualenv should be added to the reticulate error message?

[chasemc]

Thanks for documenting this.

I can confirm version 1.14 was doing the same for me. Installed commit 72f3dce and it works now.

[mikldk]

I did this

      - name: Install python3-venv
        if: runner.os == 'Linux'
        run: |
          sudo apt-get install -y python3-venv python3-pip python3-setuptools
          sudo -H pip3 install virtualenv

and later

        run: |
          library(reticulate)
          reticulate::py_install("sympy", method = "auto", conda = "auto")
        shell: Rscript {0}

to get it to work.

Didn't see this issue until too late, so other approaches here might work, too.

[eddelbuettel]

I was a little surprised to see just how how much space all this conda biz brought in by reticulate uses so I now set

## reticulate, please use my system python
RETICULATE_PYTHON=/usr/bin/python3  

but I do admit that I only need NumPy for some sparse matrix tests and not sympy. But I am pretty sure that in my case the OS-supplied one would be good enough too.

[c1au6i0]

RETICULATE_PYTHON=/usr/bin/python3

Yes, including Sys.setenv(RETICULATE_PYTHON = "/usr/local/bin/python3") in the .Rprofile of my project solves the issue for me.

[cboettig]

@kevinushey I think we can close this as the originally reported bug is now fixed.

Using virtualenvs still feels remarkably fragile though. If I do not hardwire my virtualenv location in RETICULATE_PYTHON_VENV (as @yuvipanda notes above), then on loading reticulate py_config() finds only my system python, as expected:

> library(reticulate)
ph> py_config()
python:         /usr/bin/python3
libpython:      /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so
pythonhome:     //usr://usr
version:        3.6.9 (default, Nov  7 2019, 10:44:02)  [GCC 8.3.0]
numpy:           [NOT FOUND]

python versions found: 
 /usr/bin/python3
 /usr/bin/python

What is not expected is that I now appear stuck. Adding a use_virtualenv("/opt/venv") has no effect. If I am careful to call use_virtualenv() before calling py_config(), then it successfully sets my virtualenv first.

As a result, I cannot figure out how to configure a system that can both (a) start users off with a default virtualenv, while also (b) allowing users to switch their virtualenv once they are in the R session. (Just mentioning it here both to continue the active thread and as it seems related to the broader issue here about how reticulate is approaching virtualenv, but happy to move to a new issue thread if that would be better).

[kevinushey]

Calling py_config() initializes Python, and after Python is initialized you will no longer be able to change the active Python session without restarting R. You might want to use py_discover_config() instead.

[cboettig]

:+1: Thanks, that's perfect.

Also I've discovered that if I set WORKON_HOME instead of RETICULATE_PYTHON_ENV, I can simultaneously have a system that can (a) find my 'default' virtualenv, while still not preventing the user from choosing to set a different virtualenv at the beginning of the session, so that is quite nice.

I'm not entirely clear on which virualenv from inside WORKON_HOME it chooses, but it seems to choose alphabetically. I suppose that's fine.

Anyway, thanks for all the help and the impressive tooling you've all built here!