Reticulate: Possible to change libpython path in `py_discover_config()`?

[stephaniehicks]

I'm trying to change the libpython path as listed in py_discover_config(). I know you can force a change in the python path with

Sys.setenv('RETICULATE_PYTHON'='/yourpathhere/bin/python')
py_discover_config()

But is it possible to force a change in the library path (libpython)? Looking at line 333 in the config.R file, it looks like it should be possible to set config$LIBPL, but I couldn't figure out how based on the vignettes?

[jjallaire]

No, there isn't a way to change the path to libpython (that's something we deduce from the python binary by interrogating it). It's not 100% foolproof though so if there's a scenario where we don't successfully detect the path let us know and we can perhaps work around it.

[stephaniehicks]

I'm working on our cluster and I have installed sklearn in my home directory (/users/shicks1/.local/lib/python2.7/site-packages)

[compute-043 /users/shicks1]$ module load python/2.7.9
[compute-043 /users/shicks1]$ python
Python 2.7.9 (default, May  6 2015, 15:41:33) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sklearn
>>> import sys
>>> print '\n'.join(sys.path)
/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7
/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python27.zip
/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7/plat-linux2
/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7/lib-tk
/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7/lib-old
/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7/lib-dynload
/users/shicks1/.local/lib/python2.7/site-packages
/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7/site-packages

When I load the reticulate package in R, the libpython points to /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/libpython2.7.so[NOT FOUND]. I saw on issue 227 that it will always prefer a python path with numpy to one that doesn't have it. numpy is installed in this path /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7/site-packages/numpy, but not sklearn. Hypothetically, if I could force the libpython path to something else, would that solve the problem?

> library(reticulate)
> py_discover_config() # gives configuration details
python:         /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/bin/python
libpython:      /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/libpython2.7.so[NOT FOUND]
pythonhome:     /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9:/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9
version:        2.7.9 (default, May  6 2015, 15:41:33)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)]
numpy:          /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7/site-packages/numpy
numpy_version:  1.9.2

python versions found: 
 /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/bin/python
 /usr/bin/python
> sessionInfo()
R version 3.5.0 Patched (2018-04-30 r74679)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server release 6.9 (Santiago)

Matrix products: default
BLAS: /jhpce/shared/jhpce/core/conda/miniconda-3/envs/svnR-3.5/R/3.5/lib64/R/lib/libRblas.so
LAPACK: /jhpce/shared/jhpce/core/conda/miniconda-3/envs/svnR-3.5/R/3.5/lib64/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] reticulate_1.8

loaded via a namespace (and not attached):
[1] compiler_3.5.0  Matrix_1.2-14   tools_3.5.0     Rcpp_0.12.17   
[5] grid_3.5.0      jsonlite_1.5    lattice_0.20-35

Any advice is greatly appreciated. Thank you JJ!

[jjallaire]

No, changing libpython wouldn't work here as that refers to the shared python library not the sklearn library. I'm not sure why sklearn isn't getting picked up from your local directory. Do you need to set the PYTHONPATH environment variable?

[stephaniehicks]

Forgive my ignorance, but set $PYTHONPATH environmental variable to what? This is currently what the variable returns

[compute-044 /users/shicks1]$ module load python/2.7.9
[compute-044 /users/shicks1]$ $PYTHONPATH
-bash: /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7: is a directory

[jjallaire]

PYTHONPATH tells Python where to find modules (see
https://users-cs.au.dk/chili/PBI/pythonpath.html) Since you've got modules
installed in your home directory you may need to set that variable to let
it know where to find them. Just guessing here as I haven't seen your exact
configuration/scenario before.

On Fri, Jun 22, 2018 at 1:39 PM Stephanie Hicks [e-mail]
wrote:

Forgive my ignorance, but set $PYTHONPATH environmental variable to what?
This is currently what the variable returns

[compute-044 /users/shicks1]$ module load python/2.7.9
[compute-044 /users/shicks1]$ $PYTHONPATH
-bash: /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/python2.7: is a directory

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rstudio/reticulate/issues/291#issuecomment-399521899,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGXx_6HMwQgbanzImDMnBf92JXZggLVks5t_SvkgaJpZM4Uv3cX
.

[stephaniehicks]

Hmm... I just noticed there is no libpython2.7.so file where libpython is looking (/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/libpython2.7.so[NOT FOUND]) but rather a libpython2.7.a file. I'm assuming that's the problem?

[compute-061 /users/shicks1]$ cd /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/
[compute-061 /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib]$ ls
libpython2.7.a  pkgconfig  python2.7

I did a quick google search of libpython2.7.so and libpython2.7.a, but it wasn't exactly clear to me what the difference was? I'm guessing it has something to do with the version of python that's installed on the cluster?

[jjallaire]

That implies that there is no shared library for Python but rather a static
library (which means that you can’t link R/reticulate to it)

On Fri, Jun 22, 2018 at 2:12 PM Stephanie Hicks [e-mail]
wrote:

Hmm... I just noticed there is no libpython2.7.so file where libpython is
looking (/jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/
libpython2.7.so[NOT FOUND]) but rather a libpython2.7.a file. I'm
assuming that's the problem?

[compute-061 /users/shicks1]$ cd /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib/
[compute-061 /jhpce/shared/community/compiler/gcc/4.4.7/python/2.7.9/lib]$ ls
libpython2.7.a pkgconfig python2.7

I did a quick google search of libpython2.7.so and libpython2.7.a, but it
wasn't exactly clear to me what the difference was? I'm guessing it has
something to do with the version of python that's installed on the cluster?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/rstudio/reticulate/issues/291#issuecomment-399531597,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGXx8QvKXY9fODd2sIZ-IiyXDNwkMqRks5t_TODgaJpZM4Uv3cX
.

[stephaniehicks]

Ah, ok. Thank you for helping me figure this out. Really appreciate it!

[vjtrost88]

I'm having a similar problem to @stephaniehicks - I'm on Mac OSX and I have a libpython2.7.a file instead of (in my case) a libpython2.7.dylib file. My py_discover_config() command says the following:

python:         /Users/vincenttrost/.pyenv/shims/python
libpython:      /Users/vincenttrost/.pyenv/versions/2.7.15/lib/libpython2.7.dylib[NOT FOUND]
pythonhome:     /Users/vincenttrost/.pyenv/versions/2.7.15:/Users/vincenttrost/.pyenv/versions/2.7.15
virtualenv:     /Users/vincenttrost/.pyenv/shims/activate_this.py
version:        2.7.15 (default, Sep 18 2018, 13:00:05)  [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
numpy:          /Users/vincenttrost/.pyenv/versions/2.7.15/lib/python2.7/site-packages/numpy
numpy_version:  1.15.1

NOTE: Python version was forced by RETICULATE_PYTHON

I am using pyenv which could be complicating things. Is there anywhere to find this libpython2.7.dylib file or is there another solution? Thanks in advance.

[jjallaire]

You need to use a version of Python built with the dynamically loadable
library (.dylib). R can't link to a static Python library.

On Thu, Sep 27, 2018 at 10:26 AM Vince Trost [e-mail]
wrote:

I'm having a similar problem to @stephaniehicks
https://github.com/stephaniehicks - I'm on Mac OSX and I have a
libpython2.7.a file instead of (in my case) a libpython2.7.dylib file. My
py_discover_config() command says the following:

python: /Users/vincenttrost/.pyenv/shims/python
libpython: /Users/vincenttrost/.pyenv/versions/2.7.15/lib/libpython2.7.dylib[NOT FOUND]
pythonhome: /Users/vincenttrost/.pyenv/versions/2.7.15:/Users/vincenttrost/.pyenv/versions/2.7.15
virtualenv: /Users/vincenttrost/.pyenv/shims/activate_this.py
version: 2.7.15 (default, Sep 18 2018, 13:00:05) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]
numpy: /Users/vincenttrost/.pyenv/versions/2.7.15/lib/python2.7/site-packages/numpy
numpy_version: 1.15.1

NOTE: Python version was forced by RETICULATE_PYTHON

I am using pyenv
https://amaral.northwestern.edu/resources/guides/pyenv-tutorial#Installation
which could be complicating things. Is there anywhere to find this
libpython2.7.dylib file or is there another solution? Thanks in advance.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rstudio/reticulate/issues/291#issuecomment-425112564,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGXx8txQUGKmIYqtR1zosH-JX4boWxwks5ufN_ygaJpZM4Uv3cX
.

[machow]

Just a note, to resolve the issue of using pyenv with reticulate, you can run

pyenv uninstall 3.6.7
env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.6.7

Notes

• replace 3.6.7 with the version you want to install
• you may need to delete the target version if it already exists

see https://github.com/pyenv/pyenv/issues/99