Reticulate: conda installed modules fail to properly load on linux

[blairj09]

Modules installed via conda don't properly load into R using reticulate::import() while modules installed via pip do. I believe this is related to #292. The following two .yml files define two separate conda environments, both with python 3.6. The only difference is that in one environment, pandas is installed via pip and in the other it is installed via conda.

name: pandas-pip
dependencies:
  - python=3.6
  - pip:
    - pandas

name: pandas-conda
dependencies:
  - python=3.6
  - pandas

After creating conda environments using these two files (conda env create -f pandas-conda.yml; conda env create -f pandas-pip.yml) the following shows the behavior of these two environments in R when trying to import pandas using reticulate::import():

library(reticulate)
use_condaenv("pandas-pip", required = TRUE)
pd <- import("pandas")

library(reticulate)
use_condaenv("pandas-conda", required = TRUE)
pd <- import("pandas")
#> Error in py_module_import(module, convert = convert): ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/_libs/window.cpython-36m-x86_64-linux-gnu.so)
#> 
#> Detailed traceback: 
#>   File "/home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/__init__.py", line 42, in <module>
#>     from pandas.core.api import *
#>   File "/home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/core/api.py", line 10, in <module>
#>     from pandas.core.groupby.groupby import Grouper
#>   File "/home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/core/groupby/__init__.py", line 2, in <module>
#>     from pandas.core.groupby.groupby import (
#>   File "/home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/core/groupby/groupby.py", line 49, in <module>
#>     from pandas.core.frame import DataFrame
#>   File "/home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/core/frame.py", line 74, in <module>
#>     from pandas.core.series import Series
#>   File "/home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/core/series.py", line 3980, in <module>
#>     Series._add_series_or_dataframe_operations()
#>   File "/home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/core/generic.py", line 8891, in _add_series_or_dataframe_operations
#>     from pandas.core import window as rwindow
#>   File "/home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/core/window.py", line 36, in <module>
#>     import pandas._libs.window as _window

When using reticulate::repl_python() with the conda installed version of pandas, the following results from trying to import pandas:

> repl_python()
#> Python 3.6.6 (/home/james/miniconda3/envs/pandas-conda/bin/python)
#> Reticulate 1.9 REPL -- A Python interpreter in R.
>>> import pandas as pd
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/james/miniconda3/envs/pandas-conda/lib/python3.6/site-packages/pandas/_libs/window.cpython-36m-x86_64-linux-gnu.so)
>>>

While the pip installed version returns the following:

> repl_python()
#> Python 3.6.6 (/home/james/miniconda3/envs/pandas-pip/bin/python)
#> Reticulate 1.9 REPL -- A Python interpreter in R.
>>> import pandas as pd
>>>

Starting the python interpreter from the command line for each of these environments results in the following:

$ conda activate pandas-pip
(pandas-pip) $ which python
/home/james/miniconda3/envs/pandas-pip/bin/python
(pandas-pip) $ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>>

$ conda activate pandas-conda
(pandas-conda) $ which python
/home/james/miniconda3/envs/pandas-conda/bin/python
(pandas-conda) $ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>>

Notice that in the native interpreter, both environments are able to import pandas as expected. This behavior has been observed on a server running Ubuntu 14.04.5 LTS with RStudio Server Pro Version 1.2.750-1. The following provides session details about the R installation:

devtools::session_info()
#> Session info -------------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.5.1 (2018-07-02)
#>  system   x86_64, linux-gnu           
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_US.UTF-8                 
#>  tz       Etc/UTC                     
#>  date     2018-07-11
#> Packages -----------------------------------------------------------------
#>  package    * version date       source        
#>  backports    1.1.2   2017-12-13 RSPM (R 3.5.0)
#>  base       * 3.5.1   2018-07-10 local         
#>  compiler     3.5.1   2018-07-10 local         
#>  datasets   * 3.5.1   2018-07-10 local         
#>  devtools     1.13.6  2018-06-27 RSPM (R 3.5.1)
#>  digest       0.6.15  2018-01-28 RSPM (R 3.5.0)
#>  evaluate     0.10.1  2017-06-24 RSPM (R 3.5.0)
#>  graphics   * 3.5.1   2018-07-10 local         
#>  grDevices  * 3.5.1   2018-07-10 local         
#>  grid         3.5.1   2018-07-10 local         
#>  htmltools    0.3.6   2017-04-28 RSPM (R 3.5.0)
#>  jsonlite     1.5     2017-06-01 RSPM (R 3.5.0)
#>  knitr        1.20    2018-02-20 RSPM (R 3.5.0)
#>  lattice      0.20-35 2017-03-25 CRAN (R 3.5.1)
#>  magrittr     1.5     2014-11-22 RSPM (R 3.5.0)
#>  Matrix       1.2-14  2018-04-13 CRAN (R 3.5.1)
#>  memoise      1.1.0   2017-04-21 RSPM (R 3.5.1)
#>  methods    * 3.5.1   2018-07-10 local         
#>  Rcpp         0.12.17 2018-05-18 RSPM (R 3.5.0)
#>  reticulate * 1.9     2018-07-06 RSPM (R 3.5.1)
#>  rmarkdown    1.10    2018-06-11 RSPM (R 3.5.0)
#>  rprojroot    1.3-2   2018-01-03 RSPM (R 3.5.0)
#>  stats      * 3.5.1   2018-07-10 local         
#>  stringi      1.2.3   2018-06-12 RSPM (R 3.5.0)
#>  stringr      1.3.1   2018-05-10 RSPM (R 3.5.0)
#>  tools        3.5.1   2018-07-10 local         
#>  utils      * 3.5.1   2018-07-10 local         
#>  withr        2.1.2   2018-03-15 RSPM (R 3.5.0)
#>  yaml         2.1.19  2018-05-01 RSPM (R 3.5.0)

Interestingly enough, this is not an issue on my MacBook. Both conda and pip installed modules behave as expected.

[ijlyttle]

I have spent far too much time today on a similar problem.

For me, this happened on Ubuntu 14.04, using conda with pandas-0.23.2.

Someone suggested trying this at the command line, to see which versions of GLIBCXX are available:

strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX

On my system (well, the Travis system), the latest version of GLIBCXX was 3.4.19, which is insufficient. Updating libgcc (sudo apt-get install libgcc) was ineffective because, as I already had the latest version of libgcc for 14.04. Trying to use an experimental repository (sudo apt-add-repository -y ppa:ubuntu-toolchain-r/test) just gave me a different flavor of the same problem.

Finally, I came across this suggestion to downgrade pandas, which worked for me:

# your environment name may be different
conda install -y pandas=0.22.0 --name r-reticulate

Perhaps someone who knows more than I do can pipe in here; all I can offer is a somewhat-unsatisfactory workaround.

(For your entertainment: https://travis-ci.org/ijlyttle/altair/builds - see 2018-07-14)

[eddelbuettel]

Well Ubuntu 14.04 is from 2014 so ... I think that if you can you should really try 16.04 which has been out for two full years (and is of course now being replaced by 18.04). [ I happen to update every six months, typically two to three month after the releases came out. No issue. R-sig-debian can help too. ]

[ijlyttle]

I get your point, but in my particular case I seem to be limited by what Travis offers which is (unfortunately for me) 14.04.

[eddelbuettel]

Sure, but we are all bound by the same Travis constraint. What I sometimes do is a combination of external PPAs (ie for newer g++-5,6,7,... when I need them) or my own PPA where I rebuild newer packages for that version. Some more enterprising folks even bring their own containers to Docker...

But yes, complicated builds remain complicated builds. :-/ And there is #stillNoFreeLunch

[ijlyttle]

Thanks - this seems like an interesting direction. I had fiddled a bit with the PPAs without success; this could be useful for me to look at again.

[eddelbuettel]

Was just out running and thinking about this a little; maybe we can take this off-line for a parallel thread but this may be worth investigating. Look at this .travis.yml from one of my upstream libraries (in Debian and R); we may be able to cook up something similar on top of Rocker to supply R (possibly different versions) plus whatever "batteries included" we need. Interested?

[ijlyttle]

I'm not sure I wrapped my head completely around this - I have not yet used Docker - but if I understand, the idea would to use Travis (at 14.04) run a Docker (Rocker) image (at 16.04, or whatever) that would have, say, Conda on it. As an example, such a Rocker image could be "specific" to reticulate and Conda, and be made available for packages that use reticulate.

In practice, for a CI build, Travis would install this Rocker image, then "tell" the Rocker to install the particular R package and any dependencies it does not already have, then run the R commands - including R CMD check, while reporting back to the Travis host.

Keeping in mind my lack of experience with Docker, I can try to pitch in - but I don't want to slow things down because of my learning curve. Please tag me in on a parallel issue, and I'll do what I can.

[eddelbuettel]

Yes. use Travis to launch Docker at which point one has the freedom to run a combination of tools within Docker as one pleases: old R vs new R, old g++ vs new g++, clang, ... Ditto for reticulate and various Python needs. It seems doable with some work, I do not know if I have the time to start it. It is more your itch to scratch, but you never used Docker it may not be the best way to go. But thought I should mention the possibility.