reticulate marks numpy as missing in Anaconda installation

Created on 18 Oct 2018 · 7Comments · Source: rstudio/reticulate

library(reticulate)
reticulate::use_python("~/Anaconda3/python.exe", required = TRUE)
reticulate::py_config()
numpy <- import("numpy", convert = TRUE)
numpy[["__path__"]]

I see:

> library(reticulate)
> reticulate::use_python("~/Anaconda3/python.exe", required = TRUE)
> reticulate::py_config()
python:         C:/Users/kevin/Anaconda3/python.exe
libpython:      C:/Users/kevin/Anaconda3/python37.dll
pythonhome:     C:\Users\kevin\ANACON~1
version:        3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
Architecture:   64bit
numpy:           [NOT FOUND]

NOTE: Python version was forced by use_python function
> numpy <- import("numpy", convert = TRUE)
> numpy[["__path__"]]
[1] "C:\\Users\\kevin\\ANACON~1\\lib\\site-packages\\numpy"

This was with Anaconda 3 just installed today. If I try to import numpy from the console directly, I see:

>>> import numpy
Traceback (most recent call last):
  File "C:\Users\kevin\Anaconda3\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
    from . import multiarray
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\kevin\Anaconda3\lib\site-packages\numpy\__init__.py", line 142, in <module>
    from . import add_newdocs
  File "C:\Users\kevin\Anaconda3\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "C:\Users\kevin\Anaconda3\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
    from .type_check import *
  File "C:\Users\kevin\Anaconda3\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "C:\Users\kevin\Anaconda3\lib\site-packages\numpy\core\__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: DLL load failed: The specified module could not be found.

But attempting to import numpy works fine if run from the Anaconda Prompt. I'm guessing it's because it puts a whole bunch of stuff on the PATH:

(base) C:\Users\kevin>echo %PATH%
C:\Users\kevin\Anaconda3;C:\Users\kevin\Anaconda3\Library\mingw-w64\bin;C:\Users\kevin\Anaconda3\Library\usr\bin;C:\Users\kevin\Anaconda3\Library\bin;C:\Users\kevin\Anaconda3\Scripts;C:\Users\kevin\Anaconda3\bin;<...>

Do we need to do that as well before running the config script? (It's worth noting that all of this stuff becomes available on the PATH _after_ Python has been initialized)

Source

kevinushey

👍4

Most helpful comment

I encountered the same issue.:

> library(reticulate) 
> py_config()
python:         C:\ANACON\~3\python.exe
libpython:      C:/ANACON\~3/python37.dll
pythonhome:     C:\ANACON\~3
version:        3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
Architecture:   64bit
numpy:           [NOT FOUND]

python versions found: 
 C:\ANACON\~3\python.exe
 C:\Anaconda3\python.exe
 C:\Anaconda3\envs\reticulate\python.exe`

Curiously, once Python has been initialised, it is able to import numpy correctly and would subsequently discover numpy if Python was reloaded:

> numpy <- import("numpy")
> py_discover_config()
python:         C:\ANACON\~3\python.exe
libpython:      C:/ANACON\~3/python37.dll
pythonhome:     C:\ANACON\~3
version:        3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:\ANACON\~3\lib\site-packages\numpy
numpy_version:  1.15.1

python versions found: 
 C:\ANACON\~3\python.exe
 C:\Anaconda3\python.exe
 C:\Anaconda3\envs\reticulate\python.exe

This is probably because the py_initialize code sets any paths correctly. Unfortunately, this doesn't help as reticulate doesn't attempt to reload numpy internally:

> py_config()
python:         C:\ANACON\~3\python.exe
libpython:      C:/ANACON\~3/python37.dll
pythonhome:     C:\ANACON\~3
version:        3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
Architecture:   64bit
numpy:           [NOT FOUND]

python versions found: 
 C:\ANACON\~3\python.exe
 C:\Anaconda3\python.exe
 C:\Anaconda3\envs\reticulate\python.exe

Using a conda environment (reticulate::use_condaenv()) does not help.

As noted by kevinushey, this is an issue with the PATH in Windows. During installation, Anaconda on Windows suggests _not_ adding Python to the path. Without Python in the path, reticulate is unable to find numpy. This is because it relies on system2 to execute config.py, and run the import numpy statement; this fails as it cannot find the correct DLL for numpy, causing numpy loading to fail.

One solution to this is to add the Anaconda libraries\bin directory to the path prior to initialising Python:

if(.Platform$OS.type == "windows") Sys.setenv(PATH= paste("C:/Anaconda3/Library/bin",Sys.getenv()["PATH"],sep=";"))
library(reticulate)

Alternatively users could load R/Rstudio from a conda command prompt that has already set the PATH correctly.

A more permanent fix might be to import numpy after initialising Python, rather than simply storing the result from running the config.py script.

twlee79 on 25 Oct 2018

👍5

All 7 comments

I encountered the same issue.:

> library(reticulate) 
> py_config()
python:         C:\ANACON\~3\python.exe
libpython:      C:/ANACON\~3/python37.dll
pythonhome:     C:\ANACON\~3
version:        3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
Architecture:   64bit
numpy:           [NOT FOUND]

python versions found: 
 C:\ANACON\~3\python.exe
 C:\Anaconda3\python.exe
 C:\Anaconda3\envs\reticulate\python.exe`

Curiously, once Python has been initialised, it is able to import numpy correctly and would subsequently discover numpy if Python was reloaded:

> numpy <- import("numpy")
> py_discover_config()
python:         C:\ANACON\~3\python.exe
libpython:      C:/ANACON\~3/python37.dll
pythonhome:     C:\ANACON\~3
version:        3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:\ANACON\~3\lib\site-packages\numpy
numpy_version:  1.15.1

python versions found: 
 C:\ANACON\~3\python.exe
 C:\Anaconda3\python.exe
 C:\Anaconda3\envs\reticulate\python.exe

This is probably because the py_initialize code sets any paths correctly. Unfortunately, this doesn't help as reticulate doesn't attempt to reload numpy internally:

> py_config()
python:         C:\ANACON\~3\python.exe
libpython:      C:/ANACON\~3/python37.dll
pythonhome:     C:\ANACON\~3
version:        3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)]
Architecture:   64bit
numpy:           [NOT FOUND]

python versions found: 
 C:\ANACON\~3\python.exe
 C:\Anaconda3\python.exe
 C:\Anaconda3\envs\reticulate\python.exe

Using a conda environment (reticulate::use_condaenv()) does not help.

One solution to this is to add the Anaconda libraries\bin directory to the path prior to initialising Python:

if(.Platform$OS.type == "windows") Sys.setenv(PATH= paste("C:/Anaconda3/Library/bin",Sys.getenv()["PATH"],sep=";"))
library(reticulate)

Alternatively users could load R/Rstudio from a conda command prompt that has already set the PATH correctly.

A more permanent fix might be to import numpy after initialising Python, rather than simply storing the result from running the config.py script.

twlee79 on 25 Oct 2018

👍5

I encountered a similar error trying to run an R-script that uses 'reticulate' to import the 'umap' package. Others have also encountered the error. https://github.com/satijalab/seurat/issues/958
Nonetheless, the suggestion described by @twlee79 works (THANK YOU!) and can be considered as a temporary workaround. In the long run this is a high priority issue for 'reticulate' that needs to be fixed.

Here is my setup using a python 3.7 conda environment

Sys.setenv(PATH= paste("D:/DEVTOOLS/Anaconda2/envs/py37/Library/bin",Sys.getenv()["PATH"],sep=";"))
Sys.setenv(RETICULATE_PYTHON = "D:/DEVTOOLS/Anaconda2/envs/py37/python.exe")
library(reticulate)
use_condaenv("py37")
py_config()
import("numpy")
import("umap")
np<-import("numpy")
print(np$version$full_version)

the script output is

python:         D:/DEVTOOLS/Anaconda2/envs/py37/python.exe
libpython:      D:/DEVTOOLS/Anaconda2/envs/py37/python37.dll
pythonhome:     D:\DEVTOOLS\ANACON~1\envs\py37
version:        3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]
Architecture:   64bit
numpy:          D:\DEVTOOLS\ANACON~1\envs\py37\lib\site-packages\numpy
numpy_version:  1.15.1

NOTE: Python version was forced by RETICULATE_PYTHON
Module(numpy)
Module(umap)
[1] "1.15.1"

imoutsatsos on 7 Feb 2019

👍3 ❤1

thank you @twlee79 !!!
really appreciate the work around

sidlawrence on 5 Mar 2019

@kevinushey We should see if there is something we can do during configuration (i.e. fixup the PATH proactively when we know this is going to happen) so that no workaround is required.

jjallaire on 5 Mar 2019

I'll take a look.

kevinushey on 5 Mar 2019

I have a similar issue with

> py_config()
python:         /home/iago/anaconda3/envs/r-reticulate/bin/python
libpython:      /home/iago/anaconda3/envs/r-reticulate/lib/libpython3.8.so
pythonhome:     /home/iago/anaconda3/envs/r-reticulate:/home/iago/anaconda3/envs/r-reticulate
version:        3.8.3 | packaged by conda-forge | (default, Jun  1 2020, 17:43:00)  [GCC 7.5.0]
numpy:           [NOT FOUND]

NOTE: Python version was forced by use_python function

in Debian 9. It is very difficult to get reticulate working. Just by luck (not now for the moment).

iago-pssjd on 9 Jun 2020

if(.Platform$OS.type == "windows") { Sys.setenv(PATH= paste("D:/Anaconda/envs/scvi-env/Library/bin/","D:/Anaconda/envs/scvi-env/Lib/site-packages/",Sys.getenv()["PATH"],sep=";")) Sys.setenv(RETICULATE_PYTHON = "D:/Anaconda/envs/scvi-env/python.exe") } library(reticulate) use_condaenv("scvi-env", required=TRUE)

I tried all the solutions above, all not work for me.

py_config()
python: D:/Anaconda/envs/scvi-env/python.exe
libpython: D:/Anaconda/envs/scvi-env/python37.dll
pythonhome: D:/Anaconda/envs/scvi-env
version: 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)]
Architecture: 64bit
numpy: [NOT FOUND]
leidenalg: [NOT FOUND]

NOTE: Python version was forced by use_python function

py_discover_config()
python: D:/Anaconda/envs/scvi-env/python.exe
libpython: D:/Anaconda/envs/scvi-env/python37.dll
pythonhome: D:/Anaconda/envs/scvi-env
version: 3.7.9 (default, Aug 31 2020, 17:10:11) [MSC v.1916 64 bit (AMD64)]
Architecture: 64bit
numpy: D:/Anaconda/envs/scvi-env/Lib/site-packages/numpy
numpy_version: 1.19.2

NOTE: Python version was forced by RETICULATE_PYTHON`