Theano: Package import conflict with Theano

Created on 20 Jul 2017  路  19Comments  路  Source: Theano/Theano

I've observed a conflict between Theano (via PyMC3) and OpenTurns on Windows 7 and 10, 64-bit. If Theano is imported first, OpenTurns fails to import with the following error

Traceback (most recent call last):

  File "<ipython-input-3-704e69d30dff>", line 1, in <module>
    import openturns as ot

  File "C:\Anaconda3\lib\site-packages\openturns\__init__.py", line 61, in <module>
    from .common import *

  File "C:\Anaconda3\lib\site-packages\openturns\common.py", line 20, in <module>
    _common = swig_import_helper()

  File "C:\Anaconda3\lib\site-packages\openturns\common.py", line 19, in swig_import_helper
    return importlib.import_module('_common')

  File "C:\Anaconda3\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)

ModuleNotFoundError: No module named '_common'

If I reverse the import order (i.e., OpenTurns first, then Theano/PyMC3), then Theano fails to import with

Traceback (most recent call last):

  File "<ipython-input-2-e9f7838173a6>", line 5, in <module>
    import pymc3 as pm

  File "C:\Anaconda3\lib\site-packages\pymc3\__init__.py", line 5, in <module>
    from .distributions import *

  File "C:\Anaconda3\lib\site-packages\pymc3\distributions\__init__.py", line 1, in <module>
    from . import timeseries

  File "C:\Anaconda3\lib\site-packages\pymc3\distributions\timeseries.py", line 1, in <module>
    import theano.tensor as tt

  File "C:\Anaconda3\lib\site-packages\theano\__init__.py", line 80, in <module>
    from theano.scan_module import (scan, map, reduce, foldl, foldr, clone,

  File "C:\Anaconda3\lib\site-packages\theano\scan_module\__init__.py", line 41, in <module>
    from theano.scan_module import scan_opt

  File "C:\Anaconda3\lib\site-packages\theano\scan_module\scan_opt.py", line 60, in <module>
    from theano import tensor, scalar

  File "C:\Anaconda3\lib\site-packages\theano\tensor\__init__.py", line 9, in <module>
    from theano.tensor.subtensor import *

  File "C:\Anaconda3\lib\site-packages\theano\tensor\subtensor.py", line 26, in <module>
    import theano.gof.cutils  # needed to import cutils_ext

  File "C:\Anaconda3\lib\site-packages\theano\gof\cutils.py", line 320, in <module>
    compile_cutils()

  File "C:\Anaconda3\lib\site-packages\theano\gof\cutils.py", line 285, in compile_cutils
    preargs=args)

  File "C:\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 2325, in compile_str
    return dlimport(lib_filename)

  File "C:\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 302, in dlimport
    rval = __import__(module_name, {}, {}, [module_name])
ImportError: DLL load failed: The specified procedure could not be found.

Both OpenTurns and Theano/PyMC3 work fine independently, but having these packages work together would be extremely useful. I am using Theano 0.9.0 and OpenTurns 1.9.1. Any thoughts? Thank you!

picture Aorus42

Most helpful comment

Ok. Package mingw is not recommended anymore, m2w64-toolchain is recommended instead: http://deeplearning.net/software/theano_versions/dev/install_windows.html . Could you install it ?

conda remove mingw
conda install m2w64-toolchain

Also, I think it's better to use only one package system as much as possible (ie. either conda or pip, but not both, except when we can't do otherwise, or when a package system has a too old version for a desired package). So, I suggest to replace pymc3 pip package with conda package.

pip uninstall pymc3
conda install pymc3

NB: If numpy and scipy were installed with pip, consider replacing them too.

With these changes, can you retry to import both theano and openturns ? If it fails again, maybe should you consider creating a new conda environment and reinstalling everything as I did in my comments above.

picture notoraptor on 22 Aug 2017
👍3

All 19 comments

Can you check what the content of sys.path is before importing the first package, before importing the second, and then when it crashes? Also, check what is in the environment, in particular if %PATH% has been changed.
If you can do that for both import orders, that would be useful.

lamblin picture lamblin on 21 Jul 2017

sys.path yields the same regardless of which of the two packages are imported:

['',
 'C:\\Anaconda3\\lib\\site-packages\\spyder\\utils\\site',
 'C:\\Anaconda3\\python36.zip',
 'C:\\Anaconda3\\DLLs',
 'C:\\Anaconda3\\lib',
 'C:\\Anaconda3',
 'C:\\Anaconda3\\lib\\site-packages',
 'C:\\Anaconda3\\lib\\site-packages\\Sphinx-1.5.6-py3.6.egg',
 'C:\\Anaconda3\\lib\\site-packages\\win32',
 'C:\\Anaconda3\\lib\\site-packages\\win32\\lib',
 'C:\\Anaconda3\\lib\\site-packages\\Pythonwin',
 'C:\\Anaconda3\\lib\\site-packages\\setuptools-27.2.0-py3.6.egg',
 'C:\\Anaconda3\\lib\\site-packages\\IPython\\extensions',
 'C:\\Users\\fernandp\\.ipython']

My PATH variable contains

C:\ProgramData\Oracle\Java\javapath;
%SystemRoot%\system32;
%SystemRoot%;
%SystemRoot%\System32\Wbem;
%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\;
C:\Program Files (x86)\Microsoft Application Virtualization Client;
C:\PROGRA~1\CONDUS~1\DISKEE~1\;
C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;
C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64;
C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;
C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64;
C:\Program Files (x86)\Graphviz2.38\bin;C:\TDM-GCC-64\bin;
C:\openblas\lib;
C:\openblas\bin;
%systemroot%\System32\WindowsPowerShell\v1.0\;
C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;
C:\Program Files\MATLAB\bin;C:\Anaconda3;C:\Anaconda3\Scripts;
C:\Anaconda3\Library\bin;C:\Anaconda3\Lib\site-packages
Aorus42 picture Aorus42 on 18 Aug 2017

How did you print the PATH env variable? Mostly, inside Python before any import do:
print(sys.path)
print(os.environ["PATH"])

then import one of the packages. Then print them again and compare. Probably one of them change something to that and this cause problems.

nouiz picture nouiz on 18 Aug 2017

Oh I see. Thanks for the clarification.

Before any imports:

print(sys.path)
['', 'C:\\Anaconda3\\lib\\site-packages\\spyder\\utils\\site', 'C:\\Anaconda3\\python36.zip', 'C:\\Anaconda3\\DLLs', 'C:\\Anaconda3\\lib', 'C:\\Anaconda3', 'C:\\Anaconda3\\lib\\site-packages', 'C:\\Anaconda3\\lib\\site-packages\\Sphinx-1.5.6-py3.6.egg', 'C:\\Anaconda3\\lib\\site-packages\\win32', 'C:\\Anaconda3\\lib\\site-packages\\win32\\lib', 'C:\\Anaconda3\\lib\\site-packages\\Pythonwin', 'C:\\Anaconda3\\lib\\site-packages\\setuptools-27.2.0-py3.6.egg', 'C:\\Anaconda3\\lib\\site-packages\\IPython\\extensions', 'C:\\Users\\fernandp\\.ipython']

print(os.environ["PATH"])
C:\Anaconda3\Library\bin;C:\Anaconda3\Library\bin;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Microsoft Application Virtualization Client;C:\PROGRA~1\CONDUS~1\DISKEE~1\;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64;C:\Program Files (x86)\Graphviz2.38\bin;C:\TDM-GCC-64\bin;C:\openblas\lib;C:\openblas\bin;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\MATLAB\bin;C:\Anaconda3;C:\Anaconda3\Scripts;C:\Anaconda3\Library\bin;C:\Anaconda3\Lib\site-packages;



md5-11b9e7fbe19081de8804704ee9f53391



print(sys.path)
['', 'C:\\Anaconda3\\lib\\site-packages\\spyder\\utils\\site', 'C:\\Anaconda3\\python36.zip', 'C:\\Anaconda3\\DLLs', 'C:\\Anaconda3\\lib', 'C:\\Anaconda3', 'C:\\Anaconda3\\lib\\site-packages', 'C:\\Anaconda3\\lib\\site-packages\\Sphinx-1.5.6-py3.6.egg', 'C:\\Anaconda3\\lib\\site-packages\\win32', 'C:\\Anaconda3\\lib\\site-packages\\win32\\lib', 'C:\\Anaconda3\\lib\\site-packages\\Pythonwin', 'C:\\Anaconda3\\lib\\site-packages\\setuptools-27.2.0-py3.6.egg', 'C:\\Anaconda3\\lib\\site-packages\\IPython\\extensions', 'C:\\Users\\fernandp\\.ipython']



md5-7c03d14f5a3c3c5f124951609a0a25e0



print(os.environ["PATH"])
C:\Anaconda3\Library\bin;C:\Anaconda3\Library\bin;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Microsoft Application Virtualization Client;C:\PROGRA~1\CONDUS~1\DISKEE~1\;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64;C:\Program Files (x86)\Graphviz2.38\bin;C:\TDM-GCC-64\bin;C:\openblas\lib;C:\openblas\bin;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\MATLAB\bin;C:\Anaconda3;C:\Anaconda3\Scripts;C:\Anaconda3\Library\bin;C:\Anaconda3\Lib\site-packages;



md5-73a429ecd2c12fa27fb21afd5207ec07



print(sys.path)
['', 'C:\\Anaconda3\\lib\\site-packages\\spyder\\utils\\site', 'C:\\Anaconda3\\python36.zip', 'C:\\Anaconda3\\DLLs', 'C:\\Anaconda3\\lib', 'C:\\Anaconda3', 'C:\\Anaconda3\\lib\\site-packages', 'C:\\Anaconda3\\lib\\site-packages\\Sphinx-1.5.6-py3.6.egg', 'C:\\Anaconda3\\lib\\site-packages\\win32', 'C:\\Anaconda3\\lib\\site-packages\\win32\\lib', 'C:\\Anaconda3\\lib\\site-packages\\Pythonwin', 'C:\\Anaconda3\\lib\\site-packages\\setuptools-27.2.0-py3.6.egg', 'C:\\Anaconda3\\lib\\site-packages\\IPython\\extensions', 'C:\\Users\\fernandp\\.ipython']



md5-7c03d14f5a3c3c5f124951609a0a25e0



print(os.environ["PATH"])
C:\Anaconda3\Library\bin;C:\Anaconda3\Library\bin;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Microsoft Application Virtualization Client;C:\PROGRA~1\CONDUS~1\DISKEE~1\;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x86;C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64;C:\Program Files (x86)\Graphviz2.38\bin;C:\TDM-GCC-64\bin;C:\openblas\lib;C:\openblas\bin;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\MATLAB\bin;C:\Anaconda3;C:\Anaconda3\Scripts;C:\Anaconda3\Library\bin;C:\Anaconda3\Lib\site-packages;

It seems they are identical.

Aorus42 picture Aorus42 on 18 Aug 2017

Can you go in pdb when the error in Theano happen and print those value at that moment and compare?

nouiz picture nouiz on 21 Aug 2017

see https://stackoverflow.com/questions/44422100/python-package-import-conflicts-theano-and-openturns

jschueller picture jschueller on 22 Aug 2017

Note that using packages from conda-forge when it's not the default channel is unsupported.

That being said, a possible solution would be to build openturns from source with m2w64 but it seems it's not possible yet for 32 bits:
https://github.com/conda-forge/openturns-feedstock/pull/39

jschueller picture jschueller on 22 Aug 2017

Hello @jschueller , I am using Windows 10 64 bits with Anaconda3-4.4.0-Windows-x86_64, but I am currently unable to reproduce the problem. This is how I tested it:

(test2) C:\Users\notoraptor\mila\dev\git>conda create -n test3 python=3
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment C:\Users\notoraptor\data\programs\Anaconda3\envs\test3:

The following NEW packages will be INSTALLED:

    pip:            9.0.1-py36_1
    python:         3.6.2-0
    setuptools:     27.2.0-py36_1
    vs2015_runtime: 14.0.25420-0
    wheel:          0.29.0-py36_0

Proceed ([y]/n)?

vs2015_runtime 100% |###############################| Time: 0:00:00   8.18 MB/s
python-3.6.2-0 100% |###############################| Time: 0:00:03   9.19 MB/s
#
# To activate this environment, use:
# > activate test3
#
# To deactivate this environment, use:
# > deactivate test3
#
# * for power-users using bash, you must source
#

(test2) C:\Users\notoraptor\mila\dev\git>activate test3



md5-7c03d14f5a3c3c5f124951609a0a25e0



(test3) C:\Users\notoraptor\mila\dev\git>conda install numpy scipy mkl-service libpython m2w64-toolchain nose sphinx
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment C:\Users\notoraptor\data\programs\Anaconda3\envs\test3:

The following NEW packages will be INSTALLED:

    alabaster:                      0.7.10-py36_0
    babel:                          2.4.0-py36_0
    colorama:                       0.3.9-py36_0
    docutils:                       0.14-py36_0
    imagesize:                      0.7.1-py36_0
    jinja2:                         2.9.6-py36_0
    libpython:                      2.0-py36_0
    m2w64-binutils:                 2.25.1-5
    m2w64-bzip2:                    1.0.6-6
    m2w64-crt-git:                  5.0.0.4636.2595836-2
    m2w64-gcc:                      5.3.0-6
    m2w64-gcc-ada:                  5.3.0-6
    m2w64-gcc-fortran:              5.3.0-6
    m2w64-gcc-libgfortran:          5.3.0-6
    m2w64-gcc-libs:                 5.3.0-7
    m2w64-gcc-libs-core:            5.3.0-7
    m2w64-gcc-objc:                 5.3.0-6
    m2w64-gmp:                      6.1.0-2
    m2w64-headers-git:              5.0.0.4636.c0ad18a-2
    m2w64-isl:                      0.16.1-2
    m2w64-libiconv:                 1.14-6
    m2w64-libmangle-git:            5.0.0.4509.2e5a9a2-2
    m2w64-libwinpthread-git:        5.0.0.4634.697f757-2
    m2w64-make:                     4.1.2351.a80a8b8-2
    m2w64-mpc:                      1.0.3-3
    m2w64-mpfr:                     3.1.4-4
    m2w64-pkg-config:               0.29.1-2
    m2w64-toolchain:                5.3.0-7
    m2w64-tools-git:                5.0.0.4592.90b8472-2
    m2w64-windows-default-manifest: 6.4-3
    m2w64-winpthreads-git:          5.0.0.4634.697f757-2
    m2w64-zlib:                     1.2.8-10
    markupsafe:                     1.0-py36_0
    mkl:                            2017.0.3-0
    mkl-service:                    1.1.2-py36_3
    msys2-conda-epoch:              20160418-1
    nose:                           1.3.7-py36_1
    numpy:                          1.13.1-py36_0
    pygments:                       2.2.0-py36_0
    pytz:                           2017.2-py36_0
    requests:                       2.14.2-py36_0
    scipy:                          0.19.1-np113py36_0
    six:                            1.10.0-py36_0
    snowballstemmer:                1.2.1-py36_0
    sphinx:                         1.6.3-py36_0
    sphinxcontrib:                  1.0-py36_0
    sphinxcontrib-websupport:       1.0.1-py36_0

Proceed ([y]/n)?

docutils-0.14- 100% |###############################| Time: 0:00:00   5.84 MB/s
libpython-2.0- 100% |###############################| Time: 0:00:00   2.91 MB/s
markupsafe-1.0 100% |###############################| Time: 0:00:00   1.80 MB/s
numpy-1.13.1-p 100% |###############################| Time: 0:00:00   9.70 MB/s
sphinxcontrib- 100% |###############################| Time: 0:00:00 125.68 kB/s
scipy-0.19.1-n 100% |###############################| Time: 0:00:02   6.87 MB/s
sphinxcontrib- 100% |###############################| Time: 0:00:00   2.36 MB/s
sphinx-1.6.3-p 100% |###############################| Time: 0:00:00   7.11 MB/s



md5-7c03d14f5a3c3c5f124951609a0a25e0



(test3) C:\Users\notoraptor\mila\dev\git>conda install theano pygpu
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment C:\Users\notoraptor\data\programs\Anaconda3\envs\test3:

The following NEW packages will be INSTALLED:

    libgpuarray: 0.6.9-vc14_0 [vc14]
    mako:        1.0.6-py36_0
    pygpu:       0.6.9-py36_0
    theano:      0.9.0-py36_0

Proceed ([y]/n)?

libgpuarray-0. 100% |###############################| Time: 0:00:00   5.05 MB/s
mako-1.0.6-py3 100% |###############################| Time: 0:00:00   5.15 MB/s
pygpu-0.6.9-py 100% |###############################| Time: 0:00:00   7.88 MB/s
theano-0.9.0-p 100% |###############################| Time: 0:00:00   9.32 MB/s



md5-7c03d14f5a3c3c5f124951609a0a25e0



(test3) C:\Users\notoraptor\mila\dev\git>conda install -c conda-forge openturns
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment C:\Users\notoraptor\data\programs\Anaconda3\envs\test3:

The following NEW packages will be INSTALLED:

    openturns: 1.9-py36_205 conda-forge

Proceed ([y]/n)?

openturns-1.9- 100% |###############################| Time: 0:00:13   4.26 MB/s



md5-63a554df5e4630468c9f662752f75c38



(test3) C:\Users\notoraptor\mila\dev\git>python -c "import theano; import openturns"



md5-7c03d14f5a3c3c5f124951609a0a25e0



(test3) C:\Users\notoraptor\mila\dev\git>python -c "import openturns; import theano"

Do you find any difference with how you installed theano and openturns on your own computer ?

notoraptor picture notoraptor on 22 Aug 2017

hi,

I didn't try it myself.

Why did you install dependencies first ?
I think these would be enough:

conda install theano
conda install -c conda-forge openturns
jschueller picture jschueller on 22 Aug 2017

Indeed, thse two commands are enough as we are constantly using only conda.

Still no errors on my computer. Could you try it in a new conda env ?

notoraptor picture notoraptor on 22 Aug 2017

I just tested on my machine: no error either.

jschueller picture jschueller on 22 Aug 2017

maybe @Aorus42 could provide more details on how he installed the packages

jschueller picture jschueller on 22 Aug 2017

Ah, sorry, I haven't quoted the right username !

Indeed, @Aorus42 , can you give more details to help reproduce the problem ?

notoraptor picture notoraptor on 22 Aug 2017

My install history is:

  1. Anaconda 4.3.1
  2. pip install pymc3, which installs theano as a dependency
  3. conda install mingw libpython
  4. Manually place openblas files into C:/openblas and add to PATH as described by Subhajit Mandal and IshanAntony in https://www.kaggle.com/c/otto-group-product-classification-challenge/discussion/13973
  5. conda install -c conda-forge openturns

From debugging, the values at 

File "C:\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 302, in dlimport
    rval = __import__(module_name, {}, {}, [module_name])

are

fullpath = 'C:\\Users\\myname\\AppData\\Local\\Theano\\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_58_Stepping_9_GenuineIntel-3.6.1-64\\cutils_ext\\cutils_ext.pyd'
module_name = 'cutils_ext.cutils_ext'
rval = None
suffix = '.pyd'
t0 = 1503415497.5404065
workdir = 'C:\\Users\\myname\\AppData\\Local\\Theano\\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_58_Stepping_9_GenuineIntel-3.6.1-64'

I haven't yet tried @jschueller's suggestion posted on StackOverflow, because I've been needing uninterrupted pymc3 functionality (with compiling theano models to C) and am hesitant to risk breaking my installation. Hopefully I will have a chance to try soon, however.

Aorus42 picture Aorus42 on 22 Aug 2017

what is the list of dlls you manually installed ?

jschueller picture jschueller on 22 Aug 2017

libgcc_s_seh-1.dll
libgfortran-3.dll
libopenblas.dll
libquadmath-0.dll

Aorus42 picture Aorus42 on 22 Aug 2017

Ok. Package mingw is not recommended anymore, m2w64-toolchain is recommended instead: http://deeplearning.net/software/theano_versions/dev/install_windows.html . Could you install it ?

conda remove mingw
conda install m2w64-toolchain

Also, I think it's better to use only one package system as much as possible (ie. either conda or pip, but not both, except when we can't do otherwise, or when a package system has a too old version for a desired package). So, I suggest to replace pymc3 pip package with conda package.

pip uninstall pymc3
conda install pymc3

NB: If numpy and scipy were installed with pip, consider replacing them too.

With these changes, can you retry to import both theano and openturns ? If it fails again, maybe should you consider creating a new conda environment and reinstalling everything as I did in my comments above.

notoraptor picture notoraptor on 22 Aug 2017
👍3

@Aorus42 these could totally conflict with the ones we provide and be the source of the problem

jschueller picture jschueller on 22 Aug 2017

@notoraptor replacing mingw with m2w64-toolchain did the trick! This is excellent, thanks everyone for your help!

Aorus42 picture Aorus42 on 22 Aug 2017
👍2
Was this page helpful?
0 / 5 - 0 ratings

Related issues

aviveise picture aviveise  路  6Comments
subzerofun picture subzerofun  路  8Comments
grayfall picture grayfall  路  3Comments
ibab picture ibab  路  4Comments
QuantumLiu picture QuantumLiu  路  4Comments