Spyder: Compiled modules of non-default interpreters fail to import on Windows

[DuncanMcRae]

Problem Description

Spyder v3.3.3 not loading numpy from different conda environment.

All modules installed using default channels (.condarc). I have read a recent support ticket and have followed the modular approach (detailed below). This doesn't work.

What steps reproduce the problem?

I have a fresh install of Miniconda3. I have created a env called spyder3 and installed spyder. I have then created another env called test and installed the spyder kernels and numpy.

1. Download and install miniconda3
2. Update conda: conda update conda -y
3. Created an env called spyder3: conda create --name spyder3 -y
4. Activate spyder3 env: conda activate spyder3
5. Install spyder: conda install spyder
6. Deactivate spyder3 env: conda deactivate
7. Create a new env called test2: conda create --name test2 -y
8. Activate test2: conda activate test2
9. Install spyder kernels: conda install spyder-kernels=0.* -y
10. Install numpy: conda install numpy
11. Deactivate test2 env: conda deactivate
12. Activate spyder3 env: conda activate spyder3
13. Run spyder: spyder
14. Change python interpreter: Tools>Preferences>Python interpreter>...pointed to ...Miniconda3\envs\test2\python.exe.
15. Open new console
16. Import numpy in console

What is the expected output? What do you see instead?

Expecting numpy to load without error like other modules (example: xlsxwriter).

Paste Traceback/Error Below (if applicable)spyder

Traceback (most recent call last):

File "", line 1, in
import numpy

File "C:\Users\dunca\Miniconda3\envs\test2\lib\site-packages\numpy__init__.py", line 140, in
from . import _distributor_init

File "C:\Users\dunca\Miniconda3\envs\test2\lib\site-packages\numpy_distributor_init.py", line 34, in
from . import _mklinit

ImportError: DLL load failed: The specified module could not be found.

```

Versions

• Spyder version: 3.3.3
• Python version: 3.7.3 64bit
• Qt version: 5.9.6
• PyQt version: 5.9.2
• Operating System name/version: Windows 10

Dependencies

IPython >=4.0 : 7.4.0 (OK)
cython >=0.21 : None (NOK)
jedi >=0.9.0 : 0.13.3 (OK)
matplotlib >=2.0.0: None (NOK)
nbconvert >=4.0 : 5.4.1 (OK)
numpy >=1.7 : None (NOK)
pandas >=0.13.1 : None (NOK)
pycodestyle >=2.3 : 2.5.0 (OK)
pyflakes >=0.6.0 : 2.1.1 (OK)
pygments >=2.0 : 2.3.1 (OK)
pylint >=0.25 : 2.3.1 (OK)
qtconsole >=4.2.0 : 4.4.3 (OK)
rope >=0.9.4 : 0.12.0 (OK)
sphinx >=0.6.6 : 1.8.5 (OK)
sympy >=0.7.3 : None (NOK)

[ccordoba12]

Thanks for reporting. This was also found by one of our developers (@CAM-Gerlach) a few weeks ago. We'll fix it in our next version (3.3.5)

[mingwandroid]

AFAICT this shouldn't be supported? Conda environments are very cheap so why not install numpy and spyder in the same environment? That is why we have environments at all is it not?

[CAM-Gerlach]

@mingwandroid This results in a the user having to run and maintain a separate Spyder instance (and all its dependencies) for every environment, rather than being able to select their desired interpreter like essentially every other IDE out there. This, naturally, has a substantial number of downsides for essentially no gain, defaulting most of the benefits of the big separation of Spyder and Spyder-Kernels and making creating new environments much less cheap:

• Requires the user to boot up an entirely separate IDE instance (which won't even work without disabling single instance mode, on by default) just to change their Python interpreter environment
• Users cannot mix kernels running in different environments or Python installs in the same Spyder
• Spyder and all its dependencies must be installable on the target environment/Python installation, rather than just spyder-kernels
• Spyder + dependencies add up to 1 GB or more for every environment, which is not a trivial amount on a fast SSD after accounting for the space needed by the OS, data, programs, files, and other packages
• Needless extra time, work and complexity for the user to install, run and maintain
• User must first activate the desired environment before launching Spyder, rather than launching the same Spyder every time (also increasing the possibility of mistakes)
• With the Spyder environment not being kept clean Increases probability of something going wrong with any one Spyder or its large dependency tail
• Could result in unsolvable dependency conflicts if user packages require incompatible versions of dependencies (e.g. Qt/PyQt/PySide)
• Doesn't allow us to potentially maintain support for running old Python 2 files, while only having to support Python 3 in Spyder itself in the future

[ccordoba12]

@mingwandroid, the workflow we want to support is to only install spyder-kernels in a conda environment, and then select that environment in our preferences so our consoles are easily associated with it.

According to @CAM-Gerlach, this was working correctly before the DLL changes the Anaconda team did some months ago. So we would like to have that functionality back.

[CAM-Gerlach]

According to @CAM-Gerlach, this was working correctly before the DLL changes the Anaconda team did some months ago.

I don't remember saying that, specifically, and I can't state it definitively now, but I can say I have never encountered this problem before using a version of Python that had the DLL search path changes applied. While I cannot be absolutely certain I would have triggered it, given it appears to occur with numpy, netcdf4, cartopy (and likely basemap), etc. and I am fairly confident that I imported and used those packages in non-default environments in the past when they were not installed in my Spyder environment, it seems like a possible candidate. If you provide me the specific Python versions directly before and after the changes, I can test with both my Spyder and interpreter environments on the two respective versions and see if I can reproduce the issue.

[mingwandroid]

@ccordoba12

According to @CAM-Gerlach, this was working correctly before the DLL changes the Anaconda team did some months ago. So we would like to have that functionality back.

@CAM-Gerlach

a version of Python that had the DLL search path changes applied

Can I ask that you be specific about which change you mean here? Please if you can, select it from the git history at https://github.com/AnacondaRecipes/python-feedstock/commits/master/recipe

To my mind, the DLL changes (by which I mean the dropping of the automatic addition of of Library/bin to PATH) were done more than a year ago now. Every release of Python 3.7.0 from Anaconda Inc, has included them. I believe that what changed was that conda 4.6.0 to 4.6.13 inclusive had a bug to do with activation mishandling PATH on Windows. Fixing this bug was a bit of a nightmare due to cmd.exe - on PowerShell, kudos to Microsoft, it was a pleasant 5 minute job - probably I did it badly then though!

In that period, we also implemented CONDA_DLL_SEARCH_MODIFICATION_ENABLE (it is disabled by default, in all but a single, old point release of python) and I think that has muddied the waters about what's changed and why.

Forgive me if I misunderstand here, but I think Spyder wants to be stacking conda environments to achieve this single Spyder env, multiple kernels in multiple other envs? That's what we do in conda-build when activating host then build envs. The build env is stacked on-top of the host env such that things in build get found first.

Does Spyder customise itself for conda much? Would something like this be acceptable? You end up with a lot of entries added to PATH mind you (12!) so the user needs a good amount of space free on PATH for that (and depending on how you manage PATH the limit can be ~8190 or it can be ~32k - I'm not sure what it is for conda, I suspect ~8190 when using cmd.exe and ~32k when using PowerShell).

[CAM-Gerlach]

Thanks for your response, @mingwandroid . To be clear, I didn't mean to call you over here to accuse you or Anaconda of causing this issue without robust evidence, and I don't recall previously making that claim @ccordoba12 mentioned above (though its not impossible I could have forgotten). I only did so due to knowing your deep expertise in this area and the fact you'd been summoned to take a look at DLL loading issues previously. I just wanted to preemptively apologize if you garnered that impression from any of the posts here.

---

I did extensive testing on Windows 8.1 x64 Miniconda x64, with Spyder running under both Python 3.7.2-h8c8aaf0_2 with the DLL search order enabled by default, and the latest 3.7.3-h8c8aaf0_1 with it disabled, and replicated in two different interpreter environments (in addition to Spyder's own), a wide variety of Python builds/versions and in some cases multiple repeat trials of each. In summary, the results support the DLL modification I was referring to being the main culprit when enabled (i.e. on 3.7.2-h8c8aaf0_2 and 3.6.8-h9f7ef89_1 by default, both of which I had tested on previously), and secondarily Python 3.7 not adding the interpreter env to the PATH while Python 3.6 did, which spyder-ide/spyder-kernels#109 fixes so that this error does not occur except on those two Python versions, or with CONDA_DLL_SEARCH_MODIFICATION_ENABLE=1 system-wide.

Full details:

Without spyder-ide/spyder-kernels#109 applied (i.e. running Spyder-Kernels 1.3.2), I tested the following Python versions for this problem [with various affected packages; all versions given are kernel interpreter versions, not the version Spyder is running on]:

3.7.3-h8c8aaf0_1  ERROR
3.7.3-h8c8aaf0_0  ERROR
3.7.2-h8c8aaf0_10 ERROR
3.7.2-h8c8aaf0_2  ERROR *DLL Change on by default*
3.7.2-h8c8aaf0_0  ERROR
3.7.1-h8c8aaf0_6  ERROR
3.7.0-hea74fb7_0  ERROR

3.6.8-h9f7ef89_7  GOOD
3.6.8-h9f7ef89_1  ERROR *DLL Change on by default*
3.6.8-h9f7ef89_0  GOOD
3.6.7-h9f7ef89_2  GOOD
3.6.7-h33f27b4_1  GOOD

However, I noticed that os.environ["PATH"] included C:\\Miniconda3\\envs\\INTERPRETER_ENV\\Library\\bin on Python 3.6. while it only had references to Spyder's env on Python 3.7 (there were no other differences than this one entry, and this was consistent between DLL search change on by default and all other builds). Manually adding the former to the latter got me the exact same results as with the spyder-kernels PR applied (see below); i.e. changing it manually to include that path did not work with the DLL search path change enabled but did work otherwise.

---

With that PR applied (adding the relevant PATH entries) iIt doesn't appear to be an issue with the logic adding the entries to the path, since they are still added just the same on all the builds, good or error (and is_anaconda(), is_conda = osp.exists(osp.join(sys.prefix, 'conda-meta')), returns True in either case):

3.7.3-h8c8aaf0_1  GOOD
3.7.3-h8c8aaf0_0  GOOD
3.7.2-h8c8aaf0_10 GOOD
3.7.2-h8c8aaf0_2  ERROR *DLL Change on by default*
3.7.2-h8c8aaf0_0  GOOD
3.7.1-h8c8aaf0_6  GOOD

3.6.8-h9f7ef89_7  GOOD
3.6.8-h9f7ef89_1  ERROR *DLL Change on by default*
3.6.8-h9f7ef89_0  GOOD
3.6.7-h9f7ef89_2  GOOD
3.6.6-hea74fb7_0  GOOD
3.6.5-h0c2934d_0  GOOD

---

Can I ask that you be specific about which change you mean here?

I was referring to this, which does indeed to be the primary source of the problem that doesn't appear easily solvable on our side.

the DLL changes (by which I mean the dropping of the automatic addition of of Library/bin to PATH) were done more than a year ago now.

I'm referring to the changed prompted in part by spyder-ide/spyder#7357 , which dropped in mid-February (less than three months ago), and the exact versions of Python mentioned that experiences the aberrant behavior are the ones you refer to in which it was enabled by default. Therefore, it seems having that enabled is the root of the issue here.

I believe that what changed was that conda 4.6.0 to 4.6.13 inclusive had a bug to do with activation mishandling PATH on Windows.

This problem has occurred consistently under conda 4.5.12 and 4.5.13. I don't believe there's any direct interaction with the conda executable involved here currently, as Spyder does not call it to activate the kernel environment (if it did, it seems doubtful we would be having this problem).

Forgive me if I misunderstand here, but I think Spyder wants to be stacking conda environments to achieve this single Spyder env, multiple kernels in multiple other envs?

Ideally, it would only have the PATH entries for the kernel's env, since it should only be able to access packages and binaries in the current running environment as would be the case if it had been activated normally from the Anaconda prompt. Otherwise, theoretically at least, packages could work (or otherwise behave differently) if they have access to DLLs in Spyder's env that are not actually present in the "isolated" kernel env. Doesn't seem like a very big problem, but it also makes the PATH longer than it needs to be and there's always some chance of a bug lurking there.