spyder 🚀 - Clean conda Windows install, followed by 'conda update --all' results in wrong mkl dlls being ...

but something in Spyder needs the dlls from mkl-2018.0.2-1 instead

We depend on Numpy to display arrays in our Variable Explorer, so that could be causing the problem here.

@mingwandroid, is there a problem with the latest MKL conda packages? Sorry to bother you, but it looks to be the case here.

ccordoba12 on 27 Jun 2018

No need to apologise Carlos! @msarahan is our mkl expert though, so I'm passing the buck here.

mingwandroid on 28 Jun 2018

I install Miniconda3 and then run conda install pandas.. i got the same issue with mkl

Missing ordinal 242 in Miniconda3\Library\bin\mkl_intel_thread.dll

this message appear when i try to import pandas or other package

VELCpro on 13 Jul 2018

@ccordoba12 @mingwandroid @msarahan This appears to be potentially more serious than previously stated, to the point of essentially making a Anaconda environment with mkl 2018.0.3 completely unusable. I installed Spyder 3.3.0 (conda-forge) and its optional dependencies (numpy, pandas, matplotlib, scipy, sympy and cython) into a fresh conda environment on my on my Win 8.1 x64 system, and while import pandas doesn't trigger the error message in that environment, anything with matplotlib will, among other things, e.g.

import matplotlib.pyplot as plt
ax = plt.subplot(111, projection='polar')

This not only happens in Spyder, but also QtConsole, ipython, and even a bare python interpreter; each crashes with the above two lines. I also tested in another new, clean environment with just Python 3.6.6 and matplotlib installed, and got the bug immediately on import matplotlib.pyplot as plt in the basic Python interpreter, crashing it. I then installed pandas, and got the same bug immediately upon importing it, again crashing the interpreter. I then downgraded to mkl 2018.0.2-1 from 2018.0.3-1 with conda, and everything worked fine.

So, users can fix it relatively painlessly (for now) by just running conda install mkl=2018.0.2 under Anaconda Prompt in any affected environment. However, this would appear to be very serious for those creating a new environment that uses numpy or other packages that depend on it, anyone running conda update --all, or anyone installing or updating a package that triggers an install of that new version, since it will cause their environment to be essentially broken and unusable unless and until they are aware of and implement the above fix.

CAM-Gerlach on 15 Jul 2018

👍3

This is not serious in the least, conda-forge and The Anaconda Distribution are incompatible with each other at present.

Please do not report bugs against such mixed installs until we announce full compatibility.

Is there any good reason for you to mix things like this?

mingwandroid on 15 Jul 2018

So, users can fix it relatively painlessly (for now) by just running conda install mkl=2018.0.2 under Anaconda Prompt in any affected environment. However, this would appear to be very serious for those creating a new environment that uses numpy or other packages that depend on it, anyone running conda update --all, or anyone installing or updating a package that triggers an install of that new version, since it will cause their environment to be essentially broken and unusable unless and until they are aware of and implement the above fix.

None of this applies if the rules are followed. Our users are meant to pick AD or CF at present (in any given environment, in fact that's largely one of the main benefits of conda environments; mutually incompatible software (developed by different teams with different approaches to compilers and binary compatibility, and btw, we handle that stuff far better than CF does) can be ring-fenced from each other.

mingwandroid on 15 Jul 2018

I installed Spyder 3.3.0 (conda-forge)

Why did you do that? Please pick one or the other for now. CF or AD, never both.

mingwandroid on 15 Jul 2018

This is not serious in the least, conda-forge and The Anaconda Distribution are incompatible with each other at present.

Well, that sounds like a pretty serious and high-priority bug if installing one package (and a few dependencies) from CF into a supposedly "isolated" conda environment can break an entire Anaconda install, including entirely separate, freshly created environments without any such mixing and just a small number of official AD packages. However, there may be something else going on that neither of us have considered.

Please do not report bugs against such mixed installs until we announce full compatibility.

Perhaps I misinterpret something, but not sure I understand the rationale for what I did wrong here based on the information I had available, in sharing my above observed findings on our own bug tracker on a problem affecting our own software, along with a potential workaround for our users that were experiencing an issue with the same.

Multiple users of our software apparently following reasonable good practices reported a seemingly concerning issue that affected it, which our maintainer indicated was likely with a third party, which you acknowledged above. Incidentally, I came across the same error under similar circumstances, and so followed up with a detailed further investigation of the problem that seemingly revealed it was potentially much more serious than initially thought, which in my testing affected multiple independent, freshly-created and minimal environments, and affected not only Spyder but any other Python interpreted in the environment.

While there was still a significant amount of uncertainty present (and thus I qualified my statements with "potentially", "apparently", etc) as to the nature and magnitude of the issue, given the potential for quite substantial impacts to both our own users and the wider Anaconda community, I considered it prudent to tag a couple of other individuals other participants had already named as being potentially interested parties, in case my findings were helpful in resolving the issue expeditiously. Somewhat ironically given the above statements, one of my other considerations in wanting to help head off any potential issue quickly was the number of previous problems with third-party packages outside our control that have resulted in sometimes up to dozens of bug reports (mostly not following our recommended practices) per day on our own plates that @ccordoba12 and I have had to deal with, despite our best efforts to stem the tide.

Is there any good reason for you to mix things like this?

As a matter of fact, I believe there seemed to be, at the time. As (among other things) the maintainer of Spyder's new docs, and the primary writer for the project's official blog, social media accounts, funding platforms, and other web properties, I was asked to update the former with the install and usage with Spyder for our newly independent spyder-kernels package, and needed to grab several screenshots of the new features in action for the latter, both of which practicably necessitated actually installing the new package from somewhere.

Despite it being well over a week from the PyPI release, the defaults-channel conda packages were still not out (the release of which we wanted the aforementioned blog post to shortly follow); pip was not a desirable option having heard to no end of the woe which it could bring to an Anaconda install (and after helping many Spyder users pick up the pieces after its ravages), and running from a Github clone wouldn't work without a prohibitive amount of setup and configuration due to the major spyder-kernels changes and lack of availability of that package on conda either, not to mention the need to create a new Anaconda environment for that anyway. Therefore, protected by (what I thought was) the relative safety of an isolated, clean conda-env, I gave installing Spyder 3.3.0 from conda-forge a shot, fully expecting the possibility of problems and prepared to either deal with them myself, try another approach in a fresh environment, or simply wait for the official packages to do the screenshots if I had no other choice.

Indeed, was at first an immediate crash due to incompatible qt builds, but switching that around resolved it and allowed Spyder to launch and work fine, up until the problem loading matplotlib. I normally would have dismissed it as a result of some conda-forge compatibility imperfection (as you did) and moved on, but I had happened to have just reviewed this issue thread in the process of starting to get back to my usual job of issue triage on this repo. Given that several other users had reported the same problem under much less questionable circumstances, and the concerning results of my consequent further investigation, it clearly seemed prudent to share my findings to make others like yourself aware in case they were indeed helping in resolving the issue, and assist our users with a potential workaround in the meantime. However, it seems you have found my concerns are not so worrisome after all, which would be be quite gratifying.

CAM-Gerlach on 15 Jul 2018

None of this applies if the rules are followed. Our users are meant to pick AD or CF at present.

As I discussed in my original report, the problem persisted across multiple fresh environments with only packages from the defaults channel, and the CF-channel packages (which did not, I note, include mkl or any of its related packages) were installed into a separate, isolated environment from everything else.

CAM-Gerlach on 15 Jul 2018

Why did you do that? Please pick one or the other for now. CF or AD, never both.

I'm not sure I understand why I'm being asked what appears to be the same thing twice. Are you saying I should maintain an entirely separate full Anaconda/Miniconda install just for conda-forge packages, not to mention other channels? That would seem to negate a lot of the Anaconda's advantages with environments and channels, not to mention quite inefficient in multiple respects.

CAM-Gerlach on 15 Jul 2018

In any case, back on topic, I did several hours of rather extensive testing on another machine, this one running Windows 10 x64 (on nearly identical hardware) and with a base Anaconda install from a roughly similar period as my other one, with some use, a handful of other packages and several conda environments present, but otherwise relatively "clean" as it mostly was used as a test machine.

I tried every procedure that produced a problem on my primary machine and several more besides (including essentially exact steps from before when I'd installed from conda-forge)l, all in separate, fresh environments, and was never able to reproduce the error. The only other distinction I can think of between the two machines was the Win10 one was running on a non-admin account, with Anaconda installed for all users from the admin account, but no problems or other differences seemingly resulted from that (aside from the need to authenticate to install shortcuts).

Meanwhile, back on my main machine, creating several new, fresh environments again with no conda-forge involved, and in new Anaconda prompt windows still manifests the same issues as before, unless and until the mkl version is rolled back by one in the relevant environment.

CAM-Gerlach on 15 Jul 2018

You need to show the output from conda list --show-channel-urls. To be clear, if I see defaults and conda-forge in that list then you need to remedy that. If you don't and see only defaults but still see this problem then report it at ContinuumIO/anaconda-issues here on github. If you see only conda-forge then I'm baffled!

mingwandroid on 15 Jul 2018

This is not serious in the least, conda-forge and The Anaconda Distribution are incompatible with each other at present.

Well, that sounds like a pretty serious and high-priority bug if installing one package (and a few dependencies) from CF into a supposedly "isolated" conda environment can break an entire Anaconda install, including entirely separate, freshly created environments without any such mixing and just a small number of official AD packages. However, there may be something else going on that neither of us have considered.

If it's a high priority bug it's not a bug that anyone can do something about. conda-forge is a separate entity from AD. But that's ok, we implemented environments for that. If a package isn't on one and it's high priority then that's the issue that can be fixed here, I cannot magically make CF and AD binary compatible (and attempts to reach that stage are progressing very slowly unfortunately).

mingwandroid on 15 Jul 2018

You need to show the output from conda list --show-channel-urls.

Like I've stated previously, I've repro'd this on my main machine in brand-new conda-envs with just python and matplotlib installed from defaults—it's pretty hard to mess that up, even for me, heh. Nevertheless, here you go as an example:

(mkl-test-2) C:\Users\C. A. M. Gerlach\Documents\dev>python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 11:27:44) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.pyplot as plt
Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll.

(mkl-test-2) C:\Users\C. A. M. Gerlach\Documents\dev>conda list --show-channel-urls
# packages in environment at C:\Anaconda3\envs\mkl-test-2:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl    defaults
ca-certificates           2018.03.07                    0    defaults
certifi                   2018.4.16                py36_0    defaults
cycler                    0.10.0           py36h009560c_0    defaults
freetype                  2.8                  h51f8f2c_1    defaults
icc_rt                    2017.0.4             h97af966_0    defaults
icu                       58.2                 ha66f8fd_1    defaults
intel-openmp              2018.0.3                      0    defaults
jpeg                      9b                   hb83a4c4_2    defaults
kiwisolver                1.0.1            py36h12c3424_0    defaults
libpng                    1.6.34               h79bbb47_0    defaults
matplotlib                2.2.2            py36h153e9ff_1    defaults
mkl                       2018.0.3                      1    defaults
mkl_fft                   1.0.2            py36hb217b18_0    defaults
mkl_random                1.0.1            py36h9258bd6_0    defaults
numpy                     1.14.5           py36h9fa60d3_3    defaults
numpy-base                1.14.5           py36h5c71026_3    defaults
openssl                   1.0.2o               h8ea7d77_0    defaults
pip                       10.0.1                   py36_0    defaults
pyparsing                 2.2.0                    py36_1    defaults
pyqt                      5.9.2            py36h1aa27d4_0    defaults
python                    3.6.6                hea74fb7_0    defaults
python-dateutil           2.7.3                    py36_0    defaults
pytz                      2018.5                   py36_0    defaults
qt                        5.9.6            vc14h62aca36_0    defaults
setuptools                39.2.0                   py36_0    defaults
sip                       4.19.8           py36h6538335_0    defaults
six                       1.11.0                   py36_1    defaults
sqlite                    3.24.0               h7602738_0    defaults
tornado                   5.0.2            py36hfa6e2cd_0    defaults
vc                        14                   h0510ff6_3    defaults
vs2015_runtime            14.0.25123                    3    defaults
wheel                     0.31.1                   py36_0    defaults
wincertstore              0.2              py36h7fe50ca_0    defaults
zlib                      1.2.11               h8395fce_2    defaults

In the first environment it happened in, Spyder and a few of its necessary dependencies were from conda-forge (aside from itself and spyder-kernels, these were freetype, icu, jpeg, keyright, libpng, libsodium, pywin32-cytpes, sqlite, zeromq, and zlib); otherwise, everything was from defaults.

In testing this, I was unable to create a pure conda-forge environment, even after creating and activating an empty environment, doing conda config --env --prepend channels conda-forge then conda install -c python=3.6 and finally conda install -c matplotlib, since several dependencies of matplotlib, namely the offending mkl package along with its Intel friends icc_rt, itnel-openmp, and the numpy and numpy-base that directly depended on it were all defaults only; my guess would be due to licensing issues. That environment, interestingly, failed with a different error:

(mkl-cf) C:\Users\C. A. M. Gerlach\Documents\dev>python
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 16:13:55) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib.pyplot as plt
Traceback (most recent call last):
  File "C:\Anaconda3\envs\mkl-cf\lib\site-packages\numpy\core\__init__.py", line 16, in <module>
    from . import multiarray
ImportError: DLL load failed: The operating system cannot run %1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda3\envs\mkl-cf\lib\site-packages\matplotlib\__init__.py", line 127, in <module>
    from . import cbook
  File "C:\Anaconda3\envs\mkl-cf\lib\site-packages\matplotlib\cbook\__init__.py", line 35, in <module>
    import numpy as np
  File "C:\Anaconda3\envs\mkl-cf\lib\site-packages\numpy\__init__.py", line 158, in <module>
    from . import add_newdocs
  File "C:\Anaconda3\envs\mkl-cf\lib\site-packages\numpy\add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "C:\Anaconda3\envs\mkl-cf\lib\site-packages\numpy\lib\__init__.py", line 8, in <module>
    from .type_check import *
  File "C:\Anaconda3\envs\mkl-cf\lib\site-packages\numpy\lib\type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "C:\Anaconda3\envs\mkl-cf\lib\site-packages\numpy\core\__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: DLL load failed: The operating system cannot run %1.

However, yet again, reverting to mkl version 2018.0.2 (still on defaults, since no conda-forge package appears to exist) fixed the problem and worked perfectly. I again tried the same steps on my Win 10 test machine, and both versions worked just fine with no errors. Therefore, it appears to be a combination of something in my software environment (either OS or other supporting packages, or something wrong with my Anaconda install, with which I've had no serious problems until today), and that specific package version, and it appears I'm not alone.

@VELCpro can you provide the same details as the OP did with regard to your various versions and dependencies (and conda list --show-channel-urls for the aforementioned environment) in case that helps narrow it down?

CAM-Gerlach on 15 Jul 2018

If it's a high priority bug it's not a bug that anyone can do something about. conda-forge is a separate entity from AD. But that's ok, we implemented environments for that.

Right, but to be clear as I've been stating all along, the problem with mkl 2018.0.3 occurs in multiple brand new conda environments that have nothing to do with CF or the similarly isolated environment any of its packages are installed in. Also, if it matters, I don't have any non-defaults dependencies installed in base (just a couple domain-specific packages from conda-forge and a few spyder testing and development related packages from spyder-ide, non of which shouldn't have any reverse dependencies that might actually pose a problem, nor were used or at all related to any of the above).

Anyway, its almost 7am here, so I probably should get some sleep :tired_face:

CAM-Gerlach on 15 Jul 2018

You should get some sleep, I agree!

as I've been stating all along, the problem with mkl 2018.0.3

I've demonstrated one environment where this cannot be reproduced which includes mkl 2018.0.3 and closely resembles the details given by you.

mingwandroid on 15 Jul 2018

You should get some sleep, I agree!

[x] Sleep acquired!

I've demonstrated one environment where this cannot be reproduced

Indeed; I've demonstrated an entire machine on which this cannot be reproduced. However, every one of the half-dozen environments I tested on my main machine does experience this issue on mkl 2018.0.3, while mkl 2018.0.0through .2 are all fine. The point I was trying to make there, was that I was experiencing the issue in multiple completely separate, all-defaults environments, as your statements ("But that's ok, we implemented environments for that.") seemed to suggest you still thought otherwise.

Therefore, as I discussed a few paragraphs above, this seems to be either a conda bug that has resulted in something wrong with my install that has persisted across environments, or something specific to the combination of my machine and that mkl version. To try to rule out the former, I tried to reproduce the exact series of steps used to setup the Spyder 3.3.0 conda-forge environment on my main machine as well as many other combinations besides, and trying other mkl versions on my main machine to see if any of them also had a problem, and neither produced any error. To attempt to address the latter, my other machine where it didn't happen had nearly identical hardware and a very similar Anaconda setup, although it had Win 10 (vs. 8.1), so did at least one other reporter here, which is why I inquired of them what software environment they were trying it under. Other than that, I'd truly baffled—I'd be normally incline to dismiss it as install-specific and move on, but the clear fix (downgrading) and that fact that several other users have reported it seem to suggest further investigation is warranted.

CAM-Gerlach on 15 Jul 2018

I'm not sure what we are talking about regarding mixed installs. I just did a fresh install and then did a "conda update --all" and everything broke as people described above. Are we not supposed to update the packages ourselves? The navigator showed that there was a new version that needed updating.

bigcatz on 2 Aug 2018

This is still an issue. I just used Anaconda3-5.2.0-Windows-x86_64 (downloaded on 09/13/2018). Then did conda update --all and spyder broke. I tried to use conda install mkl=2018.0.2 however that would involve downgrading numpy to 1.14 the original problem that I was trying to solve by updating all was to get numpy 1.15. Manually copying the dlls does not work.

jojker on 13 Sep 2018

spyder broke

Can you please be specific? Namely, did you experience the exact error message above, or what do you mean?

I tried to use conda install mkl=2018.0.2

What happened when you tried it? It did fix the problem, correct?

the original problem that I was trying to solve by updating all was to get numpy 1.15

Not that it necessarily contributed to this particular problem, but if you specifically, want to get NumPy 1.15, then just do conda update numpy (or conda install numpy=1.15).

CAM-Gerlach on 13 Sep 2018

@CAM-Gerlach Good catch, no it is not the same error, just the same cause. I failed to notice.

There are two error messages.

When I start spyder the ipython console fails to load with this traceback:

An error ocurred while starting the kernel
Traceback (most recent call last):
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\__main__.py", line 11, in 
start.main()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\start.py", line 284, in main
kernel.initialize()
File "", line 2, in initialize
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 474, in initialize
self.init_io()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 326, in init_io
sys.stdout.flush()
AttributeError: 'NoneType' object has no attribute 'flush'
Traceback (most recent call last):
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\__main__.py", line 11, in 
start.main()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\start.py", line 284, in main
kernel.initialize()
File "", line 2, in initialize
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 474, in initialize
self.init_io()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 326, in init_io
sys.stdout.flush()
AttributeError: 'NoneType' object has no attribute 'flush'
Traceback (most recent call last):
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\__main__.py", line 11, in 
start.main()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\start.py", line 284, in main
kernel.initialize()
File "", line 2, in initialize
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 474, in initialize
self.init_io()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 326, in init_io
sys.stdout.flush()
AttributeError: 'NoneType' object has no attribute 'flush'
Traceback (most recent call last):
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\__main__.py", line 11, in 
start.main()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\start.py", line 284, in main
kernel.initialize()
File "", line 2, in initialize
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 474, in initialize
self.init_io()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 326, in init_io
sys.stdout.flush()
AttributeError: 'NoneType' object has no attribute 'flush'
Traceback (most recent call last):
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "C:\Users\XXXXX\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\__main__.py", line 11, in 
start.main()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\spyder_kernels\console\start.py", line 284, in main
kernel.initialize()
File "", line 2, in initialize
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\traitlets\config\application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 474, in initialize
self.init_io()
File "C:\Users\XXXXX\Anaconda3\lib\site‑packages\ipykernel\kernelapp.py", line 326, in init_io
sys.stdout.flush()
AttributeError: 'NoneType' object has no attribute 'flush'

In the anaconda prompt from which I call spyder I see:
js: NOt allowed to load local resource: file:///C:/Users/XXXXXX/Anaconda3/lib/site-packages/spyder/utils/help/static/css/default.css

Also I note that there is no "C:/Users/XXXXXX/Anaconda3/lib" folder, when installing anaconda on Windows 10 with the Anaconda3-5.2.0-Windows-x86_64.exe the correct folder is "C:/Users/XXXXXX/Anaconda3/Lib". I'm not sure how big a deal the capital letter is, but maybe it points in the right direction.

Lastly, the numpy version 1.15 thing turned out to be something completely unrelated. I was calling python modules from Matlab, what I really needed to do was to start Matlab from Anaconda prompt so that the environment variables would be correct.

I'm reinstalling Anaconda and will not use "conda update --all" until I know this bug has been fixed.

I should also add that everytime I run Anaconda prompt I run as administrator. Also I always start spyder from within Anaconda prompt.

jojker on 13 Sep 2018

@jojker, your errors are unrelated with what is discussed here. The first one is fixed by downgrading ipykernel with

conda install ipykernel=4.8.2

The second one is already fixed and it'll be part of our 3.3.2 version.

ccordoba12 on 13 Sep 2018

The first one is fixed by downgrading ipykernel with
conda install ipykernel=4.8.2

Thanks!

jojker on 13 Sep 2018

🎉1

Also I note that there is no "C:/Users/XXXXXX/Anaconda3/lib" folder, when installing anaconda on Windows 10 with the Anaconda3-5.2.0-Windows-x86_64.exe the correct folder is "C:/Users/XXXXXX/Anaconda3/Lib".

The Windows NTFS filesystem is case-insensitive (i.e. case does not matter), so it doesn't matter. However, it never hurts to ask like you did.

Glad you got it resolved!

CAM-Gerlach on 13 Sep 2018

A clean py3.6.6 install is broken with 2018.0.3 and 2019.0,

The ordinal 191 could not be located in the dynamic link library C:\conda\Library\bin\mkl_intel_thread.dll

2018.0.2 works. 2018 complains about ordinal 242, 2019 about 191.

e: note this is with miniconda and not the full-fat anaconda distribution.

brandondube on 17 Sep 2018

@ccordoba12 So is there anything that can be done on our or Anaconda's side, or is it due to some specific user-side action?

CAM-Gerlach on 17 Sep 2018

We can't do anything about this in Spyder. It has to be fixed in Anaconda.

ccordoba12 on 17 Sep 2018

Would it be helpful to file a bug report for Anaconda? Is there already one?

luciansmith on 17 Sep 2018

It may be helpful to install a different Visual Studio runtime. We have had some reports that installing the VS 2013 runtime helps.

conda install vs2013_runtime

msarahan on 17 Sep 2018

The problem exists when no Visual Stuido runtime is installed at all. Are you suggesting that installing it will fix the problem?

luciansmith on 17 Sep 2018

Try it?

msarahan on 17 Sep 2018

@msarahan @mingwandroid @ccordoba12 I had a Python 3.6, all-Anaconda defaults channel environment on Windows 8.1 x64 and conda 4.5.11 for running the current master of Spyder 4, which had only Spyder's dependencies (and nothing else) installed under Python 3.6, and of course everything worked. However, when I ran conda install matplotlib, which generated the following output:

Solving environment: done

## Package Plan ##

  environment location: C:\Anaconda3\envs\spyder-4-lsp

  added / updated specs:
    - matplotlib


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    mkl_fft-1.0.6              |   py36hdbbee80_0         120 KB
    blas-1.0                   |              mkl           6 KB
    numpy-base-1.15.2          |   py36h8128ebf_0         3.9 MB
    numpy-1.15.2               |   py36ha559c80_0          48 KB
    matplotlib-3.0.0           |   py36hd159220_0         6.7 MB
    ------------------------------------------------------------
                                           Total:        10.8 MB

The following NEW packages will be INSTALLED:

    blas:         1.0-mkl
    cycler:       0.10.0-py36h009560c_0
    freetype:     2.9.1-ha9979f8_1
    icc_rt:       2017.0.4-h97af966_0
    intel-openmp: 2019.0-118
    kiwisolver:   1.0.1-py36h6538335_0
    matplotlib:   3.0.0-py36hd159220_0
    mkl:          2019.0-118
    mkl_fft:      1.0.6-py36hdbbee80_0
    mkl_random:   1.0.1-py36h77b88f5_1
    numpy:        1.15.2-py36ha559c80_0
    numpy-base:   1.15.2-py36h8128ebf_0

The following packages will be UPDATED:

    certifi:      2018.8.13-py36_0      --> 2018.8.24-py36_1

I then started Spyder (via boostrap.py) and it immediately crashed with the following dialog:

and output (before it didn't get to print any of the usual startup messages, except fort the very first print statement we have at the beginning of the script, and none of the others immediately after):

Executing Spyder from source checkout
Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll.

I then tried the following, all of which had no effect:

Upgrading to Python 3.7
Running conda install -f mkl
Downgrading to mkl 2018.0.3 (2018.0.2 isn't available for Python 3.7)
Re-upgrading to the latest mkl
Running conda install vs2013_runtime as you suggested,
Downgrading a number of versions to match my base env and other envs which work fine, each individually and in combination:
- matplotlib 2.2.2
- mkl-service 1.1.2
- numpy and numpy-base 1.14.6
- mkl_fft 1.0.2 (earliest available for Python 3.7)
conda update --all, after this was all done

Therefore, I am uncertain on how to proceed.

where mkl_core.dll outputs

C:\Anaconda3\envs\spyder-4-lsp\Library\bin\mkl_core.dll
C:\Anaconda3\Library\bin\mkl_core.dll

where mkl_intel_thread.dll outputs

C:\Anaconda3\envs\spyder-4-lsp\Library\bin\mkl_intel_thread.dll
C:\Anaconda3\Library\bin\mkl_intel_thread.dll

conda list mkl outputs

# Name                    Version                   Build  Channel
mkl                       2019.0                      118
mkl_fft                   1.0.6            py37hdbbee80_0
mkl_random                1.0.1            py37h77b88f5_1

Thanks!

CAM-Gerlach on 14 Oct 2018

That's pretty confusing. When you run spyder's bootstrap.py, do you have the env activated? If not, it may be falling through to the root env, which may have an older, out-of-date mkl. What are the versions of mkl-related stuff in the root env?

Maybe spyder's way of subprocessing or something is escaping the activated env somehow? Can you try:

install just a plain, clean miniconda root installation somewhere
install your deps there into the root env
remove all Anaconda-related entries from PATH
launch Anaconda Prompt for that miniconda
run your spyder stuff from that Anaconda prompt

msarahan on 14 Oct 2018

When you run spyder's bootstrap.py, do you have the env activated? If not, it may be falling through to the root env, which may have an older, out-of-date mkl. What are the versions of mkl-related stuff in the root env?

@msarahan Thanks for your quick response. I considered something like that at first, but the fact that the environment was activated, the first entry on both where calls was the correct dll from the environment (whereas if environment activation fails for some reason, silently or not, where will only return the base path), and most importantly the path listed on the actual error dialog was to the environment's copy of the dll (that existed and was the correct, signed and verified version 2019.0), it seemed very unlikely. I also tested it in a fresh, stock Anaconda prompt and also had the same negative result. EDIT: Further results make this possibility yet more unlikely; see below

In my base env I have the same versions of mkl_random, mkl-service, blas, and icc_rt (as well as cycler and kiwisolver), and after downgrading and restesting, the same version of matplotlib (2.2.2) and intel-openmp (2018.0.0). The closest I was able to get mkl itself was 2018.0.3 (the version with hte first reported problems above and elsewhere) vs 2018,0.2 in base, and mkl_fft 1.0.2 vs. 1.0.1, due to limited availability of Python 3.7 builds. I couldn't go further without downgrading back to 3.6 (and I was trying to test Python 3.7 after all); similarly, I could only get numpy 1.14.5 (vs 1.13.3) and numpy-base 1.14.5 (vs. 1.14.3).

However, I can just go ahead and downgrade my test env back to 3.6 the exact same versions of those packages (plus numpy, matplotlib, etc for a good measure) and see if the problem reoccurs. If it does not, that would narrow it down to either a problem with mkl >=2018.0.3 / mkl_fft >=1.0.2 / numpy >1.14.3, or an issue like that you describe, with versions somewhat still falling through to the incorrect ones, which can be further investigated by alternatively upgrading base and the test env and testing the results. DONE, see below

EDIT: Okay, so downgrading those 4 packages (plus Python to 3.6) did indeed get things working again. ~~I will now try upgrading each of them to their previous versions, to try to isolate the culprit(s), Once I find it/them, I will upgrade my base env to match and see if it resolves anything.~~ DONE, see below

Also, the call that triggers the error must be happening between line 75 and line 123 of bootstrap.py, given the print statements on those respective lines. However, there are no obvious culprits that would be calling anything outside the os module of the standard library, except for get_git_revision and its spyder.utils.vcs module on the line before and at the print statement on 123. Refined further; see below.

CAM-Gerlach on 14 Oct 2018

@msarahan Okay, so updating numpy to 1.14.3 in base and the test env still worked, but updating it and numpy-base to 1.14.4 required a later version of mkl, as did updating mkl_fft. Updating just mkl to 2018.0.3 did not result in this specific error and the consequent Spyder crash, and nor did updating mkl_fft to 1.0.2. However, updating numpy and numpy-base to 1.14.4 (from 1.14.3) was the culprit (and it continued to occur even if mkl_fft was downgraded back to 1.0.1.

Then, I updated base to numpy and numpy-base 1.14.4, which also updated mkl to 2018.0.3 but nothing else. Then, still in base, upon attempting to launch spyder 3,3,1 release version from the normal conda package that had always worked previously, I indeed get the same error, which is also still the case from my test env with the same versions. Therefore, the one common factor in the error is NumPy (and numpy-base) >=1.14.4, while the error does not occur with <=1.14.3 of both, regardless of environment. Downgrading just numpy and numpy-base in base (and the test env) to 1.14.3 again made that particular error disappear in the respective environment.

With mkl at 2018.0.3 and numpy/numpy base at 1.14.3 in both base and my test env (and every aforementioned package at identical versions too), launching Spyder 4 latest master in my test environment and Spyder 3.3.1 in base didn't produce the error dialog at first and crash, but Spyder 4 in the test env did produce it much later in the startup process (perhaps at the attempted automatic import of matplotlib somewhere) and continued loading normally. For both, the console started fine, as did running import numpy as np and some basic numpy functions (e.g.test_array = np.array([4, 3, 2, 3])).

After running import matplotlib.pyplot as plt (which did not display an error), I ran plt.plot(test_array) or plt.plot([3, 2, 3, 4]) on both and got the Ordinal 241 error dialog yet again, and the kernel also crashed, displaying simply b'' as the An error occurred while starting the kernel message on Spyder 4 and Kernel died, restarting with a successful auto restart on Spyder 3/base. Installing mkl 2018.0.2 on each env in turn resolved those issues on their respective envs, while the versions of each had no effect on the other. However, these versions of numpy and mkl are not available for Python 3.7 (the last such versions of each, in fact), thus preventing this being a viable solution even as a band-aid.

Anyway, the specific line in bootstrap.py that triggers the error is indeed from spyder.utils.vcs import get_git_revision, in turn, the line there is from spyder.utils import programs, which drilling further implicates the line from spyder.config.utils import is_anaconda, in turn calling the failing line spyder_kernels.utils import iofuncs. This file, then, is the ultimate culprit.

Barring the unlikely possibility the error is triggered at from spyder_kernels.py3compat import getcwd, pickle, PY2, to_text_string, the offending line is very probably 158, import numpy as np, as it actually executes at import time while the remaining scipy-related code not inside a def block does not, because scipy is not installed in the environment and thus the rest of the block would be skipped.

Does this help any?

CAM-Gerlach on 14 Oct 2018

@msarahan @mingwandroid Thanks for your help so far. Any progress on this? Is there anything more you need from me or that I can otherwise do to help?

Now ``conda install numpy-base-1.14.3 numpy=1.14.3 isn't even available under python=3.7 even with mkl 2018.0.3. This issue is utterly crippling to my ability to test and develop Spyder under Python 3.7 and Anaconda, since under Python 3.7 I can't even run the test suite locally nor do anything numpy/pandas/matplotlib/etc. related, which is a large part of Spyder's functionality, and I've done everything I and everyone else could think of to rule out user error or an issue with my environment, short of a full on clean re-install that would require hours upon hours of time, effort and disruption to download, install and setup my environments again and I hope to avoid if at all possible.

Looking forward to hearing further. Thanks!

CAM-Gerlach on 23 Oct 2018

@aperchak reported another instance of the issue on our Gitter:

I'm having a problem launching Spyder 3 - getting the error message: "The ordinal 242 could not be located in the dynamic link library". I searched a bit around the web and I understand this has something to do with a new update? I could swear it worked just the other day. Any solutions to that? I'm just taking my first steps in Python.

I'm using Windows 10.0 build 17134, and Anaconda 5.3, Python 3.7, in 64-bit - the most recent one available on anaconda.com/download.

I instructed them to run conda install numpy=1.14.3 numpy-base=1.14.3 mkl=2018.0.2 and they responded

I tried the command, and although Spyder now loads up and seems to be operating fine, a new error has popped up upon launching - "The procedure entry point mkl_blas_dgem2vu could not be located in the dynamic link library". I did notice some error string upon typing "import matplotlib", which is required for my Intro to Python course.

This closely parallels what I observed above, when I had 2018.0.3 installed alongside numpy 1.14.3 .

CAM-Gerlach on 24 Oct 2018

Hi all. Just prefacing that to say I'm a novice would be an understatement. I downloaded the most recent version available on anaconda.com/download (Anaconda 5.3, Python 3.7, in 64-bit) on a PC of mine, and Spyder is launching without any error messages. This to contrast an installation on a different PC where I'm receiving the error messages described here (ordinal 242...). Both versions were installed cleanly, both on the same Windows 10.0 17134 build. I've included the conda list output from the working version:

# Name Version Build Channel
_ipyw_jlab_nb_ext_conf 0.1.0 py37_0
alabaster 0.7.11 py37_0
anaconda 5.3.0 py37_0
anaconda-client 1.7.2 py37_0
anaconda-navigator 1.9.2 py37_0
anaconda-project 0.8.2 py37_0
appdirs 1.4.3 py37h28b3542_0
asn1crypto 0.24.0 py37_0
astroid 2.0.4 py37_0
astropy 3.0.4 py37hfa6e2cd_0
atomicwrites 1.2.1 py37_0
attrs 18.2.0 py37h28b3542_0
automat 0.7.0 py37_0
babel 2.6.0 py37_0
backcall 0.1.0 py37_0
backports 1.0 py37_1
backports.shutil_get_terminal_size 1.0.0 py37_2
beautifulsoup4 4.6.3 py37_0
bitarray 0.8.3 py37hfa6e2cd_0
bkcharts 0.2 py37_0
blas 1.0 mkl
blaze 0.11.3 py37_0
bleach 2.1.4 py37_0
blosc 1.14.4 he51fdeb_0
bokeh 0.13.0 py37_0
boto 2.49.0 py37_0
bottleneck 1.2.1 py37h452e1ab_1
bzip2 1.0.6 hfa6e2cd_5
ca-certificates 2018.03.07 0
certifi 2018.8.24 py37_1
cffi 1.11.5 py37h74b6da3_1
chardet 3.0.4 py37_1
click 6.7 py37_0
cloudpickle 0.5.5 py37_0
clyent 1.2.2 py37_1
colorama 0.3.9 py37_0
comtypes 1.1.7 py37_0
conda 4.5.11 py37_0
conda-build 3.15.1 py37_0
conda-env 2.6.0 h36134e3_1
console_shortcut 0.1.1 3
constantly 15.1.0 py37h28b3542_0
contextlib2 0.5.5 py37_0
cryptography 2.3.1 py37h74b6da3_0
curl 7.61.0 h7602738_0
cycler 0.10.0 py37_0
cython 0.28.5 py37h6538335_0
cytoolz 0.9.0.1 py37hfa6e2cd_1
dask 0.19.1 py37_0
dask-core 0.19.1 py37_0
datashape 0.5.4 py37_1
decorator 4.3.0 py37_0
defusedxml 0.5.0 py37_1
distributed 1.23.1 py37_0
docutils 0.14 py37_0
entrypoints 0.2.3 py37_2
et_xmlfile 1.0.1 py37_0
fastcache 1.0.2 py37hfa6e2cd_2
filelock 3.0.8 py37_0
flask 1.0.2 py37_1
flask-cors 3.0.6 py37_0
freetype 2.9.1 ha9979f8_1
get_terminal_size 1.0.0 h38e98db_0
gevent 1.3.6 py37hfa6e2cd_0
glob2 0.6 py37_0
greenlet 0.4.15 py37hfa6e2cd_0
h5py 2.8.0 py37h3bdd7fb_2
hdf5 1.10.2 hac2f561_1
heapdict 1.0.0 py37_2
html5lib 1.0.1 py37_0
hyperlink 18.0.0 py37_0
icc_rt 2017.0.4 h97af966_0
icu 58.2 ha66f8fd_1
idna 2.7 py37_0
imageio 2.4.1 py37_0
imagesize 1.1.0 py37_0
incremental 17.5.0 py37_0
intel-openmp 2019.0 118
ipykernel 4.10.0 py37_0
ipython 6.5.0 py37_0
ipython_genutils 0.2.0 py37_0
ipywidgets 7.4.1 py37_0
isort 4.3.4 py37_0
itsdangerous 0.24 py37_1
jdcal 1.4 py37_0
jedi 0.12.1 py37_0
jinja2 2.10 py37_0
jpeg 9b hb83a4c4_2
jsonschema 2.6.0 py37_0
jupyter 1.0.0 py37_7
jupyter_client 5.2.3 py37_0
jupyter_console 5.2.0 py37_1
jupyter_core 4.4.0 py37_0
jupyterlab 0.34.9 py37_0
jupyterlab_launcher 0.13.1 py37_0
keyring 13.2.1 py37_0
kiwisolver 1.0.1 py37h6538335_0
lazy-object-proxy 1.3.1 py37hfa6e2cd_2
libcurl 7.61.0 h7602738_0
libiconv 1.15 h1df5818_7
libpng 1.6.34 h79bbb47_0
libsodium 1.0.16 h9d3ae62_0
libssh2 1.8.0 hd619d38_4
libtiff 4.0.9 h36446d0_2
libxml2 2.9.8 hadb2253_1
libxslt 1.1.32 hf6f1972_0
llvmlite 0.24.0 py37h6538335_0
locket 0.2.0 py37_1
lxml 4.2.5 py37hef2cd61_0
lzo 2.10 h6df0209_2
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
markupsafe 1.0 py37hfa6e2cd_1
matplotlib 2.2.3 py37hd159220_0
mccabe 0.6.1 py37_1
menuinst 1.4.14 py37hfa6e2cd_0
mistune 0.8.3 py37hfa6e2cd_1
mkl 2019.0 118
mkl-service 1.1.2 py37hb217b18_5
mkl_fft 1.0.4 py37h1e22a9b_1
mkl_random 1.0.1 py37h77b88f5_1
more-itertools 4.3.0 py37_0
mpmath 1.0.0 py37_2
msgpack-python 0.5.6 py37he980bc4_1
msys2-conda-epoch 20160418 1
multipledispatch 0.6.0 py37_0
navigator-updater 0.2.1 py37_0
nbconvert 5.4.0 py37_1
nbformat 4.4.0 py37_0
networkx 2.1 py37_0
nltk 3.3.0 py37_0
nose 1.3.7 py37_2
notebook 5.6.0 py37_0
numba 0.39.0 py37h830ac7b_0
numexpr 2.6.8 py37h9ef55f4_0
numpy 1.15.1 py37ha559c80_0
numpy-base 1.15.1 py37h8128ebf_0
numpydoc 0.8.0 py37_0
odo 0.5.1 py37_0
olefile 0.46 py37_0
openpyxl 2.5.6 py37_0
openssl 1.0.2p hfa6e2cd_0
packaging 17.1 py37_0
pandas 0.23.4 py37h830ac7b_0
pandoc 1.19.2.1 hb2460c7_1
pandocfilters 1.4.2 py37_1
parso 0.3.1 py37_0
partd 0.3.8 py37_0
path.py 11.1.0 py37_0
pathlib2 2.3.2 py37_0
patsy 0.5.0 py37_0
pep8 1.7.1 py37_0
pickleshare 0.7.4 py37_0
pillow 5.2.0 py37h08bbbbd_0
pip 10.0.1 py37_0
pkginfo 1.4.2 py37_1
pluggy 0.7.1 py37h28b3542_0
ply 3.11 py37_0
prometheus_client 0.3.1 py37h28b3542_0
prompt_toolkit 1.0.15 py37_0
psutil 5.4.7 py37hfa6e2cd_0
py 1.6.0 py37_0
pyasn1 0.4.4 py37h28b3542_0
pyasn1-modules 0.2.2 py37_0
pycodestyle 2.4.0 py37_0
pycosat 0.6.3 py37hfa6e2cd_0
pycparser 2.18 py37_1
pycrypto 2.6.1 py37hfa6e2cd_9
pycurl 7.43.0.2 py37h74b6da3_0
pyflakes 2.0.0 py37_0
pygments 2.2.0 py37_0
pylint 2.1.1 py37_0
pyodbc 4.0.24 py37h6538335_0
pyopenssl 18.0.0 py37_0
pyparsing 2.2.0 py37_1
pyqt 5.9.2 py37h6538335_2
pysocks 1.6.8 py37_0
pytables 3.4.4 py37he6f6034_0
pytest 3.8.0 py37_0
pytest-arraydiff 0.2 py37h39e3cac_0
pytest-astropy 0.4.0 py37_0
pytest-doctestplus 0.1.3 py37_0
pytest-openfiles 0.3.0 py37_0
pytest-remotedata 0.3.0 py37_0
python 3.7.0 hea74fb7_0
python-dateutil 2.7.3 py37_0
pytz 2018.5 py37_0
pywavelets 1.0.0 py37h452e1ab_0
pywin32 223 py37hfa6e2cd_1
pywinpty 0.5.4 py37_0
pyyaml 3.13 py37hfa6e2cd_0
pyzmq 17.1.2 py37hfa6e2cd_0
qt 5.9.6 vc14h1e9a669_2 [vc14]
qtawesome 0.4.4 py37_0
qtconsole 4.4.1 py37_0
qtpy 1.5.0 py37_0
requests 2.19.1 py37_0
rope 0.11.0 py37_0
ruamel_yaml 0.15.46 py37hfa6e2cd_0
scikit-image 0.14.0 py37h6538335_1
scikit-learn 0.19.2 py37heebcf9a_0
scipy 1.1.0 py37h4f6bf74_1
seaborn 0.9.0 py37_0
send2trash 1.5.0 py37_0
service_identity 17.0.0 py37h28b3542_0
setuptools 40.2.0 py37_0
simplegeneric 0.8.1 py37_2
singledispatch 3.4.0.3 py37_0
sip 4.19.8 py37h6538335_0
six 1.11.0 py37_1
snappy 1.1.7 h777316e_3
snowballstemmer 1.2.1 py37_0
sortedcollections 1.0.1 py37_0
sortedcontainers 2.0.5 py37_0
sphinx 1.7.9 py37_0
sphinxcontrib 1.0 py37_1
sphinxcontrib-websupport 1.1.0 py37_1
spyder 3.3.1 py37_1
spyder-kernels 0.2.6 py37_0
sqlalchemy 1.2.11 py37hfa6e2cd_0
sqlite 3.24.0 h7602738_0
statsmodels 0.9.0 py37h452e1ab_0
sympy 1.1.1 py37_0
tblib 1.3.2 py37_0
terminado 0.8.1 py37_1
testpath 0.3.1 py37_0
tk 8.6.8 hfa6e2cd_0
toolz 0.9.0 py37_0
tornado 5.1 py37hfa6e2cd_0
tqdm 4.26.0 py37h28b3542_0
traitlets 4.3.2 py37_0
twisted 18.7.0 py37hfa6e2cd_1
unicodecsv 0.14.1 py37_0
urllib3 1.23 py37_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.15.26706 h3a45250_0
wcwidth 0.1.7 py37_0
webencodings 0.5.1 py37_1
werkzeug 0.14.1 py37_0
wheel 0.31.1 py37_0
widgetsnbextension 3.4.1 py37_0
win_inet_pton 1.0.1 py37_1
win_unicode_console 0.5 py37_0
wincertstore 0.2 py37_0
winpty 0.4.3 4
wrapt 1.10.11 py37hfa6e2cd_2
xlrd 1.1.0 py37_1
xlsxwriter 1.1.0 py37_0
xlwings 0.11.8 py37_0
xlwt 1.3.0 py37_0
yaml 0.1.7 hc54c509_2
zeromq 4.2.5 he025d50_1
zict 0.1.3 py37_0
zlib 1.2.11 h8395fce_2
zope 1.0 py37_1
zope.interface 4.5.0 py37hfa6e2cd_0

Is there any way I could be of help in supplying info from the working version? Both Anaconda's were installed cleanly, from the direct link, and involved no steps in between the moment of download to the moment of launching Spyder. On one PC Spyder is working, on another Spyder pops the 'ordinal 242' error message and fails to launch. Thank you.

aperchak on 25 Oct 2018

Do you have any (non-Metro UI) applications, drivers, utilities etc. installed on one machine and not the other? Its possible that the dlls could conflict. In particular, the program "Amplitube" was reported to have caused this over on the Anaconda issue, ContinuumIO/anaconda-issues#8561 .

Also, please post a screenshot or the full exact error message, and the output of where mkl_core.dll and where mkl_intel_thread in the environment in which you're trying to run Spyder (e.g. base by default, if you're running it from the shortcut).

Further, you can try the following:

Rebooting your machine (in case you haven't since you installed Anaconda)
Run conda install vs2013_runtime
Make sure you're still on the nominal versions of everything for anaconda: conda install anaconda=5.3
Downgrade Python to 3.6: conda install python=3.6
Create a new clean environment: conda create -n spyder-test-env python=3, activate it, then install Spyder with conda install spyder.
Uninstalling and reinstalling Anaconda

CAM-Gerlach on 25 Oct 2018

@aperchak informed us that they indeed have "Amplitude" installed on the suspect machine, so its almost certainly that for their case per ContinuumIO/anaconda-issues#8561 .

CAM-Gerlach on 25 Oct 2018

Good old amplitude! We need to figure out a way around this..

mingwandroid on 25 Oct 2018

I tried reporting this to ik multimedia, but they just offered a work around. I wanted them to change their installation.

Get Outlook for Androidhttps://aka.ms/ghei36

From: Ray Donnelly notifications@github.com
Sent: Wednesday, October 24, 2018 7:36:18 PM
To: spyder-ide/spyder
Cc: bigcatz; Comment
Subject: Re: [spyder-ide/spyder] Clean conda Windows install, followed by 'conda update --all' results in wrong mkl dlls being used for Spyder (#7357)

Good old amplitude! We need to figure out a way around this..

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/spyder-ide/spyder/issues/7357#issuecomment-432873588, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Am2qUzAlfuFGyj8-cv-zX64iAkUT1pTsks5uoQeCgaJpZM4U5AJU.

bigcatz on 25 Oct 2018

they just offered a work around.

What did they suggest?

@mingwandroid Indeed; thanks for your attention to this problem. Unfortunately, for both my own case and other users on the Anaconda thread, Amplitude is not installed and in fact a search revealed no files with mkl in the name anywhere inside the entire Windows directory; where can't find any other mkl-related dlls on my path, this can be reproduced in multiple clean environments (including base, and in environments with the same versions of everything) and throughout extensive testing, and I've tried everything suggested short of a complete reinstall without anything fixing it, other than the procedure described above (downgrading numpy, mkl, etc)—and even that only now appears to work on Python 3.6, not 3.7 due to unavailability of packages.

CAM-Gerlach on 25 Oct 2018

So yeah, Amplitude was definitely the culprit here too as I uninstalled the software and the error was gone.
I even reinstalled Amplitude thereafter and the error message popped again. Said error:

errorbfr

As per request, I ran the where commands before uninstalling Ampltiude and after.
With Amplitude:
bframp
Without Amplitude:
aftamp

There seems to be a duplication of the files with Amplitude installed, but I'll leave it to you experts to decipher what's that all about.

Thanks for the help.

aperchak on 26 Oct 2018

Hello everyone, I keep on running into exactly the same issue after installing Anaconda 3 on my old Windows 10 machine. Already tried everything but keep on getting "The ordinal 242 was not found in the DLL []", as soon as I try to run any code that imports numpy in my jupyter notebook.

error message

I already tried the above mentioned where commands to see where my mkl_core.dll and mkl_intel_thread.dll where located and got this:

where_files

Any help would be greatly appreciated as I really need to be able to import numpy for a course I am currently taking! Thanks a lot for the help.

janm97 on 14 Nov 2018

@janm97 Did you try conda install mkl=2018.0.2?

CAM-Gerlach on 14 Nov 2018

❤2

@CAM-Gerlach Thanks a lot! That fixed it.

janm97 on 15 Nov 2018

We have spoken with Intel about this issue. They said that there are some backwards incompatible changes in mkl 2019. We had been operating under the assumption that software built against 2018 would work with 2019, but that is not necessarily true. We will be rebuilding packages so that constraints are more accurate, and so thata complete set of software built with mkl 2019 is available.

msarahan on 15 Nov 2018

👍4

How did this get through testing? Can we figure that out?

mingwandroid on 15 Nov 2018

We will be rebuilding packages so that constraints are more accurate, and so thata complete set of software built with mkl 2019 is available.

Great news Mike!! Thanks for the followup.

ccordoba12 on 15 Nov 2018

@mingwandroid I believe it got through testing because as @luciansmith described, the installer itself is fine. It is a self-consistent set. Somehow updating brings in some other software that might be built against mkl 2018, and that triggers the issue. I'm not sure exactly why an update would use an older MKL, but I'll try to reproduce.

msarahan on 15 Nov 2018

Thanks so much @msarahan ! I really appreciate it! Let me know if you need more testing data from me.

CAM-Gerlach on 15 Nov 2018

@janm97 Did you try conda install mkl=2018.0.2?

This fixed my problem. Thank-you.

JimVee on 15 Nov 2018

I'm having the same problem. Just did a brand new installation of Anaconda 3.7, then tried installing spyder from Anaconda Navigator and got the error mentioned above. I tried using the "Conda install mkl=2018.0.2" mentioned above and got the following error:

(base) C:\Users\Rodrigo>conda install mkl=2018.0.2
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:

mkl=2018.0.2
numpy-base
python=3.7
Use "conda info " to see the dependencies for each package.

rpcoelho17 on 10 Jan 2019

Spyder comes pre-installed with Anaconda and therefore does not need to be installed. Did you use Minicoda instead here?

mingwandroid on 10 Jan 2019

No I used the full Anaconda. I appologize, the error occurs when you click launch in the Anaconda Navigator to run spyder.

rpcoelho17 on 10 Jan 2019

got the error mentioned above.

As you can see from your error message, this is not the exact same error, as the mentioned DLL is different. Post the full output of where libiomp5md.dll run in Anaconda prompt here; it is possible that it is due to another program's DLLs conflicting with the indicated one rather than anything to do with mkl itself.

UnsatisfiableError: The following specifications were found to be in conflict:

This is due to mkl < 2018.0.3 not being built for Python 3.7. You can try the above suggestions to fix the root problem, and if that doesn't work do conda install python=3.6 mkl=2018.0.2 though you'll be stuck on Python 3.6 until the root problem (either with mkl or with your system, whichever is the case here) is fixed.

you click launch in the Anaconda Navigator to run spyder.

FYI, AFAIK there is no advantage to launch Spyder via the method, as it is vastly faster to merely launch it via the Start menu shortcut on Windows like with any other application (just tap Win and start typing spyder ) than going through the extra layer of complexity that Navigator poses as well as potential for additional errors and problems, unless you need to launch it in a specific environment and don't want to use Anaconda prompt for that.

CAM-Gerlach on 10 Jan 2019

Here is the output of the were command in the Anaconda prompt:

(base) C:\Users\Rodrigo>where libiomp5md.dll
C:\Users\Rodrigo\Anaconda3\Library\bin\libiomp5md.dll
C:Windows\System32\libiomp5md.dll

Running spider directly from windows (outside of Navigator) generates the same error.

Will running "Conda install python=3.6 mkl=2018.0.2" replace the current 3.7 installation or create a new environment? I would like to avoid having many environments and would be ok with having just 3.6 on my machine.

Thanks for the help, by the way! :)

rpcoelho17 on 11 Jan 2019

Here is the output of the were command in the Anaconda prompt:

Thanks. As you can see, another program on your machine has installed a DLL with an identical name in directly in the System32 directory, which is a very bad practice and a recipe for all sorts of problems. I'm not sure what program it could be, but there are various ways to find out. In any case, there is nothing we can do about this on our end; this needs to be fixed on your machine by uninstalling the offending other program, removing the DLL (and seeing what breaks), etc.

Will running "Conda install python=3.6 mkl=2018.0.2" replace the current 3.7 installation or create a new environment?

It will modify the current default (base) environment.

CAM-Gerlach on 11 Jan 2019

I'm not sure what program it could be, but there are various ways to find out.

I would bet that the program in question is called Amplitube. I've tried to contact the developers of this software to ask them not to do this egregious act but have been unable to get any response from them. I might have to renew my efforts. Ideally MKL's redistribution license would prohibit this!

mingwandroid on 11 Jan 2019

https://mobile.twitter.com/_tmp_account/status/1083817363047370754

Please help to spread this if you can!

mingwandroid on 11 Jan 2019

I would bet that the program in question is called Amplitube.

I thought the problematic DLLs with Amplitube were mkl_intel_core, mkl_intel_thread and mkl_def so I wasn't sure if there were others, but I guess so if it uses all of mkl.

Ideally MKL's redistribution license would prohibit this!

Have you considered mentioning it to your Intel contacts from that end?

CAM-Gerlach on 11 Jan 2019

We have. We're trying to figure out DLL load tricks that will prevent the problem. We will also explore application manifests, but that's hard to get right - you'd have to re-generate the whole manifest for all libraries in an env any time that any of them were updated.

msarahan on 11 Jan 2019

Okay, great—thanks for the update @msarahan . Meanwhile, any progress figuring out what's going on in the cases like mine where there aren't any conflicting DLLs in System32 (or anywhere else on the where path) that could be causing the problem? Anything else you want me to try? Thanks!

CAM-Gerlach on 12 Jan 2019

I have a fresh Anaconda (2018.12) install on Windows 10 with this issue, and I do not have Amplitube installed (never heard of it). The fix for me was finding libiomp5md.dll in my system32 folder and renaming it. No idea which program uses it, but hopefully I don't have to find out. Otherwise, simply reverting to mkl 2018.0.2 and python 3.6 didn't completely help (I could at least import numpy, but certain functions still caused errors on loading).

bverem on 13 Jan 2019

👍1

I have a fresh Anaconda (2018.12) install on Windows 10 with this issue, and I do not have Amplitube installed (never heard of it). The fix for me was finding libiomp5md.dll in my system32 folder and renaming it. No idea which program uses it, but hopefully I don't have to find out. Otherwise, simply reverting to mkl 2018.0.2 and python 3.6 didn't completely help (I could at least import numpy, but certain functions still caused errors on loading).

I had the same problem "missing ordinal 242 in Anaconda3\Library\binmkl_intel_thread.dll" and renaming the libiomp5md.dll file in the system32 folder has fixed it!

chucku on 20 Jan 2019

@CAM-Gerlach sorry, I've got nothing. The one other DLL we've seen causing this issue is the openmp runtime, libiomp5md.dll

msarahan on 21 Jan 2019

Okay, thanks @msarahan . I guess I should just do a full, clean reinstall of Anaconda; I've just been putting it off since I don't want to have to re-create all my environments and such.

CAM-Gerlach on 21 Jan 2019

I really don't think that will help. I think the conflict is with other software on your system. Reinstalling Anaconda won't fix those conflicts.

msarahan on 21 Jan 2019

Holy s%%t! Amplitube uninstalled and Python libraries suddenly work!!!!!! Thanks!!!!! Troubled me entire month.

"message about a missing ordinal 242 in Anaconda3\Library\binmkl_intel_thread.dll"

GeorgePowhattan on 21 Jan 2019

Confirmed with the latest Anaconda Python 3 64-bit from www.anaconda.com/download/:

I have a way how to replicate this problem (with Anaconda Python in my PATH), just open cmd and

C:\Users\sobriety>python
Python 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import matplotlib.pyplot as plt
>>> plt.show()
>>> plt.plot([])
[<matplotlib.lines.Line2D object at 0x000001BD881038D0>]
>>> plt.show()

And we got "missing ordinal 242 in Anaconda3\Library\binmkl_intel_thread.dll".

Doesn't look like libiomp5md.dll problem:

C:\Users\sobriety>where libiomp5md.dll
C:\Users\sobriety\AppData\Local\Continuum\anaconda3\Library\bin\libiomp5md.dll

No amplitube present:

C:\Users\sobriety>conda remove amplitube
Solving environment: failed

PackagesNotFoundError: The following packages are missing from the target environment:
  - amplitube

This has fixed the problem:

Conda install python=3.6 mkl=2018.0.2

ChaoticRoman on 21 Jan 2019

@ChaoticRoman Amplitube isn't an Anaconda package, so it naturally wouldn't be uninstalled by conda remove amplitube. It needs to be checked for removed through Programs and Features.

Doesn't look like libiomp5md.dll problem:

No, because the error you got was

missing ordinal 242 in Anaconda3\Library\binmkl_intel_thread.dll

Therefore, you should check where mkl_core.dll and where mkl_intel_thread.dll and post the results.

Also, check your Programs and Features list for Amplitube, or any of the other applications reported here. Thanks.

CAM-Gerlach on 21 Jan 2019

Thank you for quick reply, CAM-Gerlach!

Therefore, you should check where mkl_core.dll and where mkl_intel_thread.dll and post the results.

Right now, after the Conda install python=3.6 mkl=2018.0.2, there is

C:\Users\sobriety>where mkl_core.dll
C:\Users\sobriety\AppData\Local\Continuum\anaconda3\Library\bin\mkl_core.dll

C:\Users\sobriety>where mkl_intel_thread.dll
C:\Users\sobriety\AppData\Local\Continuum\anaconda3\Library\bin\mkl_intel_thread.dll

Should I somehow revert and try it again with failing bits in place?

Also, check your Programs and Features list for Amplitube, or any of the other applications reported here. Thanks.

No discussed programs seen, it is clean Windows 10 Pro x64 installation and only this has been installed:

Install the latest Anaconda Python 3 64-bit
- https://www.anaconda.com/download/
- During the installation add Anaconda to PATH environment variable and
  
  register it as a default Python
Install the latest DAQNavi Driver for USB-4716
- https://support.advantech.com/support/DownloadSRDetail_New.aspx?SR_ID=1-FCIIF6&Doc_Source=Download
Install the latest Advantech Multimeter
- https://support.advantech.com/support/DownloadSRDetail_New.aspx?SR_ID=1-FO7FQ3&Doc_Source=Download
- It is not strictly necessary but is very useful for troubleshooting
- Connect and test your USB-4716 with Advantech Multimeter
Install Virtual COM Port emulator and create two virtual COM port pairs
- https://sourceforge.net/projects/com0com/files/latest/download
- COM3/COM4 and COM5/COM6 (avoid CNCA#-styled names)
- Check "use Ports class" checkbox for both ports
Install Mercury RT Dream Edition x64 using provided installator.
Install IDS camera driver.

ChaoticRoman on 21 Jan 2019

Thanks for the detailed reply, @ChaoticRoman . That's rather puzzling, indeed. Can you open the Anaconda prompt from the Start Menu, run the above commands in Python, and make sure you still see the error?

Assuming you do, can you post the output of echo %PATH% from Anaconda prompt in the environment in which you are running the above commands and getting the error (presumably base by default)?

The basic idea is you'll then want to search any of the directories identified on the %PATH% that aren't Anaconda-related for any .dlls that contain the string omp or mkl and report the results. I think I've actually found the culprit in my install, even though those commands came back clean. Thanks!

CAM-Gerlach on 21 Jan 2019

@mingwandroid @msarahan I finally isolated and resolved the issue on my machine; it was the report involving OpenMP that clued me in.

To make a long story short, I found a number of instances of OpenMP-related DLLs a few different places on my %PATH%, and after a good deal of testing, one in particular turned out to be the culprit: C:\Windows\System32\libiomp5md.dll, despite it not being referenced in the error message. Everything works fine now.

So, I can't be certain since I've uninstalled it many years ago, but some sleuthing revealed that GoPro Studio (at least older versions) appears to have been the probable original culprit here that people should check for. Also, interestingly enough, Intel Visual Fortran Redistributables on IA-32 (included with a proprietary scientific application, not sure which) also includes its DLLs in a common location which it places on the %PATH%; although it doesn't seem to trigger this particular problem, it has dozens of DLLs that are all in common with those in Library/bin/, as Anaconda no doubt uses the same Intel shared libraries for MKL/OpenMP/etc.

Also, FYI @ChaoticRoman you should be able to trigger the error by simply executing

import matplotlib.pyplot as plt
plt.plot([])

The other lines were not necessary, at least for me.

Here is "the long story" detailing the specifics of what I found, for anyone (morbidly) curious...

For the successfully working environment with mkl 2018.0.2 and Python 3.6.7 on Windows 8.1 x64 and the following abridged list of possibly relevant package versions:

blas                      1.0                         mkl
icc_rt                    2017.0.4             h97af966_0
intel-openmp              2018.0.0                      8
matplotlib                3.0.2            py36hc8f65d3_0
mkl                       2018.0.2                      1
mkl-service               1.1.2            py36hb217b18_4
mkl_fft                   1.0.1            py36h452e1ab_0
mkl_random                1.0.1            py36h9258bd6_0
mpmath                    1.0.0                    py36_2
numpy                     1.14.3           py36h9fa60d3_1
numpy-base                1.14.3           py36h555522e_1
python                    3.6.7                h33f27b4_1
vc                        14.1                 h0510ff6_3
vs2013_runtime            12.0.21005                    1
vs2015_runtime            15.5.2                        3

...Running ``where libiomp5md.dll'' returns (version numbers of each DLL in brackets):

[5.0.2017.525]  C:\Anaconda3\envs\spyder-4-lsp\Library\bin\libiomp5md.dll
[5.0.2017.525]  C:\Anaconda3\Library\bin\libiomp5md.dll
[5.0.2014.710]  C:\Program Files (x86)\Common Files\Intel\Shared Libraries\redist\ia32\compiler\libiomp5md.dll  # Evidently installed by Intel Visual Fortran Redistributables on IA-32, which was in turn installed by a proprietary scientific-related package (unsure which)
[5.0.2009.811]  C:\Windows\System32\libiomp5md.dll  # Most likely related to the long-removed GoPro Studio video software, which I installed to get access to the CineForm professional codec, given I installed it around this time and it has a known issue with putting this DLL in System32

where vcomp140.dll (VS2015) returns

[14.14.26405.0]  C:\Anaconda3\envs\spyder-4-lsp\vcomp140.dll
[14.14.26405.0]  C:\Anaconda3\envs\spyder-4-lsp\Library\bin\vcomp140.dll
[14.14.26405.0]  C:\Anaconda3\vcomp140.dll
[14.14.26405.0]  C:\Anaconda3\Library\bin\vcomp140.dll
[14.16.27012.6]  C:\Windows\System32\vcomp140.dll  # Installed by Visual Studio build tools which I installed as instructed to build Python C extensions; well after this problem first appeared

and where vcomp120.dll (VS2013) returns

[12.0.21005.1]  C:\Anaconda3\envs\spyder-4-lsp\vcomp120.dll
[12.0.21005.1]  C:\Anaconda3\envs\spyder-4-lsp\Library\bin\vcomp120.dll
[12.0.21005.1]  C:\Program Files\ImageMagick-7.0.8-Q16\vcomp120.dll  # Installed with ImageMagick and added to PATH more recently than problem first appeared
[12.0.21005.1]  C:\Windows\System32\vcomp120.dll  # Possibly related to GoPro Studio or other such video encoding software installed around this time and long since removed

Searching all processes with Process Explorer for any using DLLs that include the string omp returned only Python processions using the Anaconda versions, so there doesn't appear to be anything too important using the other DLLs on the path at present.

From here, if I run conda update mkl, which resolves to mkl 2019.1 (144), Conda wants to downgrade numpy to 1.14.2 (from 1.14.3) while upgrading numpy-base to 1.15.4 (also from 1.14.3) (I didn't actually execute the change);

  environment location: C:\Anaconda3\envs\spyder-4-lsp

  added / updated specs:
    - mkl


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    numpy-1.14.2               |   py36h5c71026_0         3.7 MB
    numpy-base-1.15.4          |   py36hc3f5095_0         3.9 MB
    mkl-2019.1                 |              144       158.3 MB
    scipy-1.1.0                |   py36h29ff71c_2        13.3 MB
    mkl-service-1.1.2          |   py36hb782905_5         134 KB
    ------------------------------------------------------------
                                           Total:       179.4 MB

The following packages will be UPDATED:

    icc_rt:      2017.0.4-h97af966_0   --> 2019.0.0-h0cc432a_1
    mkl:         2018.0.2-1            --> 2019.1-144
    mkl-service: 1.1.2-py36hb217b18_4  --> 1.1.2-py36hb782905_5
    numpy-base:  1.14.3-py36h555522e_1 --> 1.15.4-py36hc3f5095_0
    scipy:       1.1.0-py36h672f292_0  --> 1.1.0-py36h29ff71c_2

The following packages will be DOWNGRADED:

    numpy:       1.14.3-py36h9fa60d3_1 --> 1.14.2-py36h5c71026_0

conda update numpy numpy-base states # All requested packages already installed. and conda install numpy=1.15.4 spends about 5-10 minutes Solving environment and finally spits out

UnsatisfiableError: The following specifications were found to be in conflict:
  - mkl_fft=1.0.1
  - numpy=1.15 -> mkl_fft[version='>=1.0.4']

So then I executed conda install mkl_fft=1.0.4, which also installed mkl 2018.0.3 and nothing else:

  environment location: C:\Anaconda3\envs\spyder-4-lsp

  added / updated specs:
    - mkl_fft=1.0.4


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    mkl_fft-1.0.4              |   py36h1e22a9b_1         120 KB
    mkl-2018.0.3               |                1       178.1 MB
    ------------------------------------------------------------
                                           Total:       178.2 MB

The following packages will be UPDATED:

    mkl:     2018.0.2-1           --> 2018.0.3-1
    mkl_fft: 1.0.1-py36h452e1ab_0 --> 1.0.4-py36h1e22a9b_1

I can now reproduce the error (Ordinal 241/mkl_intel_thread.dll) in the Python interpreter by simply running

import matplotlib.pyplot as plt
plt.plot([])

I checked and none of the aforementioned DLL versions or files were modified, just the MKL ones.

After testing changing the names of the various non-Anaconda DLLs, I found the culprit: C:\Windows\System32\libiomp5md.dll. I can't be 100% sure since I've uninstalled the application itself many years ago, but I strongly suspect it was installed by GoPro Studio, a gratis video editing/encoding application which I once needed in order to get access to their Cineform professional video intermediate codec. The created/modified and last accessed dates roughly match up with when I installed and used it, and it has a known issue with installing this specific DLL in the System32 directory and causing conflicts with other video-related applications.

CAM-Gerlach on 22 Jan 2019

Yep, other applications on PATH won't interfere with Anaconda Prompt, because Anaconda Prompt puts our entries up front. Doing that would be too disruptive as a global modification to PATH - that's why we recommend Anaconda Prompt. Running our other start menu shortcuts does similar stuff.

System32 is higher in priority than anything on PATH (https://docs.microsoft.com/en-us/windows/desktop/dlls/dynamic-link-library-search-order#standard-search-order-for-desktop-applications). It doesn't matter what we try to do with PATH if a 3rd party has put DLLs into System32. Manifests may still help us if we can figure that out someday.

Thansk for your patience, @CAM-Gerlach

msarahan on 22 Jan 2019

[...]open the Anaconda prompt from the Start Menu, run the above commands in Python, and make sure you still see the error?

Everything is fine after Conda install python=3.6 mkl=2018.0.2.

Assuming you do, can you post the output of echo %PATH% from Anaconda prompt in the environment in which you are running the above commands and getting the error (presumably base by default)?

OK, uninstalling Anaconda and installing it again. I have find out that we have these crashes in my application scripts but not while trying normal replication . But I have find out that if working directory is a place, where we have these scripts, it does crash:

C:\Program Files\MercuryRT-v2.7DE>python
Python 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib
Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll.

C:\Program Files\MercuryRT-v2.7DE>

[...]any .dlls that contain the string omp or mkl and report the results. I think I've actually found the culprit in my install, even though those commands came back clean. Thanks!

C:\Program Files\MercuryRT-v2.7DE>dir *omp*.dll *mkl*.dll
 Volume in drive C has no label.
 Volume Serial Number is A432-6ABC

 Directory of C:\Program Files\MercuryRT-v2.7DE

08.10.2018  12:55           885 096 libiomp5md.dll
08.10.2018  12:56           176 512 vcomp140.dll
20.07.2018  11:01           152 064 vtkCommonComputationalGeometry-8.1.dll

So this seems to be the actual problem, not Anaconda itself, but running our scripts from bad location.

ChaoticRoman on 22 Jan 2019

Everything is fine after conda install python=3.6 mkl=2018.0.2.

Sorry, what I said was very confusing. What I meant was to switch back to the current version of mkl and run the commands

import matplotlib.pyplot as plt
plt.plot([])

from the Anaconda prompt, not the regular command line, just in case there was some issue with your %PATH% that the latter would avoid. However, that's a moot point now.

can you post the output of echo %PATH% from Anaconda prompt

OK, uninstalling Anaconda and installing it again.

Sorry, what? How did "post the output of echo %PATH%" get interpreted as "reinstall Anaconda? As @msarahan said about my case, this isn't going to fix the problem if it is a conflict, as seemed likely was the case (and in fact was).

So this seems to be the actual problem, not Anaconda itself, but running our scripts from bad location.

Yes. The problem is that the DLL search path order will always search the current directory before the PATH, so it will find those DLLs instead of the correct Anaconda ones. To resolve this, just make sure not to run your scripts from the same directory as those conflicting DLLs, or you're likely to run into more issues in the future.

CAM-Gerlach on 22 Jan 2019

Yep, and the ultimate solution is finding ways to overcome the problems with the search order, such as manifest files that specify particular versions, so that the globally installed ones won't get used if they are not compatible.

msarahan on 22 Jan 2019

Yeah, I was wondering if there was a way to simply check the version string of the first found DLL and fall through to the next one if it doesn't match what the package is compiled against. But I figured it was Microsoft so it couldn't be that easy, heh.

CAM-Gerlach on 22 Jan 2019

@CAM-Gerlach, we (Anaconda) have a potential workaround for the Windows DLL search order. It's in our latest python builds on the c3i_test2 anaconda.org channel:

python 2.7.15 build 7
3.6.8 build 1
3.7.2 build 1

Try updating python using the c3i_test2 channel:

conda update python -c c3i_test2

mingwandroid on 30 Jan 2019

Those are great new Ray!! How did you manage to do it?

ccordoba12 on 30 Jan 2019

https://github.com/AnacondaRecipes/python-feedstock/blob/master/recipe/0020-Add-CondaEcosystemModifyDllSearchPath.patch

msarahan on 30 Jan 2019

Mike beat me to it. Interestingly, numpy had adopted a very similar approach but backed off from doing it.

For sure it's a nuclear option, but the other idea I had is far worse (DLL hooking of kernel32.dll - if that's even possible at present, and if it is, if it would remain possible in the future, and there's also a big question as to whether or not virus scanners would class doing this as a virus).

mingwandroid on 30 Jan 2019

Thanks Mike!! It looks very interesting!

So what are you trying to do exactly? Putting Anaconda's Library dir in front of DLL's search path?

ccordoba12 on 30 Jan 2019

We are changing the DLL search order from:

dirname(exe_path)
C:\Windows\System32
PATH

.. to ..

dirname(exe_path)
PATH (which usually ends with C:\Windows\System32 anyway)

mingwandroid on 30 Jan 2019

@mingwandroid @msarahan Thanks so much! I tested it by removing the .bak extension I'd added to the offending DLL, ensuring the error was triggered with the current Python (3.6.7), and then upgrading to the 3.6.8 build from that channel. Indeed, no error occurred when either trying the aforementioned simplified procedure from the standard Python console that previously produced it, nor launching and running numpy/matplotlib commands from Spyder.

BTW, I got a conda error when trying to switch Python back to the defaults version, but I suspect its unrelated since its happened before several times recently after flipping package versions around. (As far as I know, I had closed Spyder and all Python processes in that environment first, and made sure no Anaconda-related directories were opened in my file manager, upgrading to the latest conda version did not resolve it either, and I couldn't find many relevant search results).

Executing transaction: / WARNING conda.gateways.disk:exp_backoff_fn(50): Uncaught backoff with errno EEXIST 17
failed
ERROR conda.core.link:_execute(543): An error occurred while uninstalling package 'c3i_test2::python-3.6.8-h9f7ef89_1'.
FileExistsError(17, 'Cannot create a file when that file already exists')
Attempting to roll back.

Rolling back transaction: done

[WinError 183] Cannot create a file when that file already exists: 'C:\\Anaconda3\\envs\\spyder-4-lsp\\DLLs\\_decimal.pyd' -> 'C:\\Anaconda3\\envs\\spyder-4-lsp\\DLLs\\_decimal.pyd.c~'

I fixed it, as I did in previous cases, by just doing find . -type f -name "*.c~" -delete on the /c/Anaconda3/envs/spyder-4-lsp directory and re-running the install command. For reference, the files found:

./DLLs/select.pyd.c~
./DLLs/sqlite3.dll.c~
./DLLs/unicodedata.pyd.c~
./DLLs/_lzma.pyd.c~
./DLLs/_multiprocessing.pyd.c~
./DLLs/_overlapped.pyd.c~
./DLLs/_socket.pyd.c~
./DLLs/_sqlite3.pyd.c~
./DLLs/_ssl.pyd.c~
./Lib/site-packages/mkl_fft/_pydfti.cp36-win_amd64.pyd.c~
./Lib/site-packages/numpy/_mklinit.cp36-win_amd64.pyd.c~
./Library/bin/libifcoremd.dll.c~
./Library/bin/mkl_rt.dll.c~
./python3.dll.c~
./python36.dll.c~
./pythonw.exe.c~

Thanks!

CAM-Gerlach on 30 Jan 2019

Thank you for the feedback.

mingwandroid on 31 Jan 2019

['System32libiomp5md.dll', 'System32mkl_core.dll', 'System32mkl_def.dll','System32mkl_intel_thread.dll', 'SysWOW64libiomp5md.dll','SysWOW64mkl_core.dll','SysWOW64mkl_intel_thread.dll', 'SysWOW64mkl_p4.dll', 'SysWOW64mkl_p4m.dll','SysWOW64mkl_p4m3.dll', 'SysWOW64mkl_p4p.dll']

Remove all these ddl files from C:Windows\SysWOW64 and C:WindowsSystem32

Just do this and restart the spyder once again and hurrrahhhh!!! ur problem will be solved and spyder will open without showing any ordinal 242 error............just do it it is the only solution..

tirthshah147 on 4 Feb 2019

👍1

Remove all these ddl files

I strongly advise against the removing the DLLs in case they break something you need (something had to put them there) and instead adding the .bak extension.

C:Windows\SysWOW64

FYI, this isn't necessary unless for some odd reason you are using the legacy 32-bit version of Anaconda, which you really shouldn't be (unless you're on a >10 year old machine that's unlikely to be able to run Anaconda adequately anyway, or accidentally bought a 32-bit version of Windows that MS keeps around for enterprises running ancient 16-bit DOS software from >3 decades ago.

it it is the only solution..

As you can see in the previous few comments, @mingwandroid stated and I confirmed this should be worked around by the upcoming next release of the Anaconda python interpreter executable, without having to do any of this.

CAM-Gerlach on 4 Feb 2019

@mingwandroid It seems there is a big discussion going on about this upstream...you involved in that?

CAM-Gerlach on 12 Feb 2019

@CAM-Gerlach we're not sure about our new approach with the library loading. It may fix a lot of people, but it may also break a lot more. We're leaning right now towards providing the new changed capability disabled by default, but switchable with an environment variable. I'll link to docs here when we have them.

msarahan on 20 Feb 2019

PS: I'm having trouble reproducing the error. I would like to try to catch it somehow in order to direct people who encounter the problem to our new docs. Do you have any specific tips on how to reproduce this? I have tried to install ancient MKL into my System32 folder (2018, 2017, even 11), but in a clean python 3.7, numpy 1.15.4, MKL 2019 env, I can't get the error to come up. Procmon shows the old libraries getting loaded from System32 - I'm just confused as to how the error is not showing up. Is there a specific numpy operation that triggers the issue for you?

PS: ancient MKL included intel-openmp (libiomp5md.dll). I have also tried installing older versions there, too.

msarahan on 20 Feb 2019

Thanks for the update @msarahan . Will the docs discuss what to look out for in terms of the types of usecases where enabling this change to the DLL search order for Anaconda would break desired functionality?

CAM-Gerlach on 20 Feb 2019

The major one we've found was with pytorch and cuda. https://github.com/pytorch/pytorch/issues/17051

Aside from this, it just scares me to have our software override whatever is in system32. To be sure, the root problem here of 3rd party vendors installing things to system32 is bad, but we should not create potential security/driver confusion by trying to compensate for that.

msarahan on 20 Feb 2019

@msarahan The fastest way I found to reliably trigger it was python then

import matplotlib.pyplot as plt
plt.plot([])

You might need to go older with your DLL versions. I attached the libiomp5md.dll that triggers it for me; its from 2009 vintage.

libiomp5md.dll.bak.zip

CAM-Gerlach on 20 Feb 2019

Aside from this, it just scares me to have our software override whatever is in system32.

I mean, if I understand correctly its only overriding the DLL loading order for A. Code running on the Anaconda Python interpreter, and B. To only move the local, Anaconda-controlled DLL search directories above System32 (do you check that its somewhere on the PATH already, and if not add it at the end just in case?).

Isn't the worst that can happen (which, isn't good but isn't the end of the world) is it breaks specific packages running under Anaconda that rely on hacky DLL search/load tricks themselves, or for some reason need the version of a DLL in Sysetem32 to appear higher in the path than anything locally?

And in terms of security/driver issues, I assume a package could cause a DoS (or much worse, with some effort) by shipping a bad or malicious ntdll.dll or something, but at that point if it can drop arbitrary DLLs on the system then it could drop any other malicious payload instead anyway, and as far as I understand they are only loaded in the context and permissions of the Python interpreter running the process, which the malicious package presumably already effectively controls?

Or is it possible to drop copies of malicious DLLs locally inside the Anaconda environment on the search path, and then that important DLL get loaded instead of the real one if another Anaconda Python interpreter process performs a low-level system call, in order to intercept data or escalate privileges (e.g. wait for the user in an admin install to run conda as admin, then take control of that process via a malicious low-level DLL dropped on the PATH? I'm no security expert, much less do I have much knowlege about the arcane internals of Windows, but I'm wondering what's really possible or not if this change goes through.

CAM-Gerlach on 20 Feb 2019

@mingwandroid Ray, I may have a use case that isn't solved by the patch.

I recently had to install the Intel Fortran compiler, and even with the c3i_test2 version I have the same dll error.

λ where mkl_core.dll
C:\Program Files x86)\IntelSWTools\compilers_and_libraries_2016.1.146\windows\redist\ia32\mkl\mkl_core.dll
C:\Users\<username>\Programs\Anaconda3\Library\bin\mkl_core.dll

Since the offending dll is not in the System32 directory, is this falling outside of what the patch is trying to accomplish? My initial (albeit uneducated) guess would be that somehow ifort is still taking precedence in the system, but I don't know nearly enough about it to know how to fix that.

dolson14 on 26 Feb 2019

@dolson14 Take all of this with a grain of salt since I'm no Windows DLL expert like @mingwandroid and @msarahan , nor am I an Anaconda employee, just a Spyder dev.

However, I'm not sure how the compiler is injecting its location higher onto the path than Anaconda. Assuming conda activate is working correctly, it should always be higher in the search order than anything else, at least when launching anything from the Anaconda prompt, except if the user has run a script/program of some sort after activateing the environment that deliberately set the path differently. In that case, its not really something that can be solved on Anaconda's side in my limited understanding, except by using very techniques like manifests and the like.

Have you run any commands, scripts or programs after activateing your environment? If you type conda activate your-env-name with the latest version of conda, and then run where mkl_core.dll again, do you see the same result? Are you getting the error in Spyder, or something else? What packages have you installed in this environment (conda list --show-channel-urls)? Does it persist in other environments?

CAM-Gerlach on 26 Feb 2019

@dolson14 is likely not activating the environment. PATH works in a line. Activating lets us come first without raising hell with other software. Not activating means you're at the mercy of order, where generally only appending is safe, and you're thus in a situation where the earliest installed software has higher priority than newer stuff.

msarahan on 26 Feb 2019

👍1

Thanks @msarahan . So if that's the case, wouldn't @dolson14 have to have selected the "Add Anaconda to PATH" option in the installer in order for it to show up on the PATH (such that where would find it) at all, as it does above? @dolson14 can you confirm, as I mention above, that you've activated your environment, and whether you selected the "Add to PATH" option when installing Anaconda?

CAM-Gerlach on 26 Feb 2019

Ah, that actually makes sense. In fact, I tried adjusting my shell to prepend the python path directories as opposed to postpending them, and that solved the problem. Looks like activating is the better solution, so I'll look into how to do that the right way in my shell environment. Thanks!

dolson14 on 26 Feb 2019

@dolson14 conda activate <env-name> (e.g. base for the root env). Although, Anaconda prompt should activate base for you automatically...

CAM-Gerlach on 26 Feb 2019

@CAM-Gerlach that much I knew already. =) My problem is that I prefer using a different shell environment than the default Anaconda prompt; I use cmder due to its linux-like feel and that it gets bundled with ssh, git, and a bunch of other very useful tools.

For reference, in case is looking for how to do this, I just put this line into cmder/config/user-profile.cmd:

call "<path to Anaconda>\condabin\activate.bat"

That activates the python environment just by opening a terminal.

dolson14 on 26 Feb 2019

@dolson14 Oh, I actually do the exact same thing with ConEmu + Git Bash (= Cmdr) + a custom task config that activates my conda environment and sets up my working directory). As long as I have it set up correctly, I've never had the issue you describe, but I'm not sure why you would conclude this is an Anaconda problem when it clearly stems from how your non-standard environment is set up.

Speaking of Git Bash/Cmdr—in case you couldn't tell by his name, the guy you quoted (Ray) was actually one of the core developers behind that very MSYS2 and Git for Windows that we're discussing. @mingwandroid Any chance we get a default/bundled Anaconda prompt that calls Git Bash rather than horrible old cmd.exe at some point in the future?

CAM-Gerlach on 26 Feb 2019

Spyder: Clean conda Windows install, followed by 'conda update --all' results in wrong mkl dlls being used for Spyder

Problem Description

What steps reproduce the problem?

What is the expected output? What do you see instead?

Paste Traceback/Error Below (if applicable)

Versions

Dependencies

Most helpful comment

All 105 comments

Related issues