conda update spyder
(or pip
, if not using Anaconda)jupyter qtconsole
(if console-related)spyder --reset
On a fresh copy of macOS 10.15.5 ("Catalina"), with a fresh install of Anaconda, it does not appear to be possible to use what your docs call "the modular approach" to set the Python interpreter if Spyder is runing under Python 3.7 and the desired version of the interpreter is Python 3.8.
I have checked, and the Python interpreter I'm trying to use is marked as executable, and can be used in a terminal shell to launch a Python REPL session. There appears to be nothing "invalid" about it.
conda create -n test python=3.8 spyder-kernels
base
environment (either from terminal or through Anaconda Navigator GUI)Preferences > Python Interpreter > Use the following Python interpreter
to /Users/username/opt/anaconda3/envs/test/bin/python
Expected: consoles will use the Python 3.8 interpreter.
Actual: an error dialog saying:
You selected an invalid Python interpreter for the console so the previous interpreter will stay. Please make sure to select a valid one.
The message is accurate; the console interpreter has in fact stayed as the spyder-internal console (confirmed by import sys; print(sys.executable)
. Note that if step 2 above is changed to conda create -n test python=3.7 spyder-kernels
then it is possible to set the interpreter, so it appears to be something specific about Python 3.8 that is the problem.
Not applicable; internal console shows only startup greeting and empty prompt
# Mandatory:
applaunchservices >=0.1.7 : 0.2.1 (OK)
atomicwrites >=1.2.0 : 1.4.0 (OK)
chardet >=2.0.0 : 3.0.4 (OK)
cloudpickle >=0.5.0 : 1.5.0 (OK)
diff_match_patch >=20181111 : 20181111 (OK)
intervaltree : None (OK)
IPython >=4.0 : 7.16.1 (OK)
jedi =0.15.2 : 0.15.2 (OK)
nbconvert >=4.0 : 5.6.1 (OK)
numpydoc >=0.6.0 : 1.1.0 (OK)
parso =0.5.2 : 0.5.2 (OK)
pexpect >=4.4.0 : 4.8.0 (OK)
pickleshare >=0.4 : 0.7.5 (OK)
psutil >=5.3 : 5.7.0 (OK)
pygments >=2.0 : 2.6.1 (OK)
pylint >=0.25 : 2.4.4 (OK)
pyls >=0.31.9;<0.32.0 : 0.31.10 (OK)
qdarkstyle >=2.8 : 2.8.1 (OK)
qtawesome >=0.5.7 : 0.7.2 (OK)
qtconsole >=4.6.0 : 4.7.5 (OK)
qtpy >=1.5.0 : 1.9.0 (OK)
rtree >=0.8.3 : 0.9.4 (OK)
sphinx >=0.6.6 : 3.1.2 (OK)
spyder_kernels >=1.9.1;<1.10.0 : 1.9.1 (OK)
watchdog : None (OK)
zmq >=17 : 19.0.1 (OK)
# Optional:
cython >=0.21 : 0.29.20 (OK)
matplotlib >=2.0.0 : 3.2.2 (OK)
numpy >=1.7 : 1.18.5 (OK)
pandas >=0.13.1 : 1.0.5 (OK)
scipy >=0.17.0 : 1.5.0 (OK)
sympy >=0.7.3 : 1.6.1 (OK)
Note that this problem does not occur under Windows or Linux; I have successfully set up the "modular approach" on both of those OSes in the last couple of weeks; in both cases Spyder was installed in an environment running Python 3.7, and I was able to set the console to use Python 3.8 from a different environment.
I'm also somewhat puzzled because the following snippet works in a macOS python REPL:
>>> import sys
>>> sys.executable
'/Users/drmccloy/opt/anaconda3/bin/python'
>>> from spyder.utils import programs
>>> spyder_version = sys.version_info[0]
>>> args = ['-c', 'import sys; print(sys.version_info[0])']
>>> pyexec = '/Users/drmccloy/opt/anaconda3/envs/test/bin/python'
>>> proc = programs.run_program(pyexec, args)
>>> console_version = int(proc.communicate()[0])
>>> assert spyder_version == console_version
>>> print(spyder_version, console_version)
3 3
snippet adapted from here:
https://github.com/spyder-ide/spyder/blob/a5dfb15c3ba51d43db9975de689a57a772d71017/spyder/preferences/maininterpreter.py#L152-L173
Hi @drammock thanks for reporting this. That was actually a bug that we fixed in our latest version 4.1.4. It is not yet in the Anaconda defaults channel, but it should be in a few days.
However, if you don't want to wait, you can get it from Conda-forge on a new environment using:
conda create -n new_env -c conda-forge spyder=4.1.4
The problem with this is that if you want to change your python interpreter for that of a new environment using python 3.8, you will have to make sure that this one has the latest version of spyder-kernels too (which is only in conda-forge for now). As it is not recommended to mix packages from defaults and conda-forge you would want to create a new test environment:
conda create -n test1 -c conda-forge python=3.8 spyder-kernels
Thanks! Could you tell me specifically which version of spyder-kernels is required? And will that version of spyder-kernels also be hitting the defaults channel in a few days?
I can confirm that the following works:
conda create -c conda-forge -n spyder python=3.7 spyder nb_conda_kernels
conda create -c conda-forge -n test python=3.8 spyder-kernels
conda activate spyder
spyder
# then in the Spyder Preferences GUI set the interpreter to /Users/username/opt/anaconda3/envs/test/bin/python3.8
However, I still would like to know when we can expect suitable versions of spyder
and spyder-kernels
to both be available on the defaults
channel. Or at least, what is the minimum version of spyder-kernels that will be compatible with spyder 4.1.4 (which I think is the version that has this fix)?
The minimum version of spyder-kernels required for spyder 4.1.4 is 1.9.2.
It actually doesn't depend on us how long does it take for the new releases to be available in the Anaconda defaults channel. It usually takes about a week so hopefully by the end of this week it will be available. You can take a look at this page https://anaconda.org/anaconda/spyder to keep track it.
@drammock, please wait until 4.1.4 is available in Anaconda to try again. We did some improvements to the way we detect a valid Python interpreter, which could help in this case.
please wait until 4.1.4 is available in Anaconda to try again
@ccordoba12 That's all I can do, really. Many people who use MNE-Python like to use Spyder, but we often have dependency conflicts between MNE-Python and Spyder, so we recently changed our installation instructions to use your "modular approach"... but it broke things for Catalina users (we had tested it only on older versions of macOS). It is important that users migrating from MATLAB / users who are not too comfortable with the terminal should be able to run spyder from the base environment and have the spyder console use MNE-Python's environment (we create one called mne
by default) --- i.e., those users should not have to think about environments at all, they can just launch spyder and have their MNE-Python scripts "just work".
Of course, all these dependency conflicts would go away if there were an easy standalone binary installer for Spyder for all platforms. :)
@juanis2112, thanks for helping @drammock with his problem. You did a great job guiding him to check that this bug was effectively fixed in our 4.1.4 version. Sorry for stepping in without reading the whole context of this issue.
Of course, all these dependency conflicts would go away if there were an easy standalone binary installer for Spyder for all platforms. :)
Then you'll be happy to hear that we're working on our own installers for Windows and macOS for 4.2.0 (to be released in September).
I noticed that spyder 4.1.4 and spyder-kernels 1.9.2 dropped on the defaults channel today. So I tested them and, if anything, the results are now slightly worse than before. Here is confirmation that I'm using the correct versions:
(base) drmccloy@Daniels-MacBook-Pro ~ % conda list | grep spyder
spyder 4.1.4 py37_0
spyder-kernels 1.9.2 py37_0
(base) drmccloy@Daniels-MacBook-Pro ~ % conda activate test
(test) drmccloy@Daniels-MacBook-Pro ~ % conda list | grep spyder
spyder-kernels 1.9.2 py38_0
Now, when I launch Spyder from the base
environment (the only env where it's installed) and try to set the python executable, I get this error:
Invalid file path: /Users/drmccloy/opt/anaconda3/envs/test/bin/python3.8
I also get another dialog after a few seconds:
Completion and linting in the editor for Python files will not work during the current session, or stopped working. This problem could be fixed by restarting Spyder. Do you want to restart Spyder now?
What else can I do to help troubleshoot this? Again, here is evidence that there is nothing "invalid" about the path I'm trying to set:
(base) drmccloy@Daniels-MacBook-Pro ~ % conda activate test
(test) drmccloy@Daniels-MacBook-Pro ~ % which python
/Users/drmccloy/opt/anaconda3/envs/test/bin/python
(test) drmccloy@Daniels-MacBook-Pro ~ % python
Python 3.8.3 (default, Jul 2 2020, 11:26:31)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; print(sys.version_info)
sys.version_info(major=3, minor=8, micro=3, releaselevel='final', serial=0)
>>> import spyder_kernels
>>> spyder_kernels.__version__
'1.9.2'
It's a probably an issue with the code in our side that validates if an executable is a Python interpreter. And it appears to happen only on macOS and for Python 3.8 (it works fine for me on Linux).
Unfortunately, there's nothing to do about it for now, sorry. We'll try to fix it in our next release.
Yes, it works fine for me on Linux too, and in a Windows VM (as I mentioned above).
That was actually a bug that we fixed in our latest version 4.1.4.
Can you point me to the issue or pull request where it was thought that this was fixed? And describe or point me to the way you test this functionality? Getting this working is a priority for me so I'd like to look and see if I can figure out what's wrong.
The error should be here:
And our tests for that function are here:
I tried some line-by-line tests of is_python_interpreter
, run_program
, is_text_file
, and is_binary_string
. All seem to give the correct behavior when run interactively and given the executable path I'm trying to set (/Users/drmccloy/opt/anaconda3/envs/test/bin/python3.8
)
So next I set up a new environment running dev versions of spyder and spyder kernels:
conda create -n spyderdev python=3.7
conda install --only-deps spyder
conda remove --force spyder-kernels
cd $HOME/opt
git clone git://github.com/spyder-ide/spyder
git clone git://github.com/spyder-ide/spyder-kernels
cd spyder-kernels
pip install -e .
cd ../spyder
pip install -e .
Then I installed the dev version of spyder-kernels into the target environment too (since without it the app complained):
conda activate test
conda remove --force spyder-kernels
cd ../spyder-kernels
pip install -e .
Then, I launch spyder:
conda activate spyder-dev
cd ../spyder
python bootstrap.py
Under those conditions, I can get the properties setting to work: spyder (in a py 3.7 env on macOS) will allow using a py3.8 executable for its internal consoles. So I guess that is good news? This problem is turning out to be rather hard to debug.
Next I ran git grep
on the error message ("Invalid file path") and found that it likely originates here:
Which looks like it was changed 8 days ago and should have made it into the 4.1.4 release? So I'm a bit confused why I'm seeing the error when running spyder from base env (running 4.1.4) and not when running spyder from source. Any insight?
Which looks like it was changed 8 days ago and should have made it into the 4.1.4 release?
Yes, that change was included in 4.1.4
Any insight?
I'll take a look at it later today.
Hey @drammock, sorry for the big delay in coming back to you. I finally got a chance to test this and was unable to reproduce it.
The commands I used were:
conda create -n test-stable python=3.7 spyder
to create a stable env with packages from defaults
using Python 3.7 and Spyder.conda create -n test-external-py38 python=3.8 spyder-kernels
to create an env with Python 3.8 and spyder-kernels
.Then I went to our Preferences and was able to select the Python interpreter in test-external-py38
without problems.
Is this still a problem for you?
Well, at least it's failing differently now. On a fresh macOS Catalina VM, with a fresh download of Anaconda, I tried creating the two environments exactly as you said. Then I used the Anaconda navigator to select the test-stable
environment, launched Spyder from there, and used the preferences dialog to change the interpreter. It seemed to work, but then when I went to the console (which was still a py37 one) and pressed Ctrl+d
to kill it and respawn a new one, I got the following message in place of the console:
Your python environment or installation doesn't have the spyder-kernels module, or the right version of it installed (>= 1.9.4 and < 1.10.0). Without this module it is not possible for Spyder to create a console for you. You can install it by running in a system terminal:
conda install spyder-kernels
orpip install spyder-kernels
Closing and re-opening spyder did not change this. In the terminal, I checked, and the error message is inaccurate, I do have version 1.9.4:
% conda activate test-external-py38
% conda list | grep spyder-kernels
spyder-kernels 1.9.4 py38_0
Could you upload an animated of the entire process, after resetting Spyder preferences?
You can use a program called Licecap for that.
@ccordoba12 here is the screencapture video https://drive.google.com/file/d/1dmOz5Pnl5fBOs_Gg24PRDmyeArDfmXK9/view?usp=sharing
@drammock, thanks for taking the time to do this. I'll try to take a look at it this weekend.
any movement here? I notice the issue is still tagged as "awaiting followup", are you actually still waiting on me for something (if so, what?) or is the tag just stale?
Hi @drammock I think no but maybe you could try testing with Spyder 4.1.5 now, please?
Hi @drammock I think no but maybe you could try testing with Spyder 4.1.5 now, please?
That is the version that I used in the screencaptured video I linked to above.
Oh my bad, I updated the labels to prevent confusions in the future
FYI, here's another user that can't get this to work, and they're not using a VM like I've been: https://github.com/mne-tools/mne-python/pull/8020#issuecomment-708653453
What's your macOS version? I ran my tests in 10.14
macOS 10.15.5 ("Catalina"), both for me and for @larsoner. This was mentioned in the problem description, and as was also mentioned in mne-tools/mne-python#8009, which was the genesis of this issue.
Hi @drammock. I followed the steps in your video and I'm not getting any error messages. I'm using MacOS Catalina 10.15. Ideas @ccordoba12 ?
Okay I think it's a version problem:
$ conda create -n test-spyder python=3.7 spyder
...
$ conda create -n test-py38 python=3.8 spyder-kernels
...
$ conda activate test-spyder
$ python
>>> from spyder.utils import programs
>>> programs.is_module_installed('spyder_kernels', interpreter='/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python3.8')
True
>>> programs.is_module_installed('spyder_kernels', interpreter='/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python3.8', version='>=1.9.4;<1.10.0)
False
>>> exit()
$ conda activate test-py38
$ python
>>> import spyder_kernels
>>> spyder_kernels.__version__
'1.10.0'
>>> exit()
$ conda install "spyder_kernels<1.10"
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- spyder_kernels[version='<1.10']
In other words, there is no version < 1.10 for Python 3.8.
Can the conditional be changed to <=1.10
, or is 1.10 still broken / not working?
FWIW I'm on Big Sur now, but I was on Catalina I think back when I was having this problem originally. For some reason the osx-64/spyder-kernels-1.9.4-py38h32f6830_0.tar.bz2
file must not work. Any ideas how I can figure out why conda install "spyder_kernels<1.10"
is not happy with this version?
My bad, bad line -- this works to install it:
$ conda install "spyder-kernels<1.10"
...
$ conda activate test-spyder
$ python
>>> from spyder.utils import programs
>>> programs.is_module_installed('spyder_kernels', interpreter='/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python3.8', version='>=1.9.4;<1.10.0)
True
>>> exit()
$ spyder
... # interpreter still doesn't work
I'll keep digging into why it doesn't work...
Okay I managed to not delete the temporary script that gets created, and also eval
just the last line of the output, and this is what we see when running it directly:
$ conda activate test-py38
$ python /var/folders/79/gcdk9jbj4txgtlp9l9nmmhyw0000gn/T/spyder-larsoner/tmp9a2luv8t.py
spyder_kernels: 1.9.4
spyder_kernels: 1.10.0
True
So far so good. And so is this:
$ conda activate test-spyder
$ python -c "from spyder.utils import programs; print(programs.run_program('/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python3.8', ['/var/folders/79/gcdk9jbj4txgtlp9l9nmmhyw0000gn/T/spyder-larsoner/tmppx296biv.py']).communicate()[0].decode())"
/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python3.8 /var/folders/79/gcdk9jbj4txgtlp9l9nmmhyw0000gn/T/spyder-larsoner/tmppx296biv.py
True
But spyder
doesn't use python
, it uses pythonw
-- and here's the rub:
pythonw -c "from spyder.utils import programs; print(programs.run_program('/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python3.8', ['/var/folders/79/gcdk9jbj4txgtlp9l9nmmhyw0000gn/T/spyder-larsoner/tmppx296biv.py']).communicate()[0].decode())"
/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python3.8 /var/folders/79/gcdk9jbj4txgtlp9l9nmmhyw0000gn/T/spyder-larsoner/tmppx296biv.py
False
$ which python
/Users/larsoner/opt/miniconda3/envs/test-spyder/bin/python
$ which pythonw
/Users/larsoner/opt/miniconda3/envs/test-spyder/bin/pythonw
So in other words, something about how pythonw
handles things versus of python
seems to be the culprit.
To confirm this, if I change the shebang of /Users/larsoner/opt/miniconda3/envs/test-spyder/bin/spyder
from the original:
#!/bin/bash /Users/larsoner/opt/miniconda3/envs/test-spyder/bin/pythonw
to
#!/Users/larsoner/opt/miniconda3/envs/test-spyder/bin/python
things work fine.
Based on this stackoverflow issue, https://github.com/ContinuumIO/anaconda-issues/issues/199 might be related, and I'm not sure what the correct fix for this is, if there is one.
Hey @larsoner, thanks a lot for digging deeper into this one. We simplified the way we detect modules in external interpreters In our 4.2.0 version (not yet part of Anaconda but present in conda-forge). That could help in this case (not sure if it does).
Could you try it from conda-forge and let us know if it works now?
Just doing:
$ conda create -n test-spyder-forge -c conda-forge python=3.7 spyder
$ conda activate test-spyder-forge
$ spyder
Didn't work at first because now it wants ">= 1.10" -- so doing:
$ conda activate test-py38
$ conda install "spyder-kernels>=1.10"
$ conda deactivate
$ spyder
Gives:
An error ocurred while starting the kernel
/Users/larsoner/opt/miniconda3/envs/test‑spyder‑forge/bin/python:
Error while finding module specification for 'spyder_kernels.console'
(ModuleNotFoundError: No module named 'spyder_kernels')
This looks like a different problem at least. Should I do another round of debugging for this error?
Please remove and recreate your test-py38
env with the following commands:
conda env remove -n test-py38
conda create -n test-py38 -c conda-forge python=3.8
conda activate test-py38
conda install -c conda-forge spyder-kernels=1.10
Got a spinning "Connecting to kernel ..." then same as before:
An error ocurred while starting the kernel
/Users/larsoner/opt/miniconda3/envs/test‑spyder‑forge/bin/python:
Error while finding module specification for 'spyder_kernels.console'
(ModuleNotFoundError: No module named 'spyder_kernels')
And just to be sure:
$ conda activate test-py38
$ python -c "import spyder_kernels; print(spyder_kernels.__version__)"
1.10.0
$ conda list | grep spyder
spyder-kernels 1.10.0 py38h50d1736_0 conda-forge
Got a spinning "Connecting to kernel ..." then same as before:
Could you post a screenshot that shows this error in our interface?
That's because I don't understand why this line appears on it:
/Users/larsoner/opt/miniconda3/envs/test‑spyder‑forge/bin/python
As you can see, this is not the interpreter of your test-py38
but the one used by Spyder.
Should I do another round of debugging for this error?
Yes, please do. We'd really appreciate your help with that.
Could you post a screenshot that shows this error in our interface?
That's because I don't understand why this line appears on it:
Here are the relevant parts, with the terminal call, error message, and preferences pane open to what it's supposed to use:
This is probably because, as mentioned above, there is very strange behavior with subprocess
when doing python
versus pythonw
:
(test-spyder-forge) bunker:~ larsoner$ which python
/Users/larsoner/opt/miniconda3/envs/test-spyder-forge/bin/python
(test-spyder-forge) bunker:~ larsoner$ which pythonw
/Users/larsoner/opt/miniconda3/envs/test-spyder-forge/bin/pythonw
(test-spyder-forge) bunker:~ larsoner$ python -c "import subprocess; print(subprocess.check_output(['/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python', '-c', 'import sys; print(sys.executable)']).decode())"
/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python
(test-spyder-forge) bunker:~ larsoner$ pythonw -c "import subprocess; print(subprocess.check_output(['/Users/larsoner/opt/miniconda3/envs/test-py38/bin/python', '-c', 'import sys; print(sys.executable)']).decode())"
/Users/larsoner/opt/miniconda3/envs/test-spyder-forge/bin/python
So if the newer code still uses subprocess
to check things, it's probably still going to have problems. Or even if it doesn't, whatever code path it is using might have these problems...
Again / still, if I change the shebang in /Users/larsoner/opt/miniconda3/envs/test-spyder-forge/bin/spyder
from the original pythonw
call:
#!/bin/bash /Users/larsoner/opt/miniconda3/envs/test-spyder-forge/bin/pythonw
to a python
call:
#!/Users/larsoner/opt/miniconda3/envs/test-spyder-forge/bin/python
Things work fine:
Based on this stackoverflow issue, ContinuumIO/anaconda-issues#199 might be related, and I'm not sure what the correct fix for this is, if there is one.
I missed this part (I just found the same issue). I think the solution is to avoid using pythonw
to start Spyder on Mac (but I think that comes with other unpleasant effects).
Latest PyQt runs happily without pythonw
on macOS – so I think there's not too many use cases for pythonw
out there anymore anyway…
Running into this same issue. OSX 10.13.6, Spyder 4.1.4 via Anaconda 1.9.12 freshly downloaded and installed yesterday. I was hoping installing spyder kernels to my python 3.6.9 OpenCV environment would satisfy Spyder's dependencies, but no luck.
Upgrading Spyder to 4.2.0 in base, then switched to the OpenCV environment, which inexplicably installed Spyder 3.2.8, which at lease works with this environment. Hope this is useful to someone.
I have an idea to fix this, but it's too late for 4.2.1 (to be released next Friday). I'll try to implement it for 4.2.2
Most helpful comment
@juanis2112, thanks for helping @drammock with his problem. You did a great job guiding him to check that this bug was effectively fixed in our 4.1.4 version. Sorry for stepping in without reading the whole context of this issue.
Then you'll be happy to hear that we're working on our own installers for Windows and macOS for 4.2.0 (to be released in September).