Spyder: RecursionError when opening a dataframe with Pandas 1.1.0 in the kernel and 1.0.5 in Spyder

[laurajanem]

Description

What steps will reproduce the problem?

import pandas as pd
df = pd.DataFrame({"a":range(10),"b":range(10)})

Try to open df by double clicking on the DataFrame in the variable explorer.

This was previously working fine. I believe that a recent install of spyder-kernels to my virtual environment caused this problem, but I'm not sure how.

To avoid having to install spyder in every virtual environment I works with, I just install spyder-kernels in the venv, launch spyder from my base environment, and point the python interpreter to the python instance inside my venv. This process has been working fine, until I just had to a create a new venv for a project. I repeated this process and am now getting this error.
This only happens when the python interpreter is pointing to my venv interpreter, not base spyder.

Quick googling seems like this may be related to incompatible versions of ipykernel and spyder-kernels (?)
https://github.com/python-poetry/poetry/issues/2133

-version of spyder-kernels = 1.9.3
-version of ipykernel: 5.3.4, which was installed with spyder-kernels

Traceback

  File "/Users/lmariano/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pandas/core/generic.py", line 5272, in __getattr__
    if self._info_axis._can_hold_identifiers_and_holds_name(name):
  File "/Users/lmariano/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pandas/core/generic.py", line 5272, in __getattr__
    if self._info_axis._can_hold_identifiers_and_holds_name(name):
  File "/Users/lmariano/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pandas/core/generic.py", line 5272, in __getattr__
    if self._info_axis._can_hold_identifiers_and_holds_name(name):
  [Previous line repeated 993 more times]
  File "/Users/lmariano/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pandas/core/generic.py", line 493, in _info_axis
    return getattr(self, self._info_axis_name)
  File "/Users/lmariano/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pandas/core/generic.py", line 5270, in __getattr__
    return object.__getattribute__(self, name)
  File "pandas/_libs/properties.pyx", line 63, in pandas._libs.properties.AxisProperty.__get__
  File "/Users/lmariano/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pandas/core/generic.py", line 5270, in __getattr__
    return object.__getattribute__(self, name)
RecursionError: maximum recursion depth exceeded while calling a Python object

Versions

• Spyder version: 4.1.4
• Python version: 3.7.7
• Qt version: 5.12.9
• PyQt5 version: 5.12.3
• Operating System: Darwin 19.6.0

Dependencies

# Mandatory:
applaunchservices >=0.1.7      :  0.2.1 (OK)
atomicwrites >=1.2.0           :  1.4.0 (OK)
chardet >=2.0.0                :  3.0.4 (OK)
cloudpickle >=0.5.0            :  1.5.0 (OK)
diff_match_patch >=20181111    :  20200713 (OK)
intervaltree                   :  None (OK)
IPython >=4.0                  :  7.16.1 (OK)
jedi =0.17.1                   :  0.17.1 (OK)
nbconvert >=4.0                :  5.6.1 (OK)
numpydoc >=0.6.0               :  1.1.0 (OK)
parso =0.7.0                   :  0.7.0 (OK)
pexpect >=4.4.0                :  4.8.0 (OK)
pickleshare >=0.4              :  0.7.5 (OK)
psutil >=5.3                   :  5.7.2 (OK)
pygments >=2.0                 :  2.6.1 (OK)
pylint >=1.0                   :  2.5.3 (OK)
pyls >=0.34.0;<1.0.0           :  0.34.1 (OK)
qdarkstyle >=2.8               :  2.8.1 (OK)
qtawesome >=0.5.7              :  0.7.2 (OK)
qtconsole >=4.6.0              :  4.7.5 (OK)
qtpy >=1.5.0                   :  1.9.0 (OK)
sphinx >=0.6.6                 :  3.1.2 (OK)
spyder_kernels >=1.9.2;<1.10.0 :  1.9.2 (OK)
watchdog                       :  None (OK)
zmq >=17                       :  19.0.1 (OK)

# Optional:
cython >=0.21                  :  None (OK)
matplotlib >=2.0.0             :  None (OK)
numpy >=1.7                    :  1.19.0 (OK)
pandas >=0.13.1                :  1.0.5 (OK)
scipy >=0.17.0                 :  1.5.1 (OK)
sympy >=0.7.3                  :  None (OK)

[laurajanem]

I believe that I identified the source of the problem.

I think that the latest version of spyder-kernels is not compatible with the latest version of pandas. The latest version of pandas is 1.1.1, but I also encounter this error using 1.1.0 - The most up-to-date version of pandas that does not cause this problem is 1.0.5.

-I'm amending this statement - the issue is an incompatibility between the pandas versions of spyder and spyder-kernels. When I upgraded the version of pandas in the base environment of the spyder install to 1.1.1, everything worked as it should.

[steff456]

Hi @laurajanem,

So you could fix your problem by upgrading pandas to 1.1.1? I'm not encountering this error in 1.0.5

[laurajanem]

Yes, but the issue is that both the .venv I'm working in, where I have
installed spyder-kernels and the base spyder environment have to have
matching pandas versions, or at least compatible versions. I did a fresh
install of spyder today, just to see if there was something wrong there,
and it auto-installs with pandas 1.0.5.
It's definitely a hazard of trying to get around having to do the full
Spyder install in every single venv I am working on - by only installing
spyder-kernels in the .venv and opening Spyder in a different
environment, I know I'm asking for trouble, but this stumped me for awhile.
I'd love for there to be a way to not have to do this at all.

On Mon, Aug 24, 2020 at 4:13 PM Stephannie Jimenez Gacha <
[email protected]> wrote:

Hi @laurajanem https://github.com/laurajanem,

So you could fix your problem by upgrading pandas to 1.1.1? I'm not
encountering this error in 1.0.5

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/spyder-ide/spyder/issues/13620#issuecomment-679341651,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEFQKPLDGYR4FEU2X45Y5W3SCLCV3ANCNFSM4QJWOKHA
.

[steff456]

We are currently working on Spyder installers, to avoid this type of errors. As your error is already fixed I will go ahead and close this issue.

Thanks for reporting!

[ccordoba12]

@laurajanem, thanks for reporting. I think this is a serious issue.

@steff456, even with the installers, people can create a new environment with an unmatching Pandas version and connect Spyder to it. So I think we need to look further at this problem.

Were you able to reproduce it following @laurajanem instructions?

[jafetgado]

Downgrading spyder-kernels to 1.9.1 and pandas to 1.0.5 solved it for me.

[ccordoba12]

@jafetgado, thanks for the feedback. However, I think this is not an issue with spyder-kernels but it's caused a mismatch between the Pandas version used in Spyder and the one used by the kernel.

[steff456]

@ccordoba12 I was able to reproduce this error by creating a new environment with the latest version of pandas and connecting this one as python interpreter. My base environment has pandas 1.0.5 and the new environment has pandas 1.1.1

[ccordoba12]

@steff456, please use cloudpickle to serialize a dataframe with Pandas 1.1.1 and try to deserialize it with Pandas 1.0.5 to see if you get the same error. Examples of how to do that can be found here

https://github.com/cloudpipe/cloudpickle#examples

That's basically what we do when asking the kernel to send us a value to display it in our dataframe viewer.

If you're able to do that, we could report the problem to the Pandas team in a way that doesn't depend on Spyder, to see if they can fix it.

[steff456]

@ccordoba12 if I have my spyder environment with pandas 1.1.1 and connect it to a kernel with pandas 1.0.5 it works as expected and the error doesn't exist. I think we need to bump our version of pandas for avoiding this error and we can report this problem to pandas so they can be aware of this behavior.

[ccordoba12]

I agree, thanks for your help @steff456!