Spyder: Improve error message when trying to view a dataframe and Pandas is not installed next to Spyder

[waageesh]

Issue Report Checklist

• [ yes] Searched the issues page for similar reports
• [ yes] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• [ ] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• [ ] Could not reproduce inside jupyter qtconsole (if console-related)
• [ ] Tried basic troubleshooting (if a bug/error)
  
  
  
  • [ ] Restarted Spyder
  
  
  • [ ] Reset preferences with spyder --reset
  
  
  • [ ] Reinstalled the latest version of Anaconda
  
  
  • [ ] Tried the other applicable steps from the Troubleshooting Guide
  
  
• [ ] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

After opening spyder in virtualenv,
checked pip freeze and lists pandas , numpy , sklearn everything needed for data analysis
the i wrote a code and ran it.
it worked well., but when i am trying to open same variable in variable explorer window, it showing error

What steps reproduce the problem?

1. install spyder using pip
2. open spyder in virtualenv
3. ran the code and check in variable explorer

What is the expected output? What do you see instead?

IPython console is working fine and giving perferct results,
only variable explorer is not able to open variables that are coming form using pandas library
i tired defining numpy array and list , i could able to open them as below

but when i am trying open pandas dataframe variable , variable explorer is throwing below error

Paste Traceback/Error Below (if applicable)

PASTE TRACEBACK HERE

Versions

• Spyder version: 4.1.2
• Python version: 3.8
• Qt version:
• PyQt version:
• Operating System name/version: Windows 10

Dependencies

# Mandatory:
atomicwrites >=1.2.0           :  1.4.0 (OK)
chardet >=2.0.0                :  3.0.4 (OK)
cloudpickle >=0.5.0            :  1.4.0 (OK)
diff_match_patch >=20181111    :  20181111 (OK)
intervaltree                   :  None (OK)
IPython >=4.0                  :  7.13.0 (OK)
jedi =0.15.2                   :  0.15.2 (OK)
nbconvert >=4.0                :  5.6.1 (OK)
numpydoc >=0.6.0               :  0.9.2 (OK)
paramiko >=2.4.0               :  2.7.1 (OK)
parso =0.5.2                   :  0.5.2 (OK)
pexpect >=4.4.0                :  4.8.0 (OK)
pickleshare >=0.4              :  0.7.5 (OK)
psutil >=5.3                   :  5.7.0 (OK)
pygments >=2.0                 :  2.6.1 (OK)
pylint >=0.25                  :  2.5.0 (OK)
pyls >=0.31.9;<0.32.0          :  0.31.10 (OK)
qdarkstyle >=2.8               :  2.8.1 (OK)
qtawesome >=0.5.7              :  0.7.1 (OK)
qtconsole >=4.6.0              :  4.7.3 (OK)
qtpy >=1.5.0                   :  1.9.0 (OK)
sphinx >=0.6.6                 :  3.0.3 (OK)
spyder_kernels >=1.9.0;<1.10.0 :  1.9.0 (OK)
watchdog                       :  None (OK)
zmq >=17                       :  19.0.0 (OK)

# Optional:
cython >=0.21                  :  None (OK)
matplotlib >=2.0.0             :  None (OK)
numpy >=1.7                    :  1.18.2 (OK)
pandas >=0.13.1                :  None (OK)
scipy >=0.17.0                 :  None (OK)
sympy >=0.7.3                  :  None (OK)

[mrclary]

I can confirm that this occurs for bootstrap startup in a clean conda environment using the latest 4.x commit.

It seems to occur only if the internal Spyder python environment does not have the package installed, even when using an external interpreter that does have the package installed. In other words, I think that the variable explorer is accessing the wrong kernel.

[goanpeca]

Thanks for the report @waageesh. Even if the external kernel has pandas installed, I think internally we still need to have pandas install to process results.

You can see that the optional pandas dependency is NOT installed.

I think if you install pandas on your Spyder environment, it should work jut fine. Let us know.

---

I can confirm that this occurs for bootstrap startup in a clean conda environment using the latest 4.x commit.

Thanks for checking @mrclary, see above comment :-)

[mrclary]

@goanpeca, do you mean that this is _not_ a bug? It seems to me that the Variable Explorer ought to access the iPython Console's kernel, not Spyder's kernel (unless it is the same as the Console's).

[goanpeca]

do you mean that this is not a bug?

Yes, exactly, since Spyder still requires pandas on the Spyder side (data frame editor) to be able to view a pandas dataframe.

https://github.com/spyder-ide/spyder/blob/4.x/spyder/plugins/variableexplorer/widgets/dataframeeditor.py#L47

[mrclary]

Ahh, I see. Clicking on an item in the Variable Explorer is supposed to activate an object _editor_. I was thinking it just gave one an expanded view of the object.

We should probably highlight this in user documentation somewhere, particularly for those that use a minimal environment from which to launch Spyder but use a separate environment for their Consoles. Note, this also applies to numpy arrays.

[goanpeca]

We should probably highlight this in user documentation somewhere, particularly for those that use a minimal environment from which to launch Spyder but use a separate environment for their Consoles. Note, this also applies to numpy arrays.

Indeed, we might need better messaging or reconsider moving numpy and pandas from optional to mandatory dependencies?

Thoughts @spyder-ide/core-developers ?

[bcolsen]

I think the error message should explain the situation. Something like optional dependency <package> not found in spyder environment

I would also be ok with them being mandatory.

[goanpeca]

Yes, indeed!

[ccordoba12]

@dalthviz, please explain @juanis2112 what's necessary to solve this issue.

[maximlt]

I encountered the same bug and it took me a while before I understood what went wrong and to find this issue eventually. I use the Variable Explorer mostly to explore DataFrames so I would expect this feature to work out-of-the-box, thus I'm +1 on having Pandas (hence Numpy) as a hard dependency here.

Note that even if pandas isn't installed the DataFrame shows up in the Variable Explorer, it's even able to parse it somehow without pandas as the column names are displayed.

Are there other types of object that could lead to the same error besides the Numpy and Pandas ones?

I'll update the wiki (see #12107) according to your decision here.

[goanpeca]

@maximlt indeed, the issues is installing pandas means also numpy and scipy which are big libraries, but given the need is the road we will probably need to take.

[mrclary]

It sure would be nice, someday, if its even possible, to provide a way for Spyder to edit Variable Explorer objects without the need for those packages to be installed in Spyder's runtime environment. In other words, use the iPython Console's kernel. That would solve these issues and reduce the "required" footprint of Spyder, especially for a standalone application.

[goanpeca]

It sure would be nice, someday, if its even possible, to provide a way for Spyder to edit Variable Explorer objects without the need for those packages to be installed in Spyder's runtime environment. In other words, use the iPython Console's kernel. That would solve these issues and reduce the "required" footprint of Spyder, especially for a standalone application.

Yeah we actually need to do that to be able to support other objects in other languages in other Kernels, so something we definitely need to address.

[ccordoba12]

In other words, use the iPython Console's kernel

Then Qt would need to be installed in the kernel, which is way bigger and harder to install correctly than Pandas. Besides, that workflow won't work for remote kernels. So I don't see much future in that idea.

[goanpeca]

Besides, that workflow won't work for remote kernels. So I don't see much future in that idea.

I guess what @mrclary means is that we could serialize some information of the object in the kernel and pass that in some format to the frontend, and then we could display that. This would be simple for small objects but not very simple for larger ones.

[ccordoba12]

Oh yeah, that's in our roadmap for Spyder 5 (sorry for the misunderstanding).