Spyder: Variable explorer slow to open in spyder 4 with large dataframes

[greg500]

Problem Description

Moderately large pandas data frames load very slowly in spyder 4 variable explorer.

What steps reproduce the problem?

import numpy as np
import pandas as pd

df=pd.DataFrame(np.arange(400000).reshape(40000,10))

What is the expected output? What do you see instead?

The above dataframe (df) takes more than 30 seconds to load in spyder 4 variable explorer
while it is instantaneous in spyder 3.

Versions

• Spyder version: 4.0.0
• Python version: 3.7.4
• Qt version: 5.9.7
• PyQt version: 5.9.2
• Operating System name/version: windows 10 x64 1903

Dependencies

cloudpickle >=0.5.0          :  1.2.2 (OK)
pygments >=2.0               :  2.5.2 (OK)
qtconsole >=4.6.0            :  4.6.0 (OK)
nbconvert >=4.0              :  5.6.1 (OK)
sphinx >=0.6.6               :  2.2.2 (OK)
pylint >=0.25                :  2.4.4 (OK)
psutil >=0.3                 :  5.6.7 (OK)
qtawesome >=0.5.7            :  0.6.0 (OK)
qtpy >=1.5.0                 :  1.9.0 (OK)
pickleshare >=0.4            :  0.7.5 (OK)
zmq >=17                     :  18.1.0 (OK)
chardet >=2.0.0              :  3.0.4 (OK)
numpydoc >=0.6.0             :  0.9.1 (OK)
spyder_kernels >=1.8.1;<2.0.0:  1.8.1 (OK)
qdarkstyle >=2.7             :  2.7 (OK)
atomicwrites >=1.2.0         :  1.3.0 (OK)
diff_match_patch >=20181111  :  20181111 (OK)
intervaltree                 :  None (OK)
watchdog                     :  None (OK)
keyring                      :  None (OK)
pexpect >=4.4.0              :  4.7.0 (OK)
pympler                      :  None (OK)
sympy >=0.7.3                :  1.4 (OK)
cython >=0.21                :  0.29.14 (OK)
IPython >=4.0                :  7.10.1 (OK)
matplotlib >=2.0.0           :  3.1.1 (OK)
pandas >=0.13.1              :  0.25.3 (OK)
numpy >=1.7                  :  1.17.4 (OK)
scipy >=0.17.0               :  1.3.2 (OK)
pyls >=0.31.2;<0.32.0        :  0.31.2 (OK)
rtree >=0.8.3                :  0.8.3 (OK)

[bcolsen]

I can confirm. ~30 seconds here as well on linux latest 4.x

[goanpeca]

Thoughts @impact27 ?

[impact27]

looks like DataFrameEditor's _sizeHintForColumn is taking a lot of time. Not sure how to optimise that though.

[ccordoba12]

@dalthviz, please take a look at this one.

[yarmun]

I second this. my data loads instantly on v3, but v4 it's taking around 20-30secs. I like rest of V4 especially Kite's integration. but this is slow wait is driving me toward downgrading.