Spyder: Variables disappear from Spyder 4.0 Variable Explorer

[eafpres]

Problem Description

Variables intermittently disappear from Variable Explorer
When running various code, a variable that was visible in the Variable Explorer disappears
_Sometimes_, scrolling away from the area of list where the variable _should_ be, then clicking the column heading once or twice, and scrolling back makes the variable appear

In this image the variable experiments, a Pandas dataframe, is missing:

Here, after scrolling and sorting as noted, experiments now appears:

What steps reproduce the problem?

1. In some cases, running a code line that replaces the variable with a new value causes it to disappear, if it is not being displayed at present (i.e. if scrolled away).
   e.g.
   plot_name = study_name + ts.timestamp() + '.png' might cause plot_name to disappear
2. In other cases, if the code is run as a script, then Variable Explorer is checked, the variable may be missing

Note: Variable Explorer Options were tested. Following is the current configuration, which did not resolve the issue:

What is the expected output? What do you see instead?

Expected outcome is that the variable appears in Variable Explorer as soon as it is created, and remains there until it is removed from the environment.

Paste Traceback/Error Below (if applicable)

PASTE TRACEBACK HERE

Versions

Spyder 4.0.0b5
Python 3.6.9 64-bit
Qt 5.13.1
PyQt5 5.13.1
Windows 10

Dependencies

cloudpickle >=0.5.0 : 1.2.2 (OK)
pygments >=2.0 : 2.4.2 (OK)
qtconsole >=4.5.5 : 4.5.5 (OK)
nbconvert >=4.0 : 5.6.0 (OK)
sphinx >=0.6.6 : 2.2.0 (OK)
pylint >=0.25 : 2.3.1 (OK)
psutil >=0.3 : 5.6.3 (OK)
qtawesome >=0.5.7 : 0.6.0 (OK)
qtpy >=1.5.0 : 1.9.0 (OK)
pickleshare >=0.4 : 0.7.5 (OK)
zmq >=17 : 18.1.0 (OK)
chardet >=2.0.0 : 3.0.4 (OK)
numpydoc >=0.6.0 : 0.9.1 (OK)
spyder_kernels >=1.5.0;<2.0.0: 1.5.0 (OK)
qdarkstyle >=2.7 : 2.7 (OK)
atomicwrites >=1.2.0 : 1.3.0 (OK)
diff_match_patch >=20181111 : 20181111 (OK)
watchdog : None (OK)
keyring : None (OK)
pexpect >=4.4.0 : 4.7.0 (OK)
pympler : None (OK)
sympy >=0.7.3 : None (NOK)
cython >=0.21 : None (NOK)
IPython >=4.0 : 7.8.0 (OK)
matplotlib >=2.0.0 : 3.1.2 (OK)
pandas >=0.13.1 : 0.25.2 (OK)
numpy >=1.7 : 1.16.5 (OK)
scipy >=0.17.0 : 1.3.1 (OK)
pyls >=0.28.2;<0.29.0 : 0.28.3 (OK)
rtree >=0.8.3 : 0.8.3 (OK)

PASTE DEPENDENCIES HERE

[ccordoba12]

We need a simple and reproducible use case that generates this problem. Else it's not possible for us to track down and solve this issue, sorry.

[ccordoba12]

@impact27, any ideas about the possible source of this problem?

[eafpres]

We need a simple and reproducible use case that generates this problem. Else it's not possible for us to track down and solve this issue, sorry.

I will look for one. It's very tricky; it happens a lot but not in obvious ways. I understand that you need a reproducible example. I wanted to establish this is occurring in case anyone else might be seeing it and have an idea. If I can definitively repro you can be assured I will report. FYI, there is another, possibly related behavior also hard to repro: Using the right-click, remove option often removes something else. I suspect it is actually a display issue, not that another var is being removed. Perhaps that provides a clue? Like, the var below the one clicked to remove disappears from the Variable Explorer, but actually the intended one has been removed. Just FYI until I can repro it!

[impact27]

@impact27, any ideas about the possible source of this problem?

This might be caused by https://github.com/spyder-ide/spyder/pull/10567: it added interrupt=True to self.call_kernel(interrupt=True, callback=self.set_namespace_view).get_namespace_view(). Maybe the variable is briefly removed from the namespace and the refresh happened at that precise moment? As interrupt=True happens in a thread it is possible that there is a race condition. If this is indeed the reason the solution would be to interrupt=True only when refreshing the namespace.

[impact27]

My theory doesn't work: Spyder 4.0.0b5 is before #10567 was merged.

[ccordoba12]

@eafpres, beta5 Is very old. Please update to rc1 and try again.

[eafpres]

@ccordoba12 I will update and then add more information here.

[ccordoba12]

Great, thanks!

[Khalilsqu]

I can reproduce somehow a similar bug.

Here is an example.

Suppose there are 4 variables named,

a = 9,
b = 3,
c = 4,
d = 'k'.

If you right-click and delete b the first time, the variable b will be removed correctly, however, if you delete b and re-run the script. The next time you try to delete b, variable c will be deleted.

In the second delete of b after clicking run, self.source_model.keys[idx_row] give value ['c'] while self.selectedIndexes() gives 1 (function)

If you click delete from the context menu the second time and run the script again to produce all four variables, the third delete will remove the variable d.

The test is made to this commit.

I hope it is understood.

[ccordoba12]

Thanks @Khalilsqu for the reproducible example! However, that seems more related to issue #10709 than this one.

[eafpres]

After updating to rc1 this appears fixed. Thank you!

[ccordoba12]

Thanks a lot for the great news!!

[eafpres]

Unfortunately, today I again see the behavior. I will continue working on reproducible example. The code I'm running where this occurs often is fairly big and uses a lot of stuff, but I don't think it is a dependency.

Question, I have this issue:

(base) C:\Users\bbate>activate keras-gpu

(keras-gpu) C:\Users\bbate>spyder
Attribute Qt::AA_UseSoftwareOpenGL must be set before QCoreApplication is created.
[7804:12336:1118/114136.143:ERROR:permission_manager_qt.cpp(82)] NOT IMPLEMENTEDUnsupported permission type: 13

And I've been unable to resolve it. It is possible this is causing issues with the GUI?

[ccordoba12]

It is possible this is causing issues with the GUI?

I don't think so.

[eafpres]

I do not have a reproducible example yet. I can reproduce it in a project I'm working but that has lots of code, data, etc. Every time I try something smaller, I cannot reproduce.

I wanted to pass along an observation. When a variable disappears from the Variable Explorer, sorting by name, scrolling all the way to the end, sorting the opposite way (i.e. clicking the name column heading again) nearly always restores the variable to the list. I'm hoping that helps someone on the dev side to narrow down where this is occurring.

[ccordoba12]

@eafpres, if it's possible for you to share your code privately with us, that'd be great to debug this problem. We can promise you total confidentiality about it.

[eafpres]

I have a reproducible example!

Problem Description

Variables intermittently disappear from Variable Explorer
Running the attached code from a clean start of Spyder 4 populates Variable Explorer with a range of variables
If you scroll down _carefully_ you will see that the 'e' variables are initially missing (important: you need to stop scrolling 1-2 lines past the 'd' vars, or the display will change!):

Continuing to scroll down _carefully_ you will see Variable Explorer 'jump' and then the 'e' variables are visible, but the last variable created, 'experiments_list' is still not present (important: as above, stop scrolling just past the 'e' vars!):

Finally, continuing to scroll all the way down, all the variables appear:

What steps reproduce the problem?

• save the file 'experiments.csv' to the working directory
  experiments.zip

• open 'spyder_issue_var_explorer_repro.py' in Spyder 4 IDE
  spyder_repro_var_explorer_issue.zip

• edit the chdir() statement to reflect the correct directory

• select all code, use 'run selection or current line to run it'
• slowly scroll down in the Variable Explorer just past the 'd' vars
• note that the list goes from 'd' to 'i' without any 'e' vars
• continue scrolling slowly and observe Variable Explorer pane 'jump'
• the 'e' vars are now shown, but 'experiments_list' is missing
• continue scrolling, observe another 'jump', all vars now shown
• clearing all vars and repeating the run will reproduce every time on my machine

[ccordoba12]

@impact27, could you take a look at the reproducible example provided by @eafpres above?

[impact27]

First, os.system('cls') doesn't work on mac, so I got rid of that.

I can not reproduce it, but I see a similar behaviour:
It looks like not all vars are loaded until the screen is scrolled down. Therefore, when scrolling down, I see several jumps.

For example, see these consecutive frames:

[eafpres]

Could you clarify if this is planned to be fixed; I'm sorry I'm unclear on your use of the labels for milestones etc.

[ccordoba12]

Yes, it's planned for a future release. The thing is I have some concerns about @impact27's proposed solution, but I haven't had time to test them, sorry.

[eafpres]

Yes, it's planned for a future release. The thing is I have some concerns about @impact27's proposed solution, but I haven't had time to test them, sorry.

Understand. My workaround is scrolling down and back and that almost always works. If I ever find anything that might shed more light on this I'll definitely share.

[HZatl]

What is your DataFrame title? Is it uppercase to start?

I was having an issue where not all my data was showing in variables explorer. It appeared to be just DataFrames not showing.

Clicking on the upper right corner there is a drop down menu. I un-clicked "Exclude Capitalized Reference" and my dataframes starting with capital letters started to show.

This seems too simple to be the solution to your problem but thought I'd throw in my half a cent.

Maybe click or unclick the option and refresh

[eafpres]

What is your DataFrame title? Is it uppercase to start?

I was having an issue where not all my data was showing in variables explorer. It appeared to be just DataFrames not showing.

Clicking on the upper right corner there is a drop down menu. I un-clicked "Exclude Capitalized Reference" and my dataframes starting with capital letters started to show.

This seems too simple to be the solution to your problem but thought I'd throw in my half a cent.

Maybe click or unclick the option and refresh

This actually was fully fixed in recent releases. It was an actual bug and the excellent team solved it.