Spyder: Variable Explorer is leaking memory when editing variables

[databoi]

Description of your problem

I am loading some .txt files into Spyder as arrays with numpy. The files load fine and I can enter editing mode in the variable explorer once the files have loaded, and there are no issues. However, upon exiting editing mode, it only takes a couple of seconds before one of the python processes already running (visible in task manager) starts to take up an increasing amount of memory (until all available memory is used). This does only happen for files of a large size. I am not sure where the threshold is, but the stylized reproduction example below should provide some indication (at least in my case).

I have tried to search for a solution both on GitHub and stackoverflow, but without any luck. However, should I have missed anything please point me in the direction of the solution. Also, I am very new to both Python and Spyder (3 days), so please forgive me if this is a stupid question.

What steps will reproduce the problem?

1. Create two .txt files that are each are a vector of zeroes separated by spaces, but with different number of entries (78396 and 1157770).
2. Load two .txt files with as arrays with numpy using: txtfile = np.loadtxt('file.txt',delimiter = ' ')
3. Enter editing mode for the smaller array, then exit - nothing interesting should happen.
4. Enter editing mode for the larger array, then exit - python process should start to eat up memory

Versions and main components

• Spyder Version: 3.2.4
• Python Version: 3.6.3
• Qt Version: 5.6.2
• PyQt Version: 5.6.0
• Operating system: Microsoft Windows 10 Pro

Dependencies

IPython >=4.0 : 6.1.0 (OK)
cython >=0.21 : 0.26.1 (OK)
jedi >=0.9.0 : 0.10.2 (OK)
nbconvert >=4.0 : 5.3.1 (OK)
numpy >=1.7 : 1.13.3 (OK)
pandas >=0.13.1 : 0.21.0 (OK)
pycodestyle >=2.3: 2.3.1 (OK)
pyflakes >=0.6.0 : 1.6.0 (OK)
pygments >=2.0 : 2.2.0 (OK)
pylint >=0.25 : 1.7.4 (OK)
qtconsole >=4.2.0: 4.3.1 (OK)
rope >=0.9.4 : 0.10.5 (OK)
sphinx >=0.6.6 : 1.6.3 (OK)
sympy >=0.7.3 : 1.1.1 (OK)

[CAM-Gerlach]

Thanks for reporting. I was able to replicate this behavior on my Win 8.1 Pro x64 machine with the same Spyder, Python, Qt, and Numpy versions as you, but with (perhaps in part thanks to Process Explorer and 24 GB of RAM) more nuance and precision with regard to what's going on. For replicability, here was the code snippet I whipped up the write and read the files:

import numpy as np

flengths = [78396, 1157770]
for flength in flengths:
    with open(str(flength) + ".txt", "w", encoding="utf-8", newline="\n") as f:
        for i in range(flength):
            f.write("0 ")
        f.write("0\n")
short_array = np.loadtxt(str(flengths[0]) + ".txt", delimiter=" ")
long_array = np.loadtxt(str(flengths[1]) + ".txt", delimiter=" ")

I opened each array by double-clicking the name in the Variable Explorer. "Cancel" and the window close button ("X") for both arrays produced zero noticeable effect on the RAM or CPU of any of the several Python and related processes my system had launched. However, an effect similar to that you described was seen on hitting "okay" for _both_ arrays; one of the four pythonw.exe subprocessies spawned by the Spyder pythonw.exe process would spike up to 12.5% CPU (I have a 4C/8T in this one, so that's 100% of one thread), and the RAM use would climb and fluctuate in near direct proportion to the size of the array; the small one would climb by about 700MB, while the large one would peak past 10GB. Further, the time taken for this to occur was also in apparent proportion—about three seconds or less for the smaller one, and around 40 for the larger. The behavior would occur in two phases, only really visible on the larger array—first, CPU would spike to full for about 15-20 seconds with modest RAM increases, then RAM would spike to near the max values and fluctuate widely for the remainder of the time, with CPU dropping back slightly, and finally both would go back to normal.

It seems reasonable to assume that each array is being fully rewritten in memory when "OK" is pressed regardless of whether anything was actually changed. Obviously not very efficient behavior, but regardless it seems somewhat odd that an array that should be on the order of 10 MB (~1 million 8-btye float64s) should require 3 orders of magnitude more the memory (10GB, 1000x) and 1-2 orders of magnitude of RAM bandwidth beyond that (12.8 GB/s * 40 s = 500 GB) and/or nearly 1 trillion CPU ops, (~25 GFLOPS/core * 40 s), depending on what was the limiting constraint) to modify.

[databoi]

Thank you for investigating @CAM-Gerlach. Pressing "Cancel" or "X" instead of "Ok" also solves the problem for me. Very helpful 👍.

[ccordoba12]

This is a very serious problem indeed! Thanks @CAM-Gerlach for verifying it.

@CAM-Gerlach, perhaps PR #5341 solves this problem. Could you try it, please? For that, you need to install first cloudpickle.

[CAM-Gerlach]

@ccordoba12 I tested it out, running both stable and dev spyder from that PR separately and together together, and unfortunately observed (to a reasonable approximation) the same behavior in terms of both with regard to CPU load, memory use, runtime, and the general pattern thereof.

[ccordoba12]

I found the problem with this one. I'll fix it after merging PR #5341.