Trio: Potential memory leak on windows

[gesslerpd]

I was recently running some scripts overnight that failed due to a MemoryError, but was unable to find anything obvious that would be leaking memory in my scripts.

During my investigation process, I have found that the following script increases memory usage by ~1 MB/s while running. If the await trio.sleep(0) is replaced by pass the process runs at a constant 9.7 MB of memory usage.

import trio

async def main():
    while True:
        await trio.sleep(0)

if __name__ == '__main__':
    trio.run(main)

Version Info

Python 3.8.1
Trio 0.13.0
Windows 10

[njsmith]

Huh, that's plausible; on Windows we have to do some tricky low-level stuff with raw memory allocations and we could easily have messed something up there. Nice catch and thanks for the very clean reproducer.

[gesslerpd]

Just tried this on ubuntu since I hadn't before and the memory usage was constant 😄

[njsmith]

Thanks for checking! That's consistent with it being something in the Windows-only cffi code. I did a quick read of _io_windows.py and nothing jumped out at me. Maybe I don't understand how cffi's memory management works? Maybe it's something else and cffi is a red herring? Since we have a clean reproducer there's probably some memory debugging tool that could easily give a definitive answer, but unfortunately I'm not very familiar with Windows dev tooling.

[gesslerpd]

@njsmith

Using objgraph I am able to see that the number of function types seems to be growing by 4 each time the await trio.sleep(0) occurs.

The printed sets are the new function types created between each iteration so maybe those function names are helpful.

{<function Task._attempt_delivery_of_any_pending_cancel.<locals>.raise_cancel at 0x03969A98>, <function checkpoint.<locals>.<lambda> at 0x03969B28>, <function _filter_impl.<locals>.filter_tree at 0x03969ED0>, <function _filter_impl.<locals>.push_tb_down at 0x03969AE0>}
-----------------------
function                   5652
tuple                      2920
dict                       2914
weakref                    1307
wrapper_descriptor         1208
getset_descriptor          954
builtin_function_or_method 942
method_descriptor          932
cell                       765
type                       725
None
{<function checkpoint.<locals>.<lambda> at 0x03969A08>, <function _filter_impl.<locals>.filter_tree at 0x03969D20>, <function Task._attempt_delivery_of_any_pending_cancel.<locals>.raise_cancel at 0x03969030>, <function _filter_impl.<locals>.push_tb_down at 0x03969348>}
-----------------------
function                   5656
tuple                      2922
dict                       2914
weakref                    1307
wrapper_descriptor         1208
getset_descriptor          954
builtin_function_or_method 942
method_descriptor          932
cell                       768
type                       725
None
{<function _filter_impl.<locals>.push_tb_down at 0x03969618>, <function _filter_impl.<locals>.filter_tree at 0x039696A8>, <function Task._attempt_delivery_of_any_pending_cancel.<locals>.raise_cancel at 0x039695D0>, <function checkpoint.<locals>.<lambda> at 0x039698E8>}
-----------------------
function                   5660
tuple                      2924
dict                       2914
weakref                    1307
wrapper_descriptor         1208
getset_descriptor          954
builtin_function_or_method 942
method_descriptor          932
cell                       771
type                       725
None
{<function _filter_impl.<locals>.push_tb_down at 0x039CF300>, <function checkpoint.<locals>.<lambda> at 0x03969588>, <function _filter_impl.<locals>.filter_tree at 0x039697C8>, <function Task._attempt_delivery_of_any_pending_cancel.<locals>.raise_cancel at 0x03969078>}

[Dobatymo]

I think it might be related to ffi.getwinerror()

Calling

import tracemalloc
tracemalloc.start()

snapshot1 = tracemalloc.take_snapshot()
trio.run(main)
snapshot2 = tracemalloc.take_snapshot()

top_stats = snapshot2.compare_to(snapshot1, 'lineno')

print("[ Top 10 differences ]")
for stat in top_stats[:10]:
    print(stat)

gives me

Python37\lib\site-packages\cffi\api.py:479: size=7520 KiB (+7520 KiB), count=100004 (+100004), average=77 B

as largest difference.

And api.py:479 is

    def getwinerror(self, code=-1):
        return self._backend.getwinerror(code) <---

Called by raise_winerror() in trio/_core/_windows_cffi.py.

EDIT:
It looks like a cffi bug.
Calling

import cffi
ffi = cffi.FFI()
while True:
    a = ffi.getwinerror()

makes the memory grow rapidly.

[njsmith]

Awesome, thanks for tracking that down!

I've filed a bug upstream here: https://bitbucket.org/cffi/cffi/issues/442/getwinerror-leaks-memory

[njsmith]

Armin just commented there that he'd committed a fix to the cffi dev branch and is wondering if someone could check and confirm it works. I don't have a Windows C compiler handy, but if someone does and can install that and then try running the original test script again that would be cool.

[gesslerpd]

It's fixed with the changes that Armin committed, I don't have a bitbucket account or else I would have posted there.

Thanks everyone for helping track this one down!

[Dobatymo]

Running the initial snippet with the fixed cffi, memory usage does not increase anymore. I haven't looked into growing number of function types reported by @gesslerpd but they are probably getting cleaned up at some point.
I let the loop run for a few minutes and didn't see any increase (but I did see some slight fluctuations, probably caused by the gc).

So I think this can be closed. The requirements probably have to be updated to reference the latest cffi version as soon as it's released.

[gesslerpd]

Yeah agreed probably taken care of by the garbage collector at some point I just didn't notice it in the short run that I did.

Wasn't aware of tracemalloc, will have to remember that it's good for debugging these types of issues.

[pquentin]

cffi 1.14 is out with the fix: https://cffi.readthedocs.io/en/latest/whatsnew.html#v1-14

Should we update setup.py to specify 1.14+ instead of 1.12+?