Libtorrent: RC_1_1 uses too much memory

[sledgehammer999]

I am comparing RC_1_0 and RC_1_1 memory usage here.
Specifically when cache_size is set to -1, which has the same meaning on both branches but different results in real life.

To rule out bad usage from qBt I used the example/client_test program to download the same torrent with each branch. The only modification to the .cpp file was that I specifically hardcoded -1 for cache_size.

For a 320MB file RC_1_0 ram usage hovers at around 12MB, while RC_1_1 goes above 250MB ( at about 70% progress).

This was done on Windows. But then I compiled under linux to use valgrind and memcheck. It has the same problem but it doesn't appear to be a memleak. Valgrind reports only a few KB of leaks. I'll attach the log at the end.

qbt+RC_1_1 gives the feel that it just forgets to empty the cache and keeps unnecessary space. Even when I remove the torrent from the session the memory footprint doesn't go down.

I can't believe that this is by design. I hope that you have overlooked something.
If it is by design, is there any setting that will give me previous behavior?

Valgrind log: https://gist.github.com/sledgehammer999/3adfdb4e9c3b353b0d8379f9a7e28160

[ssiloti]

I'm able to repo this. Looks like we are indeed failing to clean up disk buffers immediately after they are flushed. This is probably related to the changes I made to always register write buffers in the cache. I'll investigate further and report back.

[ssiloti]

Looks like the bug is actually in RC_1_0. Setting cache_size to -1 is supposed to cause libtorrent to set the cache size to 1/8th the amount of physical memory in the system. Apparently in RC_1_0 this isn't working correctly. Setting cache_size to 0 disables the cache, which should keep memory usage low.

[sledgehammer999]

hmm, interesting. I'll do tests a few hours later and report back.

[sledgehammer999]

Here are my new observations:
RC_1_1 with cache_size = 0; uses about 12MiB. Same as RC_1_0 with -1
RC_1_0 with cache_size = 65536; aka 1GiB aka 1/8 of my RAM still uses about 12MiB. Is this a bug?

I am trying to understand how the cache is supposed to work here. And under what conditions it memory is released.

1. On RC_1_1 if I set cache_expiry to 3 seconds the memory footprint doesn't ever go down. Is this normal?
2. On RC_1_1 if I remove the torrent from session the memory footprint doesn't go down. Is this normal?
3. On RC_1_1 if I set the cache to something smaller than before while running the memory footprint doesn't go down. Is this normal? Assume that the new cache size is smaller than the grown memory.
4. After a torrent is done downloading is the cache supposed to keep all pieces in memory if the cache_size is big enough?

[ssiloti]

1. The cache_expiry setting affects how quickly write buffers are flushed to disk. Flushing a buffer to disk does not necessarily cause the buffer to be freed.
2. This appears to be a problem with the pool allocator not releasing memory. If I set use_disk_cache_pool to false the problem goes away.
3. I haven't tested this, but it's probably the same root cause as 2.
4. Yes, it is expected that flushed piece data will remain in the cache as long the cache size does not exceed the limit.

[ssiloti]

The pool allocator doesn't free its memory when the torrent is removed because it thinks it still has allocated buffers. The disk_buffer_pool thinks all buffers have been freed. With the pool allocator disabled I don't see any disk buffers leaking so I don't know why the pool allocator still sees allocated buffers.

[arvidn]

iirc, there's a function one can call on the pool to return unused buffers (release_memory()). I was under the impression I was making that call every now and then.

[ssiloti]

I do see release_memory being called, but it bails out without freeing any memory because it thinks there are still allocated buffers. Looking at the documentation for boost::pool I see that release_memory only works if ordered_malloc and ordered_free are used, which libtorrent does not do.

I for one would be in favor of disabling the pool allocator by default. I think the block cache is not particularly well suited for a pool allocator. The cache only occasionally does batch releases and 16KB blocks are not a particularly small allocation. It seems to me the pool is just being used as a nested malloc replacement, which is not really what it's meant for.

[arvidn]

sounds reasonable

[arvidn]

I imagine support for it may as well be removed

[sledgehammer999]

If I set use_disk_cache_pool to false the problem goes away.

I confirm. Also, if cache is set to something smaller, the footprint drops too.

FYI, in the qbt side I opened a PR that accepts 0 for disabling the cache and also setting use_disk_cache_pool to false.

Should this be closed or wait for you to implement a fix regarding the disk cache pool (or removing it)?

[ssiloti]

Leave this open. I plan to change the default use_disk_cache_pool setting to false in RC_1_1 and remove the feature entirely in master.

[Seeker2]

Any hope this might fix the virtual memory leak in qBitTorrent?: https://github.com/qbittorrent/qBittorrent/issues/7531

[arvidn]

it should mitigate memory being returned to the runtime at least. I wouldn't really expect the runtime (i.e. malloc()) to return virtual memory though, but maybe it does sometimes.
There was another memory leak fixed recently in the 1.1.5 branch as well, but it would be relatively slow, and it would increase actual memory usage, not just virtual memory usage.

This issue reminds me of an old issue where I was using VirtualAlloc() to allocate 16 kiB, where in fact VirtualAlloc() always allocates at least 65kiB of virtual address space. That obviously caused serious problems on 32 bit systems on windows. But that should be fixed, but I wonder if there's something similar going on somewhere.

Would you (or anyone else on this thread) be able to profile the memory usage to get a better sense of where the memory usage could be attributed?

Though, I wouldn't expect that the virtual address space usage always stays in sync with actual memory usage. I think it's hard to avoid some inefficiencies in this respect, in allocators.

[Seeker2]

It was only when quickly seeding a torrent that the ram usage grew much faster than the cache size.
Seemed like a direct ratio of cache size to virtual memory usage -- maybe 1 MB cache size translates into virtual memory usage increasing by 1.3-1.5 times as much.
That gets pretty silly on qBT 64bit with max-sized cache -- it'll reserve about 5.5-6.5 GB of virtual memory for the 4 GB cache. Even its "true" ram usage would be over 5 GB, despite having only a single torrent.

Removing the torrent would sometimes free up the ram but often left the virtual memory usage high.
My qBT 32bit crashes occurred when virtual memory hit 2 GB size even though qBT's ram used was often 1.9 GB or less.
Deluge suffered the same virtual memory problems as qBT, but didn't crash until it hit 3 GB virtual memory used.

I have a sinking feeling behavior may be radically different under a Virtual Machine's environment because those often do virtual memory translation and remapping.

[sledgehammer999]

@Seeker2 does behavior change if you enable/disable OS cache in the advanced settings?

I have a hunch that the problem @Seeker2 reports is related to thy way Windows file cache works. It just keeps growing without attributing the memory to the process's memory footprint but to the system's footprint. There doesn't' seem to be an API to finetune this per app.

[Seeker2]

@sledgehammer999 No, enable/disable OS cache has no significant effect on ram used by qBitTorrent.

Once again, Deluge exhibits the exact same behavior -- it just avoids crashing at 2 GB virtual memory despite being a 32bit app because it has the "large address aware" flag set to increase its memory limit to 3GB. It too crashes when it hits 3 GB virtual memory.

I am observing the virtual memory usage using Process Explorer (3rd party app) rather than just Windows Task Manager alone. There seems to be discrepancies between the 2 over peak and virtual memory use. In any case, qBT's reported ram used is still way higher than the cache size increases suggest.

This describes other aspects of the same bug:
https://code.google.com/p/libtorrent/issues/detail?id=508 mirrored at: https://github.com/xlgjjff/libtorrent/issues/508
Seems "[email protected]" and "[email protected]" had the same problem.
Also mentioned here: https://qbforums.shiki.hu/index.php/topic,3198.0.html