Qbittorrent: 4.2.x high RAM memory usage due to HTTPS tracker announces for some users

[p43b1]

Please provide the following information

qBittorrent version and Operating System

4.2.2
Windows 10

If on linux, libtorrent-rasterbar and Qt version

(type here)

What is the problem

Excessively high RAM memory usage compared to 4.2.0

320MB after continuously running for two weeks compared to 2.5GB after running for 3 hours with the same of downloading/uploading torrents.

4.2.2 RAM USAGE

4.2.2 STATISTICS

4.2.0 RAM USAGE

4.2.0 STATISTICS

What is the expected behavior

(type here)

Steps to reproduce

(type here)

Extra info(if any)

(type here)

[ghost]

Cache settings were set to auto in 4.2.2. Maybe that could’ve caused this issue. Try setting it manually.

[FranciscoPombal]

Cache settings were set to auto in 4.2.2. Maybe that could’ve caused this issue. Try setting it manually.

Assuming all of the 2 GiB extra usage is cache, you'd have to have 80 GiB or more of system memory for libtorrent to allocate that much when the cache setting is set to auto. If you do have 80 GiB or more of system memory, then everything is working fine and the auto setting is just not suitable for your use case, in which case you should manually set it to some value that suits you. Otherwise, there is a bug in libtorrent (again, assuming all extra 2 GiB memory usage is cache).

[xavier2k6]

seems ok here....
official 4.2.2. -> disk cache (auto/default) (windows 10 x64/32gb RAM)

3 large torrents downloading.....

@p43b1 can you post a screenshot of task manager > performance -> memory.

[FranciscoPombal]

To really get to the bottom of this, and figure out if this is a problem with qBIttorrent or the system, it is necessary to use a profiler. I don't know how it would be on Windows, but it has to be something similar to the (callgrind+kcachegrind, massif) combo on Linux.
@xavier2k6 If you know how to do so on Windows, tell @p43b1 how to do it.

[xavier2k6]

@FranciscoPombal I'll have to look into that kind of thing....I'm just wondering if "compressed memory" on/off makes a difference, I have it enabled....hence why I asked for task manager screenshot.......currently can't go fiddling/testing with settings/test machines.

EDIT: Also @p43b1 do you have "Enable OS cache" checked or unchecked in the advanced settings?

[FranciscoPombal]

@xavier2k6
These were the results I found consistently when googling:
https://github.com/milostosic/MTuner
https://github.com/dynamorio/drmemory
https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap
https://hacksoflife.blogspot.com/2009/06/heap-debugging-memoryresource-leak-with.html

The Qt wiki also has a wiki page that names the first 2 among many others:
https://wiki.qt.io/Profiling_and_Memory_Checking_Tools

[xavier2k6]

@FranciscoPombal I have debugdiag attached 24/7 with full page heaps enabled......yet mem seems ok?!

I know there's a memory leak alright in Qt which will be fixed in Qt 5.14.2

[telans]

I think I may be having the same issue. Running on Linux with qbittorrent-nox 4.2.2 (libtorrent 1.2.5). Around 60 torrents with 8 active.

Been running for ~4 hours with just over 2 GiB usage. I do have 512MiB disk cache set, but the total usage still seems excessive.

Is this an issue or expected behaviour?

UPDATE: Just ran into OOM, restarted qbittorrent-nox and it's at 100MiB usage

[xavier2k6]

@FranciscoPombal did you get any further debug/info out of #11879?

[p43b1]

seems ok here....
official 4.2.2. -> disk cache (auto/default) (windows 10 x64/32gb RAM)

3 large torrents downloading.....

@p43b1 can you post a screenshot of task manager > performance -> memory.

All three taken at the very same moment

Edit:

[FranciscoPombal]

@xavier2k6 no, since I had the problems with the alert patches mentioned near the end, I did not give it another go. Perhaps I should try again.

@p43b1 Can MTuner show symbol names? Without them it is difficult to see where exactly the most memory is being allocated. Perhaps you need to point it at qBittorrent's .pdb file? Also can it export data in a massif-compatible format (or any format at all)? If so, post the export here so that others can analyze.

EDIT: I see at least the stack trace in the bottom right seems to have symbol names, but the useful bits are on the top of the stack (which would be in the bottom of the list, not visible in the screenshot). Please export a snapshot of the data when you are experiencing the increased memory usage so that others can attempt to analyze.

If this leak/increased usage is also related to the crypto library functions, this can be closed as a duplicate of the other issue and we can regroup and refocus our efforts over there.

[FranciscoPombal]

@telans Since you are on Linux, can you provide massif output from a run where you observe excessive memory usage? Read through https://github.com/qbittorrent/qBittorrent/issues/11879 if you don't know where to start.

[telans]

@telans Since you are on Linux, can you provide massif output from a run where you observe excessive memory usage? Read through #11879 if you don't know where to start.

Should I build with the suggested flags?

[xavier2k6]

@FranciscoPombal quick observation & thinking out loud..... @p43b1 has a set/custom disk cache 128MiB where as I have auto -> which follows libtorrents limits/algorithm (for amount of ram in the system)

I only presume that when a set/custom disk cache is used these limits are no longer adhered to......but is the set/custom disk cache figure restricted to just that amount & if it is....maybe the restrictor could be the issue/leaker.....

[FranciscoPombal]

@telans

Should I build with the suggested flags?

You should compile and build normally (assuming you are on Ubuntu, here is the guide: https://github.com/qbittorrent/qBittorrent/wiki/Compiling-qBittorrent-on-Debian-and-Ubuntu) and then run at least massif (example in the folder where the qbittorrent executable is built):
valgrind --tool=massif --verbose --massif-out-file=massif_output.mass qbittorrent

This comment shows other examples for memcheck and callgrind as well: https://github.com/qbittorrent/qBittorrent/issues/11879#issuecomment-578421275. But for this case, massif is the most important.

[FranciscoPombal]

@FranciscoPombal quick observation & thinking out loud..... @p43b1 has a set/custom disk cache 128MiB where as I have auto -> which follows libtorrents limits/algorithm (for amount of ram in the system)

I only presume that when a set/custom disk cache is used these limits are no longer adhered to......but is the set/custom disk cache figure restricted to just that amount & if it is....maybe the restrictor could be the issue/leaker.....

@p43b1 you can try auto, but I seriously doubt that choosing a specific value is causing a bug. As far as I know, that libtorrent code hasn't changed in a while. As an additional note, the amount of RAM used when auto is selected is chosen according to the following logic:
https://github.com/arvidn/libtorrent/blob/9ac4e6eed88c1ea1e6cdb6865889320164ad0b85/src/disk_buffer_pool.cpp#L298

[farnoy]

I managed to capture it, summary:

Command:            qbittorrent
Massif arguments:   --massif-out-file=/tmp/massif_output.mass
ms_print arguments: massif_output.mass
--------------------------------------------------------------------------------


    GB
2.859^                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #
     |                 #                                        @@
     |             ::::#:::@@::::::::::::::::::::::::::::::@:@@:@ :::::::::@@:
     | @ :: ::::::::: :#: :@ :: :: : ::: : : :: :::: ::: : @:@ :@ : :: :: :@@:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   823.0

Number of snapshots: 76
 Detailed snapshots: [1, 13 (peak), 16, 39, 41, 43, 57, 67]

Some details:

  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 58 856,650,432,519      424,960,408      416,647,537     8,312,871            0
 59 858,239,338,260      392,988,768      384,705,079     8,283,689            0
 60 859,832,688,763      419,595,648      411,267,061     8,328,587            0
 61 861,424,245,926      407,544,312      399,262,252     8,282,060            0
 62 863,013,195,999      446,006,032      437,654,179     8,351,853            0
 63 864,624,826,466      375,025,576      366,748,075     8,277,501            0
 64 866,213,876,908      412,153,368      403,844,962     8,308,406            0
 65 867,802,873,810      375,863,728      367,613,940     8,249,788            0
 66 869,391,757,318      448,007,912      439,672,697     8,335,215            0
 67 870,996,467,026      419,839,816      411,512,548     8,327,268            0
98.02% (411,512,548B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->33.31% (139,868,696B) 0x4C756E9: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| ->33.24% (139,567,624B) 0x4D7FE79: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | ->33.24% (139,558,056B) 0x4D3DC4E: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | | ->33.11% (139,020,104B) 0x4D4BC14: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | | | ->33.11% (139,020,104B) 0x4D60B23: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | | |   ->32.38% (135,925,280B) 0x4C5F58C: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | | |   | ->32.38% (135,925,280B) 0x4D8F675: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | | |   |   ->32.38% (135,925,280B) 0x65FCB23: execute_native_thread_routine (thread.cc:80)
| | | |   |     ->32.38% (135,925,280B) 0x4FF846E: start_thread (in /usr/lib/libpthread-2.31.so)
| | | |   |       ->32.38% (135,925,280B) 0x69773D2: clone (in /usr/lib/libc-2.31.so)
| | | |   |
| | | |   ->00.74% (3,094,824B) in 1+ places, all below ms_print's threshold (01.00%)
| | | |
| | | ->00.13% (537,952B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->00.00% (9,568B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.07% (301,072B) in 1+ places, all below ms_print's threshold (01.00%)
|
->13.73% (57,646,962B) 0x4E93921: libtorrent::torrent_info::parse_info_section(libtorrent::bdecode_node const&, boost::system::error_code&, int) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| ->13.73% (57,646,962B) 0x4E96C11: libtorrent::torrent_info::parse_torrent_file(libtorrent::bdecode_node const&, boost::system::error_code&, int) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|   ->13.73% (57,646,962B) 0x4E9AEC3: libtorrent::torrent_info::torrent_info(libtorrent::bdecode_node const&, boost::system::error_code&) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|     ->13.73% (57,646,962B) 0x27803C: BitTorrent::TorrentInfo::load(QByteArray const&, QString*) (in /usr/bin/qbittorrent)
|       ->13.73% (57,646,962B) 0x27AA9F: BitTorrent::TorrentInfo::loadFromFile(QString const&, QString*) (in /usr/bin/qbittorrent)
|         ->13.68% (57,423,929B) 0x258EF8: ??? (in /usr/bin/qbittorrent)
|         | ->13.68% (57,423,929B) 0x259378: BitTorrent::Session::startUpTorrents() (in /usr/bin/qbittorrent)
|         |   ->13.68% (57,423,929B) 0x223CA9: Application::exec(QStringList const&) (in /usr/bin/qbittorrent)
|         |     ->13.68% (57,423,929B) 0x21B1C7: main (in /usr/bin/qbittorrent)
|         |
|         ->00.05% (223,033B) in 1+ places, all below ms_print's threshold (01.00%)
|

Looks like thread creation might be leaking things? I killed it after reaching almost 3Gi RSS, while downloading a big torrent.

I'm not sure how to read it, but it seems to point the finger at spawning threads? But I strace-d it and it doesn't seem to clone any while downloading intensively :man_shrugging:

[FranciscoPombal]

@farnoy do you mind uploading the full output file, in case you saved it? You can simply attach it to your post.

[FranciscoPombal]

Also, if possible, compile and use libtorrent with debug symbols so that we can see the stacktrace for the libtorrent portion.

98.02% (411,512,548B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->33.31% (139,868,696B) 0x4C756E9: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| ->33.24% (139,567,624B) 0x4D7FE79: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | ->33.24% (139,558,056B) 0x4D3DC4E: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | | ->33.11% (139,020,104B) 0x4D4BC14: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | | | ->33.11% (139,020,104B) 0x4D60B23: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | | |   ->32.38% (135,925,280B) 0x4C5F58C: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
| | | |   | ->32.38% (135,925,280B) 0x4D8F675: ??? (in /usr/lib/libtorrent-rasterbar.so.10.0.0)

[farnoy]

I will recompile libtorrent with debugging, for now, this is the whole file.
massif_output.zip

[farnoy]

Ok, I'm building with autotools and a simple configure --enable-debug ... produces a build that:
1) is too slow to allocate anything substantial while downloading
2) crashes when closing qBittorrent with:

version: 1.2.5.0-9469913cb

file: '../../libtorrent-rasterbar-1.2.5/src/torrent.cpp'
line: 7875
function: check_invariant
expression: want_peers_finished() == m_links[aux::session_interface::torrent_want_peers_finished].in_list()

I'll try to compile a version with debug symbols, but with full optimizations and no assertions.

[p43b1]

Ok

1) Disk cache set to auto

2) Two screenshots from MTTuner (I think it doesn't record "live" but it makes just a snapshot)

[farnoy]

@FranciscoPombal Here's one with debug symbols massif_output.zip

I'm not sure this is correct, I killed it around 700Mi of RSS, but massif's summary is showing 255Mi peak memory used?

[mozo78]

No problems on Linux with 2648 runnung tasks:
https://i.imgur.com/S5b5G9g.png

[FranciscoPombal]

@farnoy thanks. If you could generate another one with an even bigger peak and a larger leak, that would be even better.

@arvidn can you take a look at this please? Even of that smaller one, is it normal for the receive buffer to be so big? And also parse_info_section?

[FranciscoPombal]

@p43b1 can you please compile with debug symbols/configure MTTuner to use debug symbols, and post the exported data so that others can analyze freely in more detail? I realize this is not the easiest thing to do on Windows, it's fine if you don't do it.

But just as an example, with @farnoy's output posted above, I can open it and look at it in detail in massif-visualizer:

[telans]

@FranciscoPombal Running with valgrind left the memory as what I would expect, around 750MiB usage.

There isn't a specific action I do to make the memory jump, it just seems to leak over time. A few hours with valgrind uses much less memory than an hour without valgrind.

I'll leave it overnight to really test it. (or is it expected that it should take longer?)

[FranciscoPombal]

@telans I would not say leaving it running all night is necessary. A couple of hours of usage OR until it reaches a couple of GiBs of memory usage or more, whichever comes first, should be enough to gather some interesting data.

[telans]

I've managed to trigger the 'leak' (or whatever it is) somehow twice more without valgrind, but for some reason it's just not occuring with valgrind.

Without valgrind the usage slowly climbs over 1.5GiB, but with it always remains steady at 730MiB (500 disk buffer, 230 for the rest). Are there any other memory analyzing tools I should try?

[p43b1]

I hope I could do something useful...

1) For reproducibility

I used this software https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap

2) This is the screenshot of qbittorrent's process

3) This is the file. It's the same file, one in .mmp, one in .csv file format.

qbittorrent_csv_file_format.zip
qbittorrent_mmp_file_format.zip

[farnoy]

I ran it overnight, but couldn't reproduce the long term leak on libtorrent compiled with -g -O2 and no assertions.

EDIT: I was testing with valgrind --tool=massif, so maybe it is to blame for preventing leakage.

[joy4eg]

4.2.2, fedora 32. 230+ torrents.

15 GB ...

[joy4eg]

Related issue: https://github.com/qbittorrent/qBittorrent/issues/11581

[telans]

4.2.2, fedora 32. 230+ torrents.

Are you able to trigger the leak with massif?

valgrind --tool=massif --verbose --massif-out-file=massif_output.mass qbittorrent

[arvidn]

Is 15 GB a lot a compared to your expectation or compared to your cache size setting?
If the former, what is your cache size setting?
If it's "auto", given your system, what would a reasonable automatic cache size be? (currently it's essentially a log function of the total amount of RAM)

In any other case, there could be a bug. However, probably the second most memory consuming part are socket buffers. If you have a lot of peers, it's possible for those to get quite big too. Both of these numbers are actually available in the session stats.

[arvidn]

also, if anyone is of the opinion that qBT shouldn't have a cache size setting, but it should "just work", I agree. The next major version of libtorrent will defer disk caching to the operating system. But that's still some time out.

[joy4eg]

@arvidn Yeah, I have "auto". But the previous version took much less memory - https://github.com/qbittorrent/qBittorrent/issues/11581#issuecomment-566914529

I'll try to dig in with valgrind as @telans proposed.

[joy4eg]

So it's pretty reproducible on my machine.

That screen is from massif_output2.mass: google drive.

For some reason all roads lead to on_tracker_announce()

[farnoy]

I was able to capture a profile with debug info of libtorrent. I have disk cache set to disabled (0), this is the interesting snippet:

  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 39 156,113,888,304    2,193,275,384    2,176,662,491    16,612,893            0
 40 160,544,089,185    1,338,315,976    1,327,048,580    11,267,396            0
 41 162,768,723,780      163,928,720      158,563,786     5,364,934            0
 42 167,210,833,411    1,369,959,664    1,357,736,612    12,223,052            0
 43 169,567,102,787    3,774,431,808    3,749,046,029    25,385,779            0
99.33% (3,749,046,029B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->20.27% (765,255,680B) 0x495604F: ??? (in /usr/lib/libcrypto.so.1.1)
| ->20.27% (765,255,680B) 0x4954115: BIO_ctrl (in /usr/lib/libcrypto.so.1.1)
|   ->20.27% (765,255,680B) 0x4956170: BIO_new_bio_pair (in /usr/lib/libcrypto.so.1.1)
|     ->20.27% (765,255,680B) 0x4E0D566: boost::asio::ssl::detail::stream_core::stream_core<boost::asio::executor>(ssl_ctx_st*, boost::asio::executor const&) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|       ->20.27% (765,255,680B) 0x4E07E61: libtorrent::aux::socket_type::construct(int, void*) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|         ->20.27% (765,255,680B) 0x4D0D4C0: libtorrent::aux::instantiate_connection(boost::asio::io_context&, libtorrent::aux::proxy_settings const&, libtorrent::aux::socket_type&, void*, libtorrent::utp_socket_manager*, bool, bool) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|           ->20.27% (765,255,680B) 0x4CBA85B: libtorrent::http_connection::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, int, libtorrent::aux::proxy_settings const*, bool, int, boost::optional<boost::asio::ip::address> const&, libtorrent::flags::bitfield_flag<unsigned char, libtorrent::resolver_flag_tag, void>, libtorrent::i2p_connection*) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|             ->20.27% (765,255,680B) 0x4CBBC8D: libtorrent::http_connection::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >, int, libtorrent::aux::proxy_settings const*, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::optional<boost::asio::ip::address> const&, libtorrent::flags::bitfield_flag<unsigned char, libtorrent::resolver_flag_tag, void>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, libtorrent::i2p_connection*) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|               ->20.27% (765,255,680B) 0x4CFF4E3: libtorrent::http_tracker_connection::start() (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|                 ->20.27% (765,255,680B) 0x4EB9DF0: libtorrent::tracker_manager::queue_request(boost::asio::io_context&, libtorrent::tracker_request&&, std::weak_ptr<libtorrent::request_callback>) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|                   ->20.27% (765,255,680B) 0x4DD5B12: libtorrent::aux::session_impl::queue_tracker_request(libtorrent::tracker_request&&, std::weak_ptr<libtorrent::request_callback>) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|                     ->20.27% (765,255,680B) 0x4E57B3B: libtorrent::torrent::announce_with_tracker(unsigned char) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|                       ->20.27% (765,255,680B) 0x4E59476: libtorrent::torrent::on_tracker_announce(boost::system::error_code const&) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|                       | ->20.27% (765,255,680B) 0x4E84DF6: void libtorrent::torrent::wrap<void (libtorrent::torrent::*)(boost::system::error_code const&), boost::system::error_code const&>(void (libtorrent::torrent::*)(boost::system::error_code const&), boost::system::error_code const&) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|                       |   ->20.27% (765,255,680B) 0x4E72BF7: boost::asio::detail::wait_handler<libtorrent::torrent::update_tracker_timer(std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<int, std::ratio<1l, 1l> > >)::{lambda(boost::system::error_code const&)
|                       |     ->20.27% (765,255,680B) 0x4C6427C: boost::asio::detail::scheduler::run(boost::system::error_code&) (in /usr/lib/libtorrent-rasterbar.so.10.0.0)
|                       |       ->20.27% (765,255,680B) 0x4D9D125: std::thread::_State_impl<std::thread::_Invoker<std::tuple<libtorrent::session::start(libtorrent::session_params&&, boost::asio::io_context*)::{lambda()
|                       |         ->20.27% (765,255,680B) 0x6613B23: execute_native_thread_routine (thread.cc:80)
|                       |           ->20.27% (765,255,680B) 0x501146E: start_thread (in /usr/lib/libpthread-2.31.so)
|                       |             ->20.27% (765,255,680B) 0x698E3D2: clone (in /usr/lib/libc-2.31.so)
|                       |
|                       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)

massif_output_debug.zip

[FranciscoPombal]

@arvidn
I re-opened a result from an older thread (https://github.com/qbittorrent/qBittorrent/issues/11879#issuecomment-581017871) and the results are similar to what is observed now (keep in mind the libtorrent version used here is a bit older):

Comparing to the massif outputs in this thread, looks like this is still the HTTPS tracker announce issue from before.

[FranciscoPombal]

@p43b1 I don't think those files are very useful, at least the .csv did not contain any stacktrace info. I'm sorry I can't guide you further, I don't know how to produce similar reports on Windows.

[eigerzoom]

What I dont understand is when rolling back to previous versions of qb client, before the problem existed, the issue remains. That simply doesn't add up. Can someone take a stab at explaining this?

[telans]

What I dont understand is when rolling back to previous versions of qb client, before the problem existed, the issue remains. That simply doesn't add up. Can someone take a stab at explaining this?

I assume it's because you still have the latest libtorrent installed

[farnoy]

So what's the consensus so far? Is it fair to say that the issue is in libtorrent and any changes qBittorrent has done recently have not introduced it, but perhaps made it trigger more frequently (to unleash waves of people reporting this issue here)?

[arvidn]

my theory is that the SSL async_shutdown call never completes. The SSL support in libtorrent was slapped on a little bit as an afterthought, and normally, closing a socket is not an operation that takes time or stalls. But in order to (correctly) close an SSL stream, you need to do a shutdown handshake with the other end. If the tracker doesn't bother doing clean shutdowns (which really is a security issue) or if the network is flaky and the connection is dropped, libtorrent can get stuck indefinitely waiting for the handshake.

It could be related to this change: https://github.com/arvidn/libtorrent/pull/2797

Either way, I hope to have time, at some point, to rewrite the HTTP trackers to be simpler and more robust (based on boost.beast), and fix this issue better.
I think there really need to be a time-out for this.

[farnoy]

@arvidn Out of curiosity, and sorry if this is off topic, but are you sure it's a security issue to disconnect the socket without cleanly ending the TLS session? I followed the links you posted and the example given is one where the attacker blocks a user from logging out and ending their (web) session. Doesn't seem relevant in this case.

[arvidn]

I can't think of a specific attack, other than denial of service by blocking packets. But I also don't think it's safe to assume there aren't any.

anyway, could someone please try https://github.com/arvidn/libtorrent/pull/4494 ?

[joy4eg]

@arvidn

assertion failed. Please file a bugreport at https://github.com/arvidn/libtorrent/issues
Please include the following information:

version: 1.2.5.0-9469913cb

file: '../../src/socket_type.cpp'
line: 338
function: close
expression: <unconditional>

stack:
1: libtorrent::assert_fail(char const*, int, char const*, char const*, char const*, int)
2: 
3: 
4: boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)
5: boost::asio::detail::scheduler::run(boost::system::error_code&)
6: 
7: 
8: 
9: clone


Thread 5 "qbittorrent" received signal SIGABRT, Aborted
[Switching to Thread 0x7fffe41e1700 (LWP 380573)]
0x00007ffff75e0915 in raise () from /usr/lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff75e0915 in raise () from /usr/lib64/libpthread.so.0
#1  0x00007ffff77c82ed in libtorrent::assert_fail (expr=0x7ffff7af20ea "<unconditional>", line=338, file=0x7ffff7b1dff4 "../../src/socket_type.cpp", function=0x7ffff7b058ff "close", value=<optimized out>, kind=0)
    at ../../src/assert.cpp:380
#2  0x00007ffff7848f74 in operator() (__closure=0x7fffe41df878) at /usr/include/c++/10/bits/shared_ptr_base.h:1321
#3  boost::asio::ssl::detail::shutdown_op::call_handler<libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > (ec=..., handler=..., this=0x7fffe41df850) at /usr/include/boost/asio/ssl/detail/shutdown_op.hpp:45
#4  boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >::operator()(boost::system::error_code, std::size_t, int) (this=0x7fffe41df840, ec=..., bytes_transferred=0, start=<optimized out>) at /usr/include/boost/asio/ssl/detail/io.hpp:262
#5  0x00007ffff7849cd4 in boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int>::operator() (this=0x7fffe41df840) at /usr/include/boost/asio/detail/bind_handler.hpp:162
#6  boost::asio::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int> > (function=...) at /usr/include/boost/asio/handler_invoke_hook.hpp:69
#7  boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int>, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#8  boost::asio::ssl::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int>, boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > (this_handler=0x7fffe41df840, function=...) at /usr/include/boost/asio/ssl/detail/io.hpp:316
#9  boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int>, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#10 boost::asio::detail::handler_work<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::asio::system_executor>::complete<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int> > (handler=..., function=..., this=<synthetic pointer>) at /usr/include/boost/asio/detail/handler_work.hpp:82
#11 boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > >::do_complete(void *, boost::asio::detail::operation *, const boost::system::error_code &, std::size_t) (owner=0xe168d0, base=0x7fff507c5180)
    at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:122
#12 0x00007ffff78074c5 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=<optimized out>, ec=..., owner=0xe168d0, this=0x7fff507c5180) at /usr/include/boost/asio/detail/scheduler_operation.hpp:40
#13 boost::asio::detail::scheduler::do_run_one (this=this@entry=0xe168d0, lock=..., this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:401
#14 0x00007ffff78532c1 in boost::asio::detail::scheduler::run (this=0xe168d0, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:154
#15 0x00007ffff792cb58 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<libtorrent::session::start(libtorrent::session_params&&, boost::asio::io_context*)::{lambda()#1}> > >::_M_run() ()
    at /usr/include/boost/asio/impl/io_context.ipp:62
#16 0x00007ffff6102a94 in execute_native_thread_routine () from /usr/lib64/libstdc++.so.6
#17 0x00007ffff75d5432 in start_thread () from /usr/lib64/libpthread.so.0
#18 0x00007ffff5e009d3 in clone () from /usr/lib64/libc.so.6

I have rb_torrent from fedora 32 + your PR as a patch (debug enabled).
And also crashing without debug flag:

#0  operator() (__closure=0x7fffe429f878) at ../../src/http_connection.cpp:474
#1  boost::asio::ssl::detail::shutdown_op::call_handler<libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > (ec=..., handler=..., this=0x7fffe429f850) at /usr/include/boost/asio/ssl/detail/shutdown_op.hpp:45
#2  boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >::operator()(boost::system::error_code, std::size_t, int) (this=0x7fffe429f840, ec=..., bytes_transferred=0, start=<optimized out>) at /usr/include/boost/asio/ssl/detail/io.hpp:262
#3  0x00007ffff78dac74 in boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int>::operator() (this=0x7fffe429f840) at /usr/include/boost/asio/detail/bind_handler.hpp:162
#4  boost::asio::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int> > (function=...) at /usr/include/boost/asio/handler_invoke_hook.hpp:69
#5  boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int>, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > (context=..., function=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#6  boost::asio::ssl::detail::asio_handler_invoke<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int>, boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > (this_handler=0x7fffe429f840, function=...) at /usr/include/boost/asio/ssl/detail/io.hpp:316
#7  boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int>, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > > (context=..., function=...) at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#8  boost::asio::detail::handler_work<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::asio::system_executor>::complete<boost::asio::detail::binder2<boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> >, boost::system::error_code, long unsigned int> > (handler=..., function=..., this=<synthetic pointer>) at /usr/include/boost/asio/detail/handler_work.hpp:82
#9  boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp>, boost::asio::ssl::detail::shutdown_op, libtorrent::http_connection::close(bool)::<lambda(const error_code&)> > >::do_complete(void *, boost::asio::detail::operation *, const boost::system::error_code &, std::size_t) (owner=0xe127a0, base=0x7fff542724a0)
    at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:122
#10 0x00007ffff78a4f69 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=<optimized out>, ec=..., owner=0xe127a0, this=0x7fff542724a0) at /usr/include/boost/asio/detail/scheduler_operation.hpp:40
#11 boost::asio::detail::scheduler::do_run_one (this=this@entry=0xe127a0, lock=..., this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:401
#12 0x00007ffff78e3971 in boost::asio::detail::scheduler::run (this=0xe127a0, ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:154
#13 0x00007ffff798f758 in boost::asio::io_context::run (this=<optimized out>) at /usr/include/boost/asio/impl/io_context.ipp:62
#14 operator() (__closure=<optimized out>) at ../../src/session.cpp:355
#15 std::__invoke_impl<void, libtorrent::session::start(libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#16 std::__invoke<libtorrent::session::start(libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > (__fn=...) at /usr/include/c++/10/bits/invoke.h:95
#17 std::thread::_Invoker<std::tuple<libtorrent::session::start(libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > >::_M_invoke<0> (this=<optimized out>) at /usr/include/c++/10/thread:264
#18 std::thread::_Invoker<std::tuple<libtorrent::session::start(libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > >::operator() (this=<optimized out>) at /usr/include/c++/10/thread:271
#19 std::thread::_State_impl<std::thread::_Invoker<std::tuple<libtorrent::session::start(libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > > >::_M_run(void) (this=<optimized out>)
    at /usr/include/c++/10/thread:215
#20 0x00007ffff61c2a94 in execute_native_thread_routine () from /usr/lib64/libstdc++.so.6
#21 0x00007ffff7695432 in start_thread () from /usr/lib64/libpthread.so.0
#22 0x00007ffff5ec09d3 in clone () from /usr/lib64/libc.so.6
(gdb) info locals
e = {val_ = 0, failed_ = false, cat_ = 0xbe8140 <boost::system::detail::cat_holder<void>::system_category_instance>}
self = std::shared_ptr<libtorrent::http_connection> (expired, weak count 0) = {get() = 0x1}
self = <optimized out>
e = <optimized out>

   468          {
   469  #ifdef TORRENT_USE_OPENSSL
   470                  auto self = shared_from_this();
   471                  auto handler = [&](error_code const&) {
   472                          COMPLETE_ASYNC("on_close_socket");
   473                          error_code e;
   474                          self->m_timer.cancel(e);
   475                          self->m_sock.close(e);
   476                  };

[arvidn]

@joy4eg thanks for testing! I force pushed a fix to that PR

[joy4eg]

@arvidn Awesome! I'll leave it running for a while to collect a new massif report.

[joy4eg]

@arvidn
massif_output3.zip

Looks like the issue is still here, but probably in some another way.

[arvidn]

thanks again for testing. I'm a bit hesitant to fully trust that massif capture. It looks really suspicious that the top 4 allocation callstacks all allocate exactly the same number of bytes (920.4 MiB).

It definitely seem to be pointing to the SSL context for the tracker connection. In fact, it looks like it's the http_connection object that isn't being released. In fact, the http_connection object not being released is consistent with the issue my PR is trying to fix. @joy4eg could you double check that you're actually testing with the patched libtorrent (and aren't picking up a system libtorrent by mistake).

[joy4eg]

@joy4eg could you double check that you're actually testing with the patched libtorrent (and aren't picking up a system libtorrent by mistake).

I did rpm rebuild with your patch. So it is a system library, but patched :)
I'll double check that.

UPDATE:
Yeah, everything is in place. But I have 1.2.5 version which is a bit different than in your PR.
So will try to build async-shutdown-timeout branch.

[arvidn]

thanks. I have a feeling it's something else though

[joy4eg]

@arvidn

massif_output4.zip
So this is the test with libtorrent from async-shutdown-timeout branch. As for me, it looks the same as previous test.

[FranciscoPombal]

@joy4eg @arvidn
In one of the big allocations of the peak snapshot, lines 59963 to 59985 (head -n 59985 massif_output4.mass | tail -n 23) there is something interesting (I truncated a few lines and simplified others for readability, look at the last 5 where I left the source locations intact at the end of the lines):

n14: 7497774607 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n1: 1121893376 0x49CD25E: ??? (in /usr/lib64/libcrypto.so.1.1.1d)
  n1: 1121893376 0x49CB329: BIO_ctrl (in /usr/lib64/libcrypto.so.1.1.1d)
   n1: 1121893376 0x49CD3FF: BIO_new_bio_pair (in /usr/lib64/libcrypto.so.1.1.1d)
        <truncated>
        n1: 1121893376 0x4F51C8F: libtorrent::aux::socket_type::construct          <removed>
         n1: 1121893376 0x4E7145C: instantiate<libtorrent::ssl_stream              <removed>
          n1: 1121893376 0x4E7145C: libtorrent::aux::instantiate_connection        <removed> (instantiate_connection.cpp:85)
           n1: 1121893376 0x4E3832A: libtorrent::http_connection::start            <removed> (http_connection.cpp:331)
            n2: 1121893376 0x4E39270: libtorrent::http_connection::get             <removed> (http_connection.cpp:210)
             n1: 1120117760 0x4E67371: libtorrent::http_tracker_connection::start  <removed> (http_tracker_connection.cpp:230)
              n1: 1120117760 0x4FF1C22: libtorrent::tracker_manager::queue_request <removed> (tracker_manager.cpp:283)

The callstack looks like this:

1. The tracker manager starts an http_tracker_connection
2. in http_tracker_connection, http_connection::get is called
3. in turn, http_connection::start is called
4. http_connection::start calls aux::instantiate_connection at line 331.

Line 331 of http_connection.cpp has the following comment directly on top of it:

                // assume this is not a tracker connection. Tracker connections that
                // shouldn't be subject to the proxy should pass in nullptr as the proxy
                // pointer.

So it seems (some?) tracker connections are not being treated as tracker connections. Could that be the cause of this memory leak?

[arvidn]

I tried simulating async_shutdown stalling and I could reproduce this issue. I've made another attempt at fixing it. Please give this a try! https://github.com/arvidn/libtorrent/pull/4515

[joy4eg]

@arvidn Awesome! I'll start another test :) Thanks!

[FranciscoPombal]

@arvidn Can you elaborate on the conditions you need to have to reproduce this? I could never reproduce this myself (at least with qBittorrent).

[arvidn]

well, I reproduced it in simulation, so to speak. I replaced the async_shutdown() call with a timer that would take 30 minutes to fire. As if the async_shutdown() call would stall. That way I could see that the regular timeout was deactivated and would never close the connection. I could fix that to ensure it does close the connection in this case.

[arvidn]

I think it requires a specific HTTPS tracker that doesn't do graceful shutdowns on its connections.

[joy4eg]

@arvidn

massif_output5.zip

I think now it is a bit better, but issue is still here.

[FranciscoPombal]

@joy4eg definitely a smaller peak for an even greater number of snapshots. Was the usage stabilized before you interrupted the capture? Or was it still increasing?

@arvidn
The callstack of the biggest allocations looks identical to what I described in https://github.com/qbittorrent/qBittorrent/issues/12326#issuecomment-609034342. Do you think there may be an issue related to that, based on my observation in that comment?

The line in question:
https://github.com/arvidn/libtorrent/blob/2aeb611112c392d44f94d0df0abefc8a2c200e75/src/http_connection.cpp#L331

[joy4eg]

@joy4eg definitely a smaller peak for an even greater number of snapshots. Was the usage stabilized before you interrupted the capture? Or was it still increasing?

It was running over night, and I interrupted it in the morning.

[arvidn]

I'm considering a different approach to timing out hung async_shutdown, here: https://github.com/arvidn/libtorrent/pull/4522

I don't expect it to have any different outcome, so the last massif report is probably still useful. I haven't had a chance to look at it yet though.

[txtsd]

I'm seeing upwards of 1GB RAM usage after just 6 hours.

Qt: 5.14.2
Libtorrent: 1.2.5.0
Boost: 1.72.0
OpenSSL: 1.1.1f
zlib: 1.2.11

[FranciscoPombal]

@txtsd

I'm seeing upwards of 1GB RAM usage after just 6 hours.

Qt: 5.14.2
Libtorrent: 1.2.5.0
Boost: 1.72.0
OpenSSL: 1.1.1f
zlib: 1.2.11

Depending on the amount of torrents and peers connected, the amount of RAM your system has, your disk cache settings, and the specific libtorrent commit you are using (because recently a bug was fixed with setting disk cache to auto), this amount of RAM consumption may be totally expected and reasonable.

[txtsd]

@FranciscoPombal About 40 torrents, but only 4-5 active, and disk cache is disabled.

[FranciscoPombal]

@txtsd can you post a massif capture or equivalent, to confirm whether or not you have a memory leak, and if so, it it is of the same nature as the one users are experiencing in this thread?

[txtsd]

@FranciscoPombal I ran valgrind for 2 days but the 1GB usage didn't happen. It sat at around 600MB.
qbit_massif_output.mass.zip

[FranciscoPombal]

@txtsd thanks, looks like you are not experiencing the same problem as others in this thread though.

[maestroprog]

I have this problem too.

Ubuntu 20.04 LTS
qBittorrent v4.2.5

[FranciscoPombal]

@joy4eg @farnoy @p43b1

Can you reproduce this on 4.2.5 or higher, with libtorrent >= 1.2.6 (or at least equal or newer to libtorrent@61d2c8c1f57b4750b5ff33508c098ac3813bd489)?

[farnoy]

I've had qBittorrent 4.2.5 with libtorrent 1.2.6 installed since Apr 28 and it continues to happen regularly

[AndCycle]

here is my massif with

net-libs/libtorrent-rasterbar-1.2.6
net-p2p/qbittorrent-4.2.5

massif.out.zip

[FranciscoPombal]

@AndCycle

here is my massif with

net-libs/libtorrent-rasterbar-1.2.6
net-p2p/qbittorrent-4.2.5

massif.out.zip

A substantial amount of your usage is disk cache that stabilizes pretty quickly (so not a concern, though you may wish to reduce it a bit if as you may not need that much), but once again, like in the other captures, there does seem to be a leak related to HTTP(S) tracker requests.

@arvidn
relevant lines from a cursory look:

https://github.com/arvidn/libtorrent/blob/b9b54436b8b214420745e5ff722112316d7d85c7/src/http_tracker_connection.cpp#L205

and

https://github.com/arvidn/libtorrent/blob/b9b54436b8b214420745e5ff722112316d7d85c7/src/tracker_manager.cpp#L281 (up to 283)

Could there be a mistake with some of the shared pointers? Admittedly, this part of the code is a bit over my head right now.

@ everyone else in the thread - for those affected by the leak, it would be very helpful if you could attempt to debug it, or at least git-bisect libtorrent (and/or qBittorrent, or both) to find the problematic change.

[edalquist]

I'm seeing this as well (1:4.2.5.99~202004250119-7015-2c65b79~ubuntu18.04.1). I had 3 seeding and one active torrent and memory usage grew to 3.8GiB in a LXC container with 4GiB of memory over ~30 minutes.

I did a massif trace but realized it isn't useful without debug symbols. Are there instructions for recompiling libtorrent with just debug symbols? I see a few snippets of what people have done in this thread but would appreciate an example.

[FranciscoPombal]

@edalquist

I recommend using the latest version of cmake for building. Kitware maintains an official repo here: https://apt.kitware.com/. Alternatively, there is a binary package available in the official downloads page.

For libtorrent (change the -DCMAKE_INSTALL_PREFIX= and -G options according to your preference):

$ cd libtorrent
$ cmake -S . -B cmake-build-dir -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_CXX_STANDARD=14 -Ddeveloper-options=ON -Ddeprecated-functions=OFF
$ cmake --build cmake-build-dir
$ sudo cmake --install cmake-build-dir
# to uninstall: sudo xargs rm < cmake-build-dir/install_manifest.txt

For a pure debug build, use -DCMAKE_BUILD_TYPE=Debug instead - it will be slower, and I doubt there is any advantage for the purposes of a massif capture. Maybe you could try both, see if there is any difference?

Then, for qBittorrent, assuming the -DCMAKE_INSTALL_PREFIX= you chose for libtorrent is in your system's library path:

$ cd qBittorrent
$ cmake -S . -B build -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_STANDARD=14
$ cmake --build build
# then just run the resulting executable from the build folder (could be in `build/app`, actually), no need to actually install it

This should enable you to get massif traces with debugging symbols for both qBittorrent and libtorrent.

[edalquist]

Thanks Francisco, dumb followup question. This is a headless server, what is the cmake flag I need for the configure --disable-gui equivalent?

[FranciscoPombal]

@edalquist

-DGUI=OFF

[edalquist]

No luck: https://gist.github.com/edalquist/5bad2e1041c099b71c664bb61c673c4b#file-gistfile1-txt

Or do I need the QT5 deps even with a headless compile?

[FranciscoPombal]

@edalquist

No luck: https://gist.github.com/edalquist/5bad2e1041c099b71c664bb61c673c4b#file-gistfile1-txt

Or do I need the QT5 deps even with a headless compile?

Yes, qBittorrent still depends on a few Qt5 components even without the GUI. You can see this by looking at where the configure failed (and nearby lines):

line 14

find_package(Qt5 ${requiredQtVersion} REQUIRED COMPONENTS Core Network Xml LinguistTools)
if (GUI)
    find_package(Qt5Widgets ${requiredQtVersion} REQUIRED)
    find_package(Qt5DBus ${requiredQtVersion})
endif()

As per https://github.com/qbittorrent/qBittorrent/wiki/Compiling-qBittorrent-on-Debian-and-Ubuntu, here are the Qt packages you need for this on Ubuntu >= 18.04: sudo apt install qtbase5-dev qttools5-dev libqt5svg5-dev

[edalquist]

Thanks for all the help. I did this capture starting with an empty client (no torrents) then added some ubuntu torrents one after each previous was complete (20.04 gui, 20.04 server, 18.04 gui).

• massif output: https://gist.github.com/edalquist/6b06a7cdd4bad18f33633be27823a408
• Version Info
  
  
  
  • Qt: | 5.9.5
  
  
  • Libtorrent: | 1.2.6.0
  
  
  • Boost: | 1.65.1
  
  
  • OpenSSL: | 1.1.1
  
  
  • zlib: | 1.2.11
  
  

libtorrent advanced settings:

On a LXC container with 4GiB of RAM qbt consumes ~3.7GiB after this test and doesn't appear to release it if I try to use stress to allocate more memory than is currently free.

[arvidn]

As far as I can tell, virtually all of that RAM is used by the disk cache. You set the disk cache to -1, meaning "auto".
Have you tried setting it to an explicit size, to see if that works? Say, you could set it to 2 GiB.

The way "auto" works is that it tries to learn how much RAM there is in the machine in total, and then sets the disk size proportionally to that. It's possible that, when running in an LXC container, libtorrent fails to see the amount of RAM assigned to the container, but perhaps reads the total amount of RAM in the machine as a whole.

The way libtorrent queries for the amount of RAM is:

sysconf(_SC_PHYS_PAGES) * sysconf(_SC_PAGESIZE);

Could you test to see what those sysconf calls return from within the container?

[edalquist]

Looks like libtorrent probably sees the entire server's memory and not the container's RAM resulting in the high memory usage. The server has 128GiB vs the container's 4GiB. When I first set this container up I had only given it 1GiB of RAM and things worked, unfortunately I made a bunch of updates/changes from that setup and noticing the increased memory usage which initially presented as the kernel killing the qbt process for using too much memory.

$ getconf -a |grep PAGE
PAGESIZE                           4096
PAGE_SIZE                          4096
_AVPHYS_PAGES                      199661
_PHYS_PAGES                        32978276

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           4.0G        853M        3.1G        192K         85M        3.2G
Swap:            0B          0B          0B

$ cat /proc/meminfo
MemTotal:        4194304 kB
MemFree:         3232512 kB
MemAvailable:    3320164 kB
Buffers:               0 kB
Cached:            87652 kB
SwapCached:            0 kB
Active:           898528 kB
Inactive:          23676 kB
Active(anon):     835692 kB
Inactive(anon):      132 kB
Active(file):      62836 kB
Inactive(file):    23544 kB
Unevictable:           0 kB
Mlocked:          160156 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              1452 kB
Writeback:             0 kB
AnonPages:        835632 kB
Mapped:            31284 kB
Shmem:               192 kB
KReclaimable:    3207764 kB
Slab:               0 kB
SReclaimable:          0 kB
SUnreclaim:            0 kB
KernelStack:       36272 kB
PageTables:       146828 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    65956552 kB
Committed_AS:   50433752 kB
VmallocTotal:   34359738367 kB
VmallocUsed:     1192040 kB
VmallocChunk:          0 kB
Percpu:           291840 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:    53784336 kB
DirectMap2M:    77207552 kB
DirectMap1G:     5242880 kB

[arvidn]

do you think looking at getrlimit() RLIMIT_AS would reflect the true limit of the container?

[edalquist]

Not sure, I don't have much experience with c/system level programming.

I suppose it would be better for to move my issue out to its own bug. Explicitly setting Disk Cache to 128MiB fixes my memory growth problem and in the same test keeps qbt memory usage at 181MiB total for that same set of torrents.

[arvidn]

The next major release of libtorrent (2.0, which is in RC right now, but probably still some ways out of making it into qbt) will solve this problem by not using an application-level disk cache, but rely on the kernel.

It's probably a good idea to limit the auto-cache size (in the current stable branch) based on the rlimit.

[AndCycle]

20200609-massif.out.zip

another massif with net-libs/libtorrent-rasterbar-1.2.7
changed the cache to fixed number during the test

[FranciscoPombal]

@AndCycle yep, in your capture, the tracker_manager::*_request family seems to be the culprit, like in the others.

[guillaumedsde]

Hi, this is a follow up from my posts in #12875:

A couple of things I have tried to no avail:

• disabling OS cache
• setting fixed low Disk Cache values ( 128MiB 400MiB)
• switching back to default Connection Limits
• compiling qBittorrent 4.2.5 with libtorrent 1.2.7
• using Alpine linux 3.11.6 and 3.12.0 (latest as of writing)

I've run into this issue with different setups:

• Alpine Linux with qbittorrent-nox from the official repositories
• Alpine Linux with qbittorrent-nox static binary from here

I have a fixed disk cache size of 128MiB, during the test I downloaded and seeded roughly 20(+-5) torrents at a capped speed of 600KiBs for upload and download each (about a third of my upload bandwidth).

Alpine Linux + qBittorrent from Alpine repositories

• qbittorrent 4.2.5-r0 (alpine repos)
• libtorrent-rasterbar 1.2.7-r0 (alpine repos)
• qt5-qtbase 5.14.2-r0 (alpine repos)
• boost-dev 1.72.0-r6 (alpine repos)

memory usage

1st run

run

2nd run

massif log

massif.out.zip

Alpine Linux + statically compiled qBittorrent with musl

memory usage

Massif log

could not generate one, the output file is "empty", tried twice, no success probably because this is a static binary:

desc: (none)         
cmd: qbittorrent-nox 
time_unit: i         
#-----------         
snapshot=0           
#-----------         
time=0               
mem_heap_B=0         
mem_heap_extra_B=0   
mem_stacks_B=0       
heap_tree=empty 

[ghost]

@arvidn I'm facing a similar issue where qBt process exit stalls when a particular HTTPS tracker is used. But when changed to HTTP tracker the problem goes away. Seems like the async shutdown timer hasn't fixed the delayed exit issue yet.

[arvidn]

Is it possible that this issue only happens when an HTTPS tracker responds with a redirect? It seems odd that a tracker would redirect an announce, but there is some special case logic in libtorrent in that case, that could be a hint of what's causing the leak.

[xavier2k6]

Is it possible that this issue only happens when an HTTPS tracker responds with a redirect? It seems odd that a tracker would redirect an announce, but there is some special case logic in libtorrent in that case, that could be a hint of what's causing the leak.

Good point - we would need to identify that HTTPS tracker (private/public)

@an0n666 can you shed some light on the HTTPS tracker?

[FranciscoPombal]

@arvidn

Is it possible that this issue only happens when an HTTPS tracker responds with a redirect? It seems odd that a tracker would redirect an announce, but there is some special case logic in libtorrent in that case, that could be a hint of what's causing the leak.

What kind of special logic? Could it be a problem with not limiting the number of redirects, for example?

Also, could cloudflare and the like be an issue?

[FranciscoPombal]

It would be helpful for reproducibility to know exactly _which_ HTTPS tracker(s) causes this. @an0n666 @joy4eg @farnoy have any of you managed to narrow it down to a(some) specific tracker(s)? You can mention them, even if they are private, as there is a possibility that others like me are members as well.

[arvidn]

could someone try this patch against libtorrent? https://github.com/arvidn/libtorrent/pull/4793

[ghost]

Is it possible that this issue only happens when an HTTPS tracker responds with a redirect? It seems odd that a tracker would redirect an announce, but there is some special case logic in libtorrent in that case, that could be a hint of what's causing the leak.

Not seeing any redirects in the headers.

[ghost]

Donno if relevant but I contacted the tracker admin and they changed something and now my delayed exit issue is gone. From the headers I can see they replaced chunked encoding with content length header. Though I’m not sure how that could’ve been causing an issue. From the response headers I could figure it’s running nginx.

[FranciscoPombal]

@arvidn ping

From the headers I can see they replaced chunked encoding with content length header.

^another potential clue.

[eigerzoom]

So I have discovered all the memory usage in my client is the responsibility of IPTorrents. This is likely intentional as they have been found to be quite shady in the past, or it could just be a coding issue that popped up with a libtorrent update. Once I stopped using the IPT trackers, namely https://ssl.empirehost.me/, https://routing.bgp.technology, and https://localhost.stackoverflow.tech my memory usage in qb instantly returned to around 90mb, the level of memory usage that is typical of qb in the past. It could just be one of those three trackers, I did not perform extensive testing to narrow in if it happens to be one or all of their trackers or not. I will also be reporting the issue on their forums to try and gain a response.

[eigerzoom]

After reporting the issue on IPT they immediately terminated my account....go figure
Please be aware there are a few trackers run by the same people at IPT.
Scenetime.com is the other one that I know of.

[FranciscoPombal]

@eigerzoom

After reporting the issue on IPT they immediately terminated my account....go figure
Please be aware there are a few trackers run by the same people at IPT.
Scenetime.com is the other one that I know of.

Thank you for your efforts and PSA. Fucking piece of shit admins from a shit-tier tracker. You're better off without them dude.

Hopefully someone can narrow this down to a public tracker (or private, if the admins are not stupid, and are willing to potentially supply the libtorrent developer an account for testing) that triggers this issue. It will be much easier to solve this once this is narrowed down to a(some) specific tracker(s), so that everyone can focus on the HTTPS messages sent back and forth and the problematic code paths they trigger in libtorrent.

[eigerzoom]

strong words, but fairly accurate.

I have moved a large percent of the torrents that were active on IPT to another private scene tracker (with largely the same content) that also uses SSL, verified all the files, and my memory has remained hovering around 90-125mb for qb (set to Auto)

Hopefully more people from this thread are willing to come forward and confirm if they also use IPT or their related sites.
But i understand if they dont.
To be clear I did link this thread and my original thread of the memory issue back on Jan 4th 2020 in the IPT forum.
I'm sure they deleted it, but they are aware of the issue being tracked here.
Thanks for everyones efforts to resolve and gain info on this issue.
If its intentional, then that is some high level coding at work to leverage the SSL transactions as a cryptominer.
I felt early on that it exibited cryptominer behaviour that I reported in the original thread, and I would like to apologize again for pointing my finger at the developers of qb.
Certainly it would probably take much to prove this behaviour, but I thought it obvious pretty early on.

[ghost]

After reporting the issue on IPT they immediately terminated my account....go figure
Please be aware there are a few trackers run by the same people at IPT.
Scenetime.com is the other one that I know of.

“Seeing as you think we are shady let us just confirm that for you”

Posted by a mod on your thread xD

[ghost]

@eigerzoom
Can you please share the url of the other https tracker as well which doesn’t cause a leak?
I just want to check what they’re doing differently..

[eigerzoom]

is there a private method of communication, i'd prefer my new home to remain relatively anonymous for now
maybe IRC, discord or some other method

[ghost]

The IPT tracker is using Nginx (probably for ssl offloading as a reverse proxy).

If you do not request with connection keep-alive header, nginx closes the connection less gracefully, maybe not giving the time to SSL for a complete shutdown.

And apparently it’s not sending close notify alert because majority of the browsers don’t use it:

/*
 * The majority of browsers do not send the "close notify" alert.
 * Among them are MSIE, old Mozilla, Netscape 4, Konqueror,
 * and Links.  And what is more, MSIE ignores the server's alert.
 *
 * Opera and recent Mozilla send the alert.
 */

c->ssl->no_wait_shutdown = 1;

More details here:

https://trac.nginx.org/nginx/ticket/1270
https://phabricator.wikimedia.org/T163674

[FranciscoPombal]

@an0n666

Can you please share the url of the other https tracker as well which doesn’t cause a leak?

FYI, newtrackon maintains a quite comprehensive list of stable public trackers:

https://newtrackon.com/list

Here are the listed HTTPS trackers, at the time of writing:

https://tracker.nanoha.org:443/announce
https://tracker.tamersunion.org:443/announce
https://w.wwwww.wtf:443/announce
https://tracker.imgoingto.icu:443/announce
https://tracker.nitrix.me:443/announce
https://trakx.herokuapp.com:443/announce
https://tracker.coalition.space:443/announce
https://tr.ready4.icu:443/announce
https://1337.abcvg.info:443/announce
https://tracker.hama3.net:443/announce

IPv6 only:
https://tracker6.lelux.fi:443/announce

I'd say it is extremely unlikely that _all of them_ also exhibit the problematic behavior that causes the memory leak, so you can use one of them for comparison with IPT's. The tracker for the comparison here need not be another private tracker.

I assume you have an account at IPT and are able to examine the traffic resulting from announces to their tracker, and compare them to a "known good" one?

[eigerzoom]

this is my source for current active public trackers
the lists run deep and he provides many formats, including IP only which is nice.
https://github.com/ngosang/trackerslist

[ghost]

@FranciscoPombal
The only difference is how the connection is being closed by the tracker.

Some SSL trackers are running behind Nginx proxy and they’re closing the connection immediately after the http response ends, when they see the “Connection: Close” header sent by qBt.

If qBt wants to perform a graceful ssl shutdown, which even majority of the browsers don’t bother doing, then it’ll have to use “Connection: Keep-alive” header so that it gets time to perform graceful shutdown with the tracker server.

The other solution is to not use graceful shutdown at all.

[FranciscoPombal]

@arvidn ping, could this be it? https://github.com/qbittorrent/qBittorrent/issues/12326#issuecomment-660573675

[AndCycle]

the connection related leakage gives me an idea to workaround the problem by using proxy server,
there is no more leak after I setup a squid server that can be used by qbittorrent.

just a temporary solution until this get fixed.

[ghost]

Public tracker behind nginx(for anyone to test/reproduce the leak)

https://tracker.sloppyta.co:443/announce

[arvidn]

I cannot reproduce a leak on linux with client_test and that tracker. can anyone else? could the version of asio or openssl make a difference?

[ghost]

@arvidn
Test the ipt tracker as well. Even though it’s a private tracker, the ssl part will still work even without a passkey.
https://ssl.empirehost.me/announce

The only difference between IPT tracker and the one you tested is, IPT nginx is using chunked encoding whereas the other one uses content length headers.

As for library version qbt is using 1.72 and 1.1.1g respectively for asio and openssl.

[Cqoicebordel]

Just FYI, I confirm that removing the https trackers stabilize the RAM usage (which, for me, with 300+ torrents and all trackers used, went above 15GB+ in less than 24h).

But as a side note, changing all those trackers was a pain, so I filed #13198.

[arvidn]

has anyone been able to test this patch https://github.com/arvidn/libtorrent/pull/4793

[xavier2k6]

has anyone been able to test this patch arvidn/libtorrent#4793

I'll compile a windows x64 build with this patch included later this evening & post it here, can users experiencing this issue - test & report back then please.

[FranciscoPombal]

@arvidn

Public tracker behind nginx(for anyone to test/reproduce the leak)

https://tracker.sloppyta.co:443/announce

I cannot reproduce a leak on linux with client_test and that tracker. can anyone else? could the version of asio or openssl make a difference?

I also cannot seem to reproduce the leak in qBittorrent. I added 300 hundred test torrents with only this tracker and waited for about 10 or 15 announces (so a few hours), then did about 10 manual force announces before stopping all torrents and closing cleanly.

Ubuntu 18.04 latest updates at time of writing
OpenSSL 1.1.1-1ubuntu2.1~18.04.6 
Boost 1.73.0
libtorrent d0bde6cf5ca63fac445eba688b8800b9ee977b55
qBittorrent 3cf8626317eaf442d8c9c77d0ca34270e3f12e64

The latter 3 compiled with gcc 8.4.0-1ubuntu1~18.04.

I did not observe the leak or the allocation chains similar to the previously posted captures. Perhaps I have to run it for even longer? Or is it possible that whatever the issue was, it has been fixed in the meantime, or simply by using a more recent boost version?

@an0n666

Can you reproduce the leak, either with the latest qBittorrent or libtorrent's client_test? If so, how? Can you tell if it depend on an older/newer version of boost/libtorrent/OpenSSL? Any chance you could post more data such as a massif capture, maybe even callgrind and/or a wireshark capture of the offending traffic?

[FranciscoPombal]

has anyone been able to test this patch arvidn/libtorrent#4793

I did not try it yet since I haven't been able to reproduce the issue even without it so far.

[ghost]

Can you reproduce the leak

Unfortunately no. The issue seem to be affecting random users. Maybe it requires to run the test for extended amount of time with lots of torrents and possibly a 3rd factor like unstable network or something which actually triggers the leak during the extended runtime.

[ghost]

Also try the IPT tracker as someone confirmed an issue with that one. The link that I provided is the only public ssl tracker I could find running nginx. Maybe someone else can share more public trackers that are confirmed to be causing a leak.

[FranciscoPombal]

Maybe it requires to run the test for extended amount of time with lots of torrents and possibly a 3rd factor like unstable network or something which actually triggers the leak during the extended runtime.

In terms of amount of torrents and runtime, "300+" (https://github.com/qbittorrent/qBittorrent/issues/12326#issuecomment-664390338) over a few hours should be enough to observe at least _something_, considering that in less than 24 hours a leak of 15+ GiB can be observed.

This unknown factor that determines causes some users to experience the leak and not others continues to elude us.

[arvidn]

I'm suspecting that version of openssl may be relevant

[xavier2k6]

I'm suspecting that version of openssl may be relevant

Official qBittorrent "Windows OS" has been using OpenSSL 1.1.1 series since around 4.1.6/7 (I believe)

These are the current library versions used on "official" 64 bit qBittorrent "Windows OS" releases (haven't verified 32 bit versions, but suspect they're the same)

qBittorrent x64|4.2.0|4.2.1|4.2.2|4.2.3|4.2.4|4.2.5|
---------------|-----|-----|-----|-----|-----|-----|
Qt|5.13.2|5.13.2|5.14.1|5.13.2|5.13.2|5.13.2|
Libtorrent|1.2.2.0|1.2.3.0|1.2.5.0|1.2.5.0|1.2.6.0|1.2.6.0|
Boost|1.71.0|1.71.0|1.72.0|1.72.0|1.72.0|1.72.0|
OpenSSL|1.1.1d|1.1.1d|1.1.1e|1.1.1f|1.1.1g|1.1.1g|
zlib|1.2.11|1.2.11|1.2.11|1.2.11|1.2.11|1.2.11|

[AndCycle]

has anyone been able to test this patch arvidn/libtorrent#4793

just put it under test on my linux which works fine, memory usage is stable for 4 days, so far so good.
false, can't reliable reproduce the memory leak scenario.
another edit, this take days to reproduce.

[ghost]

just put it under test on my linux which works fine, memory usage is stable for 4 days, so far so good.

And memory leak is reproducible without that patch?

[xavier2k6]

Have managed to detect 2 leaks with a program called deleaker using below build:

Windows test build of 4.3.0(Alpha1) with listed libraries:

qBittorrent master   | 4.3.0 +git 4d1c5a8
libtorrent-rasterbar | 1.2.8 +git 6b7b624
Qt                   | 5.15.0
OpenSSL              | 1.1.1g
zlib                 | 1.2.11
Boost                | 1.73

Stack trace of leak No. 1

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!EVP_CipherInit_ex Line 128 + 0xd bytes 00007ff64546c9fd
qbittorrent.exe!ctr_update Line 271 + 0x27 bytes 00007ff6454d6697
qbittorrent.exe!drbg_ctr_generate Line 346 + 0x1b bytes 00007ff6454d695b
qbittorrent.exe!RAND_DRBG_generate Line 641 + 0x18 bytes 00007ff645491e2f
qbittorrent.exe!RAND_DRBG_bytes Line 682 + 0x1d bytes 00007ff645491f71
qbittorrent.exe!libtorrent::dht::generate_random_id Line 157 + 0xd bytes 00007ff6456f49d1
qbittorrent.exe!libtorrent::dht::node::send_single_refresh Line 684 + 0x96 bytes 00007ff6456dd116
qbittorrent.exe!libtorrent::dht::dht_tracker::refresh_timeout Line 259 + 0x13b bytes 00007ff6456b4c90
qbittorrent.exe!boost::asio::detail::wait_handler<std::_Binder<std::_Unforced,void (__cdecl libtorrent::http_connection::*)(boost::system::error_code const &),std::shared_ptr<libtorrent::http_connection>,std::_Ph<1> const &>,boost::asio::detail::io_object_executor<boost: Line 73 + 0x6 bytes 00007ff645687c8c
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

Stack trace of leak No. 2

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!EVP_CipherInit_ex Line 128 + 0xd bytes 00007ff64546c9fd
qbittorrent.exe!ctr_update Line 271 + 0x27 bytes 00007ff6454d6697
qbittorrent.exe!drbg_ctr_generate Line 346 + 0x1b bytes 00007ff6454d695b
qbittorrent.exe!RAND_DRBG_generate Line 641 + 0x18 bytes 00007ff645491e2f
qbittorrent.exe!RAND_DRBG_bytes Line 682 + 0x1d bytes 00007ff645491f71
qbittorrent.exe!ecx_key_op Line 86 + 0x29 bytes 00007ff64548203a
qbittorrent.exe!pkey_ecx_keygen Line 654 + 0x21 bytes 00007ff64548366e
qbittorrent.exe!ssl_generate_pkey_group Line 4732 + 0x50 bytes 00007ff64617c456
qbittorrent.exe!add_key_share Line 602 + 0x3 bytes 00007ff64619ce91
qbittorrent.exe!tls_construct_ctos_key_share Line 688 + 0xa bytes 00007ff64619d389
qbittorrent.exe!tls_construct_extensions Line 853 + 0x1b bytes 00007ff64617e419
qbittorrent.exe!tls_construct_client_hello Line 1287 + 0x18 bytes 00007ff64618cd6e
qbittorrent.exe!write_state_machine Line 843 + 0x10 bytes 00007ff6461533a6
qbittorrent.exe!state_machine Line 443 + 0x3 bytes 00007ff646152857
qbittorrent.exe!SSL_do_handshake Line 3661 + 0x3 bytes 00007ff646144196
qbittorrent.exe!boost::asio::ssl::detail::engine::perform Line 235 + 0x9 bytes 00007ff645600547
qbittorrent.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor>,boost::asio::ssl::detail::handshake_op,std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::as Line 145 + 0x2b bytes 00007ff64562a2c9
qbittorrent.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::async_handshake<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::ex Line 444 + 0xa1 bytes 00007ff64561f59e
qbittorrent.exe!libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::connected Line 306 + 0x39 bytes 00007ff645649cbb
qbittorrent.exe!boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared Line 69 + 0x3a bytes 00007ff645656e56
qbittorrent.exe!boost::asio::detail::win_iocp_socket_connect_op<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared_ptr<std::func Line 114 + 0xd bytes 00007ff64564fd6e
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

Pics & XML file:

Snapshot #1 XML file.zip

[xavier2k6]

may or may not be relevant:
https://github.com/openssl/openssl/issues/9416

[xavier2k6]

@FranciscoPombal @arvidn ping - stack traces may be of some use. leak no.1 seems to happen on qBittorrent startup.....

[arvidn]

It sounds like some people define "memory leak" as any meory allocated and not freed. I don't think that's a reasonable definition. A singleton that's allocated exactly once and then never freed is not a leak. It's freed by the operating system on process shutdown.

This does not smell like a leak to me. Has anyone seen multiple instances leak? or even better, leaking over time, where longer wait means more allocated objects.

[xavier2k6]

9 more instances over a 12 hour period.

click to view stack trace

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!rand_drbg_new Line 191 00007ff6454911b5
qbittorrent.exe!RAND_DRBG_get0_private Line 1144 00007ff645492762
qbittorrent.exe!RAND_priv_bytes Line 927 00007ff645479593
qbittorrent.exe!ecx_key_op Line 86 + 0x29 bytes 00007ff64548203a
qbittorrent.exe!pkey_ecx_keygen Line 654 + 0x21 bytes 00007ff64548366e
qbittorrent.exe!ssl_generate_pkey_group Line 4732 + 0x50 bytes 00007ff64617c456
qbittorrent.exe!add_key_share Line 602 + 0x3 bytes 00007ff64619ce91
qbittorrent.exe!tls_construct_ctos_key_share Line 688 + 0xa bytes 00007ff64619d389
qbittorrent.exe!tls_construct_extensions Line 853 + 0x1b bytes 00007ff64617e419
qbittorrent.exe!tls_construct_client_hello Line 1287 + 0x18 bytes 00007ff64618cd6e
qbittorrent.exe!write_state_machine Line 843 + 0x10 bytes 00007ff6461533a6
qbittorrent.exe!state_machine Line 443 + 0x3 bytes 00007ff646152857
qbittorrent.exe!SSL_do_handshake Line 3661 + 0x3 bytes 00007ff646144196
qbittorrent.exe!boost::asio::ssl::detail::engine::perform Line 235 + 0x9 bytes 00007ff645600547
qbittorrent.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor>,boost::asio::ssl::detail::handshake_op,std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::as Line 145 + 0x2b bytes 00007ff64562a2c9
qbittorrent.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::async_handshake<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::ex Line 444 + 0xa1 bytes 00007ff64561f59e
qbittorrent.exe!libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::connected Line 306 + 0x39 bytes 00007ff645649cbb
qbittorrent.exe!boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared Line 69 + 0x3a bytes 00007ff645656e56
qbittorrent.exe!boost::asio::detail::win_iocp_socket_connect_op<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared_ptr<std::func Line 114 + 0xd bytes 00007ff64564fd6e
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

click to view stack trace

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!EVP_CipherInit_ex Line 128 + 0xd bytes 00007ff64546c9fd
qbittorrent.exe!drbg_ctr_init Line 413 + 0x21 bytes 00007ff6454d6b42
qbittorrent.exe!RAND_DRBG_set Line 137 + 0x3 bytes 00007ff645491100
qbittorrent.exe!rand_drbg_new Line 225 + 0x31 bytes 00007ff645491242
qbittorrent.exe!RAND_DRBG_get0_private Line 1144 00007ff645492762
qbittorrent.exe!RAND_priv_bytes Line 927 00007ff645479593
qbittorrent.exe!ecx_key_op Line 86 + 0x29 bytes 00007ff64548203a
qbittorrent.exe!pkey_ecx_keygen Line 654 + 0x21 bytes 00007ff64548366e
qbittorrent.exe!ssl_generate_pkey_group Line 4732 + 0x50 bytes 00007ff64617c456
qbittorrent.exe!add_key_share Line 602 + 0x3 bytes 00007ff64619ce91
qbittorrent.exe!tls_construct_ctos_key_share Line 688 + 0xa bytes 00007ff64619d389
qbittorrent.exe!tls_construct_extensions Line 853 + 0x1b bytes 00007ff64617e419
qbittorrent.exe!tls_construct_client_hello Line 1287 + 0x18 bytes 00007ff64618cd6e
qbittorrent.exe!write_state_machine Line 843 + 0x10 bytes 00007ff6461533a6
qbittorrent.exe!state_machine Line 443 + 0x3 bytes 00007ff646152857
qbittorrent.exe!SSL_do_handshake Line 3661 + 0x3 bytes 00007ff646144196
qbittorrent.exe!boost::asio::ssl::detail::engine::perform Line 235 + 0x9 bytes 00007ff645600547
qbittorrent.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor>,boost::asio::ssl::detail::handshake_op,std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::as Line 145 + 0x2b bytes 00007ff64562a2c9
qbittorrent.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::async_handshake<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::ex Line 444 + 0xa1 bytes 00007ff64561f59e
qbittorrent.exe!libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::connected Line 306 + 0x39 bytes 00007ff645649cbb
qbittorrent.exe!boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared Line 69 + 0x3a bytes 00007ff645656e56
qbittorrent.exe!boost::asio::detail::win_iocp_socket_connect_op<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared_ptr<std::func Line 114 + 0xd bytes 00007ff64564fd6e
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

click to view stack trace

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!EVP_CipherInit_ex Line 128 + 0xd bytes 00007ff64546c9fd
qbittorrent.exe!ctr_update Line 271 + 0x27 bytes 00007ff6454d6697
qbittorrent.exe!drbg_ctr_generate Line 346 + 0x1b bytes 00007ff6454d695b
qbittorrent.exe!RAND_DRBG_generate Line 641 + 0x18 bytes 00007ff645491e2f
qbittorrent.exe!RAND_DRBG_bytes Line 682 + 0x1d bytes 00007ff645491f71
qbittorrent.exe!libtorrent::dht::generate_random_id Line 157 + 0xd bytes 00007ff6456f49d1
qbittorrent.exe!libtorrent::dht::node::send_single_refresh Line 684 + 0x96 bytes 00007ff6456dd116
qbittorrent.exe!libtorrent::dht::dht_tracker::refresh_timeout Line 259 + 0x13b bytes 00007ff6456b4c90
qbittorrent.exe!boost::asio::detail::wait_handler<std::_Binder<std::_Unforced,void (__cdecl libtorrent::http_connection::*)(boost::system::error_code const &),std::shared_ptr<libtorrent::http_connection>,std::_Ph<1> const &>,boost::asio::detail::io_object_executor<boost: Line 73 + 0x6 bytes 00007ff645687c8c
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

click to view stack trace

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!EVP_CipherInit_ex Line 128 + 0xd bytes 00007ff64546c9fd
qbittorrent.exe!ctr_update Line 271 + 0x27 bytes 00007ff6454d6697
qbittorrent.exe!drbg_ctr_generate Line 346 + 0x1b bytes 00007ff6454d695b
qbittorrent.exe!RAND_DRBG_generate Line 641 + 0x18 bytes 00007ff645491e2f
qbittorrent.exe!RAND_DRBG_bytes Line 682 + 0x1d bytes 00007ff645491f71
qbittorrent.exe!ecx_key_op Line 86 + 0x29 bytes 00007ff64548203a
qbittorrent.exe!pkey_ecx_keygen Line 654 + 0x21 bytes 00007ff64548366e
qbittorrent.exe!ssl_generate_pkey_group Line 4732 + 0x50 bytes 00007ff64617c456
qbittorrent.exe!add_key_share Line 602 + 0x3 bytes 00007ff64619ce91
qbittorrent.exe!tls_construct_ctos_key_share Line 688 + 0xa bytes 00007ff64619d389
qbittorrent.exe!tls_construct_extensions Line 853 + 0x1b bytes 00007ff64617e419
qbittorrent.exe!tls_construct_client_hello Line 1287 + 0x18 bytes 00007ff64618cd6e
qbittorrent.exe!write_state_machine Line 843 + 0x10 bytes 00007ff6461533a6
qbittorrent.exe!state_machine Line 443 + 0x3 bytes 00007ff646152857
qbittorrent.exe!SSL_do_handshake Line 3661 + 0x3 bytes 00007ff646144196
qbittorrent.exe!boost::asio::ssl::detail::engine::perform Line 235 + 0x9 bytes 00007ff645600547
qbittorrent.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor>,boost::asio::ssl::detail::handshake_op,std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::as Line 145 + 0x2b bytes 00007ff64562a2c9
qbittorrent.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::async_handshake<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::ex Line 444 + 0xa1 bytes 00007ff64561f59e
qbittorrent.exe!libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::connected Line 306 + 0x39 bytes 00007ff645649cbb
qbittorrent.exe!boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared Line 69 + 0x3a bytes 00007ff645656e56
qbittorrent.exe!boost::asio::detail::win_iocp_socket_connect_op<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared_ptr<std::func Line 114 + 0xd bytes 00007ff64564fd6e
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

click to view stack trace

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!rand_pool_new Line 457 + 0xe bytes 00007ff645478ea5
qbittorrent.exe!RAND_DRBG_bytes Line 670 + 0xb bytes 00007ff645491edc
qbittorrent.exe!ecx_key_op Line 86 + 0x29 bytes 00007ff64548203a
qbittorrent.exe!pkey_ecx_keygen Line 654 + 0x21 bytes 00007ff64548366e
qbittorrent.exe!ssl_generate_pkey_group Line 4732 + 0x50 bytes 00007ff64617c456
qbittorrent.exe!add_key_share Line 602 + 0x3 bytes 00007ff64619ce91
qbittorrent.exe!tls_construct_ctos_key_share Line 688 + 0xa bytes 00007ff64619d389
qbittorrent.exe!tls_construct_extensions Line 853 + 0x1b bytes 00007ff64617e419
qbittorrent.exe!tls_construct_client_hello Line 1287 + 0x18 bytes 00007ff64618cd6e
qbittorrent.exe!write_state_machine Line 843 + 0x10 bytes 00007ff6461533a6
qbittorrent.exe!state_machine Line 443 + 0x3 bytes 00007ff646152857
qbittorrent.exe!SSL_do_handshake Line 3661 + 0x3 bytes 00007ff646144196
qbittorrent.exe!boost::asio::ssl::detail::engine::perform Line 235 + 0x9 bytes 00007ff645600547
qbittorrent.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor>,boost::asio::ssl::detail::handshake_op,std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::as Line 145 + 0x2b bytes 00007ff64562a2c9
qbittorrent.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::async_handshake<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::ex Line 444 + 0xa1 bytes 00007ff64561f59e
qbittorrent.exe!libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::connected Line 306 + 0x39 bytes 00007ff645649cbb
qbittorrent.exe!boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared Line 69 + 0x3a bytes 00007ff645656e56
qbittorrent.exe!boost::asio::detail::win_iocp_socket_connect_op<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared_ptr<std::func Line 114 + 0xd bytes 00007ff64564fd6e
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

click to view stack trace

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!EVP_CIPHER_CTX_new Line 42 + 0xf bytes 00007ff64546c85e
qbittorrent.exe!drbg_ctr_init Line 409 00007ff6454d6b10
qbittorrent.exe!RAND_DRBG_set Line 137 + 0x3 bytes 00007ff645491100
qbittorrent.exe!rand_drbg_new Line 225 + 0x31 bytes 00007ff645491242
qbittorrent.exe!RAND_DRBG_get0_private Line 1144 00007ff645492762
qbittorrent.exe!RAND_priv_bytes Line 927 00007ff645479593
qbittorrent.exe!ecx_key_op Line 86 + 0x29 bytes 00007ff64548203a
qbittorrent.exe!pkey_ecx_keygen Line 654 + 0x21 bytes 00007ff64548366e
qbittorrent.exe!ssl_generate_pkey_group Line 4732 + 0x50 bytes 00007ff64617c456
qbittorrent.exe!add_key_share Line 602 + 0x3 bytes 00007ff64619ce91
qbittorrent.exe!tls_construct_ctos_key_share Line 688 + 0xa bytes 00007ff64619d389
qbittorrent.exe!tls_construct_extensions Line 853 + 0x1b bytes 00007ff64617e419
qbittorrent.exe!tls_construct_client_hello Line 1287 + 0x18 bytes 00007ff64618cd6e
qbittorrent.exe!write_state_machine Line 843 + 0x10 bytes 00007ff6461533a6
qbittorrent.exe!state_machine Line 443 + 0x3 bytes 00007ff646152857
qbittorrent.exe!SSL_do_handshake Line 3661 + 0x3 bytes 00007ff646144196
qbittorrent.exe!boost::asio::ssl::detail::engine::perform Line 235 + 0x9 bytes 00007ff645600547
qbittorrent.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor>,boost::asio::ssl::detail::handshake_op,std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::as Line 145 + 0x2b bytes 00007ff64562a2c9
qbittorrent.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::async_handshake<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::ex Line 444 + 0xa1 bytes 00007ff64561f59e
qbittorrent.exe!libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::connected Line 306 + 0x39 bytes 00007ff645649cbb
qbittorrent.exe!boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared Line 69 + 0x3a bytes 00007ff645656e56
qbittorrent.exe!boost::asio::detail::win_iocp_socket_connect_op<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared_ptr<std::func Line 114 + 0xd bytes 00007ff64564fd6e
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

click to view stack trace

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!ERR_get_state Line 734 + 0xb bytes 00007ff64546b2ec
qbittorrent.exe!ERR_clear_error Line 445 00007ff64546a7df
qbittorrent.exe!boost::asio::ssl::detail::engine::perform Line 234 00007ff645600539
qbittorrent.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor>,boost::asio::ssl::detail::handshake_op,std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::as Line 145 + 0x2b bytes 00007ff64562a2c9
qbittorrent.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::async_handshake<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::ex Line 444 + 0xa1 bytes 00007ff64561f59e
qbittorrent.exe!libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::connected Line 306 + 0x39 bytes 00007ff645649cbb
qbittorrent.exe!boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared Line 69 + 0x3a bytes 00007ff645656e56
qbittorrent.exe!boost::asio::detail::win_iocp_socket_connect_op<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared_ptr<std::func Line 114 + 0xd bytes 00007ff64564fd6e
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

click to view stack trace

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!drbg_ctr_init Line 393 + 0xb bytes 00007ff6454d6ab7
qbittorrent.exe!RAND_DRBG_set Line 137 + 0x3 bytes 00007ff645491100
qbittorrent.exe!rand_drbg_new Line 225 + 0x31 bytes 00007ff645491242
qbittorrent.exe!RAND_DRBG_get0_private Line 1144 00007ff645492762
qbittorrent.exe!RAND_priv_bytes Line 927 00007ff645479593
qbittorrent.exe!ecx_key_op Line 86 + 0x29 bytes 00007ff64548203a
qbittorrent.exe!pkey_ecx_keygen Line 654 + 0x21 bytes 00007ff64548366e
qbittorrent.exe!ssl_generate_pkey_group Line 4732 + 0x50 bytes 00007ff64617c456
qbittorrent.exe!add_key_share Line 602 + 0x3 bytes 00007ff64619ce91
qbittorrent.exe!tls_construct_ctos_key_share Line 688 + 0xa bytes 00007ff64619d389
qbittorrent.exe!tls_construct_extensions Line 853 + 0x1b bytes 00007ff64617e419
qbittorrent.exe!tls_construct_client_hello Line 1287 + 0x18 bytes 00007ff64618cd6e
qbittorrent.exe!write_state_machine Line 843 + 0x10 bytes 00007ff6461533a6
qbittorrent.exe!state_machine Line 443 + 0x3 bytes 00007ff646152857
qbittorrent.exe!SSL_do_handshake Line 3661 + 0x3 bytes 00007ff646144196
qbittorrent.exe!boost::asio::ssl::detail::engine::perform Line 235 + 0x9 bytes 00007ff645600547
qbittorrent.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor>,boost::asio::ssl::detail::handshake_op,std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::as Line 145 + 0x2b bytes 00007ff64562a2c9
qbittorrent.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::async_handshake<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::ex Line 444 + 0xa1 bytes 00007ff64561f59e
qbittorrent.exe!libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::connected Line 306 + 0x39 bytes 00007ff645649cbb
qbittorrent.exe!boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared Line 69 + 0x3a bytes 00007ff645656e56
qbittorrent.exe!boost::asio::detail::win_iocp_socket_connect_op<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared_ptr<std::func Line 114 + 0xd bytes 00007ff64564fd6e
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

click to view stack trace

ntdll.dll!RtlAllocateHeap 00007ff8079a6c80
qbittorrent.exe!_malloc_base Line 34 + 0xc bytes 00007ff6461f2306
qbittorrent.exe!rand_pool_new Line 437 00007ff645478e38
qbittorrent.exe!RAND_DRBG_bytes Line 670 + 0xb bytes 00007ff645491edc
qbittorrent.exe!ecx_key_op Line 86 + 0x29 bytes 00007ff64548203a
qbittorrent.exe!pkey_ecx_keygen Line 654 + 0x21 bytes 00007ff64548366e
qbittorrent.exe!ssl_generate_pkey_group Line 4732 + 0x50 bytes 00007ff64617c456
qbittorrent.exe!add_key_share Line 602 + 0x3 bytes 00007ff64619ce91
qbittorrent.exe!tls_construct_ctos_key_share Line 688 + 0xa bytes 00007ff64619d389
qbittorrent.exe!tls_construct_extensions Line 853 + 0x1b bytes 00007ff64617e419
qbittorrent.exe!tls_construct_client_hello Line 1287 + 0x18 bytes 00007ff64618cd6e
qbittorrent.exe!write_state_machine Line 843 + 0x10 bytes 00007ff6461533a6
qbittorrent.exe!state_machine Line 443 + 0x3 bytes 00007ff646152857
qbittorrent.exe!SSL_do_handshake Line 3661 + 0x3 bytes 00007ff646144196
qbittorrent.exe!boost::asio::ssl::detail::engine::perform Line 235 + 0x9 bytes 00007ff645600547
qbittorrent.exe!boost::asio::ssl::detail::io_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor>,boost::asio::ssl::detail::handshake_op,std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::as Line 145 + 0x2b bytes 00007ff64562a2c9
qbittorrent.exe!boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::async_handshake<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::ex Line 444 + 0xa1 bytes 00007ff64561f59e
qbittorrent.exe!libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::connected Line 306 + 0x39 bytes 00007ff645649cbb
qbittorrent.exe!boost::asio::asio_handler_invoke<boost::asio::detail::binder1<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared Line 69 + 0x3a bytes 00007ff645656e56
qbittorrent.exe!boost::asio::detail::win_iocp_socket_connect_op<std::_Binder<std::_Unforced,void (__cdecl libtorrent::ssl_stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp,boost::asio::executor> >::*)(boost::system::error_code const &,std::shared_ptr<std::func Line 114 + 0xd bytes 00007ff64564fd6e
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::do_one Line 460 + 0xe bytes 00007ff645595da5
qbittorrent.exe!boost::asio::detail::win_iocp_io_context::run Line 203 + 0x27 bytes 00007ff645595945
qbittorrent.exe!std::thread::_Invoke<std::tuple<<lambda_f47be1f771a9090cdae387aa12b4d0a6> >,0> Line 43 + 0x29 bytes 00007ff645597a5b
qbittorrent.exe!thread_start<unsigned int (__cdecl*)(void *),1> Line 97 + 0x7 bytes 00007ff6461e6bda
kernel32.dll!BaseThreadInitThunk + 0xe bytes 00007ff8078a6fce
ntdll.dll!RtlUserThreadStart + 0x1b bytes 00007ff8079dcebb

[FranciscoPombal]

@xavier2k6 Thanks for this. Seems there are important symbols missing from the stacktraces. For example, what are the libtorrent functions that the leaks originate from? I can only see this information in one of the stacktraces (3rd one from the top in your latest comment), where it seems to be related to the DHT. In all the other ones, how can you tell the issue comes from tracker connections?

[AndCycle]

just put it under test on my linux which works fine, memory usage is stable for 4 days, so far so good.

And memory leak is reproducible without that patch?

tooks me 4 days to verify that the leak do happened without patch on my setup,
when I wanna switch on the patch for another round, libtorrent 1.2.8 released.
so I took the 1.2.8 for another shot, which have no sign of significant memory usage for a week by far.

[ghost]

just put it under test on my linux which works fine, memory usage is stable for 4 days, so far so good.

And memory leak is reproducible without that patch?

tooks me 4 days to verify that the leak do happened without patch on my setup,
when I wanna switch on the patch for another round, libtorrent 1.2.8 released.
so I took the 1.2.8 for another shot, which have no sign of significant memory usage for a week by far.

This was also confirmed by another user in a related ticket https://github.com/qbittorrent/qBittorrent/issues/12825#issuecomment-671676613

[mayli]

Is there a theoretical estimation of how much memory qbittorrent should use under default settings? My current setup is using 2487M (Total buffer size: 511.0 MiB) for seeding 3565 torrents. Averaging 500kB for each torrent, does that consider as high memory usage?

[ghost]

@FranciscoPombal
I think at this point we should wait for the next update before moving on with this.
There’s a good chance that the issue has already been resolved due to updated libraries(boost, libtorrent).

[FranciscoPombal]

Is there a theoretical estimation of how much memory qbittorrent should use under default settings? My current setup is using 2487M (Total buffer size: 511.0 MiB) for seeding 3565 torrents. Averaging 500kB for each torrent, does that consider as high memory usage?

Can't say for certain if this is a good/bad value per-torrent (at least it does not look unreasonable). If the total usage is stable and not blowing up like the reports of previous posts, then at least you're not affected by this leak.

[mayli]

libtorrent 1.2.9 is released, did anyone reproduce this issue with the new version?

[brauliobo]

@mayli same here, using 2.5gb of memory with Archlinux packages. BTW, still 1.2.8 for https://www.archlinux.org/packages/extra/x86_64/libtorrent-rasterbar/

[guillaumedsde]

libtorrent 1.2.9 is released, did anyone reproduce this issue with the new version?

I have not managed to reproduce the issue with recent versions of the dependencies (i.e: the memory leak seems to be solved).
I am using this static binary compiled using this script with qbttorrent 4.2.5 and the following dependencies:

After about 2 hours of use, the memory cache (2048MB in my case) has filled up as expected while seeding around 600 torrents (capped at 600KB/s upload) and downloading 255 torrents (capped at 500KB/s download). Once the cache finishes filling up, it looks like memory usage is relatively stable:

I will report back here if I notice a leak after a couple of days of use.

[FranciscoPombal]

@joy4eg
@farnoy

Mind testing with the latest commits of qBittorrent/libtorrent and possibly the latest releases of other dependencies as well? Ideally I think we would all like to get to the bottom of this, but if it go fixed somehow (e.g. possibly even due to some fix in boost or OpenSSL), we can close this and move on.

[FranciscoPombal]

tooks me 4 days to verify that the leak do happened without patch on my setup,
when I wanna switch on the patch for another round, libtorrent 1.2.8 released.
so I took the 1.2.8 for another shot, which have no sign of significant memory usage for a week by far.

This was also confirmed by another user in a related ticket #12825 (comment)

For anyone curious, this was the patch in question: https://github.com/arvidn/libtorrent/pull/4934. This _might_ have been it, after all.

[eli-schwartz]

@mayli same here, using 2.5gb of memory with Archlinux packages. BTW, still 1.2.8 for https://www.archlinux.org/packages/extra/x86_64/libtorrent-rasterbar/

1.2.9 is in testing, but results in https://bugs.archlinux.org/task/67754

[FranciscoPombal]

@eli-schwartz

The python bindings have a problem of their own, but otherwise the C++11/C++14 thing is only a problem when all of the following are true:

• You build libtorrent with C++11
• You build the client (e.g. qBittorrent) with > C++11
• (if building libtorrent with CMake) you don't pass -DCMAKE_CXX_STANDARD=11 when building libtorrent.
• Your client does not manually define TORRENT_CXX11_ABI in its compiler command line.

See https://github.com/arvidn/libtorrent/pull/5010/files for an explanation of the latter 2 points.

Keep in mind that while all of these points being true will result in the symbol look up error, doing the exact opposite for some or all of them will also lead to problems. For example, if you build libtorrent with C++14, your client should not define TORRENT_CXX11_ABI when building.

The easiest "winning combos" are:

• Just make sure both the client and libtorrent are built with C++14 or higher

or, if building libtorrent with C++11:

• When building libtorrent with CMake, make sure to pass -DCMAKE_CXX_STANDARD=11. This will cause TORRENT_CXX11_ABI to be defined both for libtorrent's build as well as a usage requirement that will automatically be propagated upstream to targets that link against libtorrent.

[eli-schwartz]

Yeah, I mentioned in the bug report that we need libtorrent built with the C++14 std. But we also need to support deluge with the python bindings...

We currently build it using the autotools configure script. Our libtorrent maintainer is going to take a look at resolving this tomorrow, the simplest solution is if there is a patch or configure parameter to guarantee that both libtorrent and the python bindings are built with C++14 using autotools.

[mayli]

@guillaumedsde I can confirm the new 1.2.9 libtorrent seems stabilized the RAM usage to normal level, as well as cpu usage.

3k seeding, 150 active, with 512MB Disk cache, used to be about 4G ram, and now it's stable at less than 900MB.

update: I am hitting segfault on high speed downloading about 90MB/s on the static musl build, the dmesg says

[928576.762706] qbittorrent-nox[22318]: segfault at 7efe297cdfc0 ip 00007efe2ac713a9 sp 00007efe297cdfc0 error 6 in qbittorrent-nox[7efe2a73c000+b59000]
[928576.762718] Code: 8b 46 68 45 31 d2 44 8d 41 ff 49 c1 e0 04 49 01 c0 48 8b 08 48 85 c9 0f 84 f4 00 00 00 f7 40 08 ff ff ff 7f 0f 85 e7 00 00 00 <49> 89 0c de 44 8d 5b 01 8b 5e 7c 41 83 c2 01 48 c7 00 00 00 00 00
[928666.072944] qbittorrent-nox[32570]: segfault at 7feb4b9403f0 ip 00007feb4d1c23a9 sp 00007feb4b9403f0 error 6 in qbittorrent-nox[7feb4cc8d000+b59000]
[928666.072952] Code: 8b 46 68 45 31 d2 44 8d 41 ff 49 c1 e0 04 49 01 c0 48 8b 08 48 85 c9 0f 84 f4 00 00 00 f7 40 08 ff ff ff 7f 0f 85 e7 00 00 00 <49> 89 0c de 44 8d 5b 01 8b 5e 7c 41 83 c2 01 48 c7 00 00 00 00 00
[928758.390470] qbittorrent-nox[8506]: segfault at 7f4f890f5a20 ip 00007f4f8a4453a9 sp 00007f4f890f5a20 error 6 in qbittorrent-nox[7f4f89f10000+b59000]
[928758.390476] Code: 8b 46 68 45 31 d2 44 8d 41 ff 49 c1 e0 04 49 01 c0 48 8b 08 48 85 c9 0f 84 f4 00 00 00 f7 40 08 ff ff ff 7f 0f 85 e7 00 00 00 <49> 89 0c de 44 8d 5b 01 8b 5e 7c 41 83 c2 01 48 c7 00 00 00 00 00
[928847.517746] qbittorrent-nox[17646]: segfault at 7f59d6749f50 ip 00007f59d78613a9 sp 00007f59d6749f50 error 6 in qbittorrent-nox[7f59d732c000+b59000]
[928847.517752] Code: 8b 46 68 45 31 d2 44 8d 41 ff 49 c1 e0 04 49 01 c0 48 8b 08 48 85 c9 0f 84 f4 00 00 00 f7 40 08 ff ff ff 7f 0f 85 e7 00 00 00 <49> 89 0c de 44 8d 5b 01 8b 5e 7c 41 83 c2 01 48 c7 00 00 00 00 00
[928935.255458] qbittorrent-nox[24470]: segfault at 7ff9faf6acb0 ip 00007ff9fc5af3a9 sp 00007ff9faf6acb0 error 6 in qbittorrent-nox[7ff9fc07a000+b59000]
[928935.255469] Code: 8b 46 68 45 31 d2 44 8d 41 ff 49 c1 e0 04 49 01 c0 48 8b 08 48 85 c9 0f 84 f4 00 00 00 f7 40 08 ff ff ff 7f 0f 85 e7 00 00 00 <49> 89 0c de 44 8d 5b 01 8b 5e 7c 41 83 c2 01 48 c7 00 00 00 00 00

my docker will automatically restart the process, but it keeps crashing for last 4 hours/

[FranciscoPombal]

update: I am hitting segfault on high speed downloading about 90MB/s on the static musl build, the dmesg says

@mayli are you using the build from here https://github.com/userdocs/qbittorrent-nox-static ?

[mayli]

@FranciscoPombal Yep.

[FranciscoPombal]

@userdocs Can you reproduce this? https://github.com/qbittorrent/qBittorrent/issues/12326#issuecomment-683533382

[userdocs]

The seg fault? no.

Using my latest musl build - https://github.com/userdocs/qbittorrent-nox-static/blob/master/bin/musl-static/amd64/qbittorrent-nox on Debian 10 Stable x64

Downloading Ubuntu at my almost max potential

I get 0 warning in the console or my log files.

(N) 2020-09-02T13:05:23 - 'ubuntu-20.04.1-desktop-amd64.iso' added to download list.
(N) 2020-09-02T13:05:59 - Successfully moved torrent: ubuntu-20.04.1-desktop-amd64.iso. New path: ...

Does not meant the fault is not occurring, i just can't reproduce it on my Debian server.

[slrslr]

Hi, not a dev. just user. In qbt 4.2.5 on Windows 10 (outdated OS) I see 16MB virtual memory used per resumed torrent

Maybe it is allright or already known issue, if not or is suspicious, let me know what i should try. Thx for improving qbt.

[FranciscoPombal]

@slrslr This excessive memory usage issue is believed to be fixed on recent builds that use libtorrent >= 1.2.8. It would be nice if you could confirm this as well: https://github.com/FranciscoPombal/qBittorrent/actions/runs/241080799 (these recent master builds have libtorrent 1.2.10)

[slrslr]

I have not know which of the files to download, so i have downloaded first one and it was qbt 4.3.0alpha1 (libtorrent 1.2.10.0). i think that you are right, i see no memory issue anymore so far.

Seeing big improvements from previous versions (After 4 hours of runtime on W10):
1) much faster startup (like 5 minutes for the torrents restoring process to complete - have 1800 added), data transfer speed is nice high (i would guess faster than previous versions)
2) it uploading in my case never unseen amount of torrents in parallel - in previous versions maybe 25, now around 150!!! (400 connected peers).
3) interface was responding well (except i have seen 2-3 times that when i not used qbt interface for some time and focus it, i had to wait 1 minute for it to start responding, not a big deal for me now)
4) memory usage is incredibly low 230MB physical and 500MB virtual
5) CPU usage seems reasonably low
🥇

After i suspend the computer and on awake it like 5-10 minutes utilized system drive and CPU thread (23% usage on quad core CPU), seems normal to me. Then it was back to normal as before sleep.

Back to the topic of this issue, so far memory issue seems to be fixed as @FranciscoPombal said.

[FranciscoPombal]

Since there have been no more complaints and multiple people on different platforms have reported good results, I think we can consider this to be addressed. So far it is believed this was the root cause: https://github.com/qbittorrent/qBittorrent/issues/12326#issuecomment-683319690

Users facing problems should try builds like https://github.com/FranciscoPombal/qBittorrent/actions/runs/241080799 or more recent - i.e., recent qBittorrent master branch + libtorrent >= 1.2.10 before posting new bug reports. The next official release will be built with libtorrent >= 1.2.10 and will thus include these fixes.

Thanks to everyone for their help and contributions!

@eli-schwartz

Yeah, I mentioned in the bug report that we need libtorrent built with the C++14 std. But we also need to support deluge with the python bindings...

We currently build it using the autotools configure script. Our libtorrent maintainer is going to take a look at resolving this tomorrow, the simplest solution is if there is a patch or configure parameter to guarantee that both libtorrent and the python bindings are built with C++14 using autotools.

If building libtorrent with C++11 and when building with CMake, passing -DCMAKE_CXX_STANDARD=11 will force it to be built with the TORRENT_CXX11_ABI compile definition and force that definition to be propagated to all targets that link against it. So you'll have to figure out how to do something similar with using autotools, or patch all packages that depend on libtorrent to pass TORRENT_CXX11_ABI to the compiler when building. The alternative is of course to fix the python bindings build with C++ >=14 and just use C++ >=14 to build libtorrent. We can track this in a separate issue if need be.