Sabnzbd: memory consumption

[pinn73]

Hi,

From a clean start, sab3 idles at around 250mb(v) &45mb (r). When downloading this peaks at approx 490/270; however, when sab is idle again rami usage is 280/63. While this is significantly better than previous iterations of Sab I wondered if this is to be expected or is there a leak?

Please see attached log
sabnzbd.log

[Safihre]

It's a good question. Only recently I did find a memory leak caused by a bad python 2 to 3 conversion, so it's definitely possible.

I will leave this issue open, so at some point I can really take my time and investigate what objects are left in memory after the queue finishes and if these are to be expected!

[sanderjo]

While this is significantly better than previous iterations of Sab I wondered if this is to be expected or is there a leak?

"better"? Better as in ... less memory usage, ... or more memory usage?

[pinn73]

Definitely less. I don't have numbers but I'd estimate 20% less usage. This is on debian btw

[Safihre]

I did a short analysis and found that when the queue is done, the last NZB stays in memory for some reason.. Even after garbage collection. But I don't know why. When a new NZB is added, the previous objects disappears.
I am getting strange errors when trying to find the references to these remaining objects. Maybe it's better when I switch to my Mac.

                       types |   # objects |   total size
============================ | =========== | ============
    sabnzbd.nzbstuff.NzbFile |          15 |      3.28 KB
    sabnzbd.nzbstuff.Article |          15 |      1.88 KB
  sabnzbd.nzbstuff.NzbObject |           1 |     64     B

Windows fatal exception: access violation

[Safihre]

Worked on this a bit and found multiple problems:
https://github.com/sabnzbd/sabnzbd/tree/bugfix/memusage

• The DirectUnpacker thread keeps a reference to the NzbObject, which keeps both in memory after the job is finished.
• In Assembler, PostProcessor and Decoder we use queue's that return tuples with one of the objects. These tuples stay alive until the next job comes along.

I fixed these things, but it still keeps the nzo alive until the next one really starts downloading. So I assume there's a reference to an Article (which in turn references a NzbFile, which references the NzbObject) in one of the Downloader objects. Maybe in the Downloader itself, or in the Newswrapper. Have to dig deeper to find this.. Thegc` tools available are very limited and I tried a couple of the packages on PyPi but they are made for small single script/thread use cases and break on SABnzbd 😢