Jackett: Leaking Memory?

[Yellow1144]

Hey there..
I just discovered Jackett abot a week ago and so far it has been working well. A few days ago i noticed my system getting sluggish took a look and noticed Jackett was using a huge amount of memory. I restarted the service and the memory was relased. I watched as the amount of RAM used increases by the second. after about 12 hours or so Jackett will be using about 10GB of RAM.
I use it with Medusa, (latest version). I am not sure if Jackett is the problem or Mono.

Jackett version 0.10.365.0
Mono version 5.16.0.179
System Linux-4.15.0-33-generic-x86_64-with-LinuxMint-18.3-sylvia

log.txt.20181027.00098-1.txt

---

[Jorman]

I think is a mono issue. In my system (Debian stretch NAS with only 3gb of ram) I had to go back to 5.14.0.177 because after less than 6hours all indexer stop working inside Radarr and Sonarr.
After the roll back all is returned to normality.
I used to limit the use of ram into the systemd service but I noticed that the virtual memory usage go very high. Do you know a better method to keep running the server without restart it after a week only?

[Yellow1144]

Thanks for the tip about the Mono version. I'll try it.

However, regarding your server question, I have nothing for you. Sorry.

[kaso17]

The only memory consumption which might grow significantly is the cache (jackett keeps the last 1000 results of each indexer in memory). Unless you have a lot of active indexers filling up the cache memory I've no idea how Jackett could reach 10G unless mono is leaking memory.

[kaso17]

2405 is related

[Jorman]

I just edited the wiki page about the systemd script, I added some info to use a script less aggressive with low powered system
https://github.com/Jackett/Jackett/wiki/systemd-service

[elliopitas]

i have the same issue with my unraid server

[sho-87]

This happens for me on unraid as well. top shows high memory usage in my jackett, sonarr, and radarr dockers, and it seems to specifically be the mono process in each thats causing the issue

[zero77]

There may be a memory leak with mono because, lots of users of Radarr which also uses mono like Jackett, are reporting high memory usage and some users are suggesting it looks like its down to mono Radarr/Radarr#1580.

[jacaru]

I think this should be looked at on its own. This leak, at least in my case, it's much more noticeable and has only appeared recently, somewhere beginning of November. It might be the same but for me jackett is leaking much more.

[dshanks01]

Jackett Version 0.10.533.0
Linux Mint 19.1
Mono 5.18.0.225

Memory leaking seems pretty severe. After running for 10 hours overnight, the resident set size (memory allocated and in RAM) went from 126728KB to 335268KB and continues to grow. This is without the systemctl "low powered system" modifications suggested in the wiki. I'll try those and report back. I'll also try rolling mono back to 5.14 to see if that has any effect. There's also a mono branch (profiler-improvements in fork kumpera/mono) that implements a flag to profile and log memory allocation within mono. If I can get a binary or figure out how to build it I can post a log here (assuming Jackett's developers would find it useful).

While this may be a mono problem, it appears to more severe in Jackett than in other mono based daemons. I also have Sonarr and Radarr loaded. Over the same time period, Sonarr grew from 113668KB to 172272KB and Radarr grew from 119728KB to 254368KB. All three continue to grow.

For now, I'll probably have cron restart these daemons every 12 hours or so to keep them from completely taking over my computer.

[kaso17]

another option might be to try run the .netcore version of jackett (doesn't need mono).

[Jorman]

@kaso17
There's some wiki that explains how to try that?

[dshanks01]

@kaso17
I found a branch named dotnetcore which appears to be a proof of concept, but there's zero information on how to use it or if it's even in a usable state. Is that what you're talking about, or is there a forked project somewhere? I'm going to assume you don't mean "Jackett2," which hasn't been touched for a couple of years.

[kaso17]

The latest master branch is fully dot net core compatible. You need to build the Jackett Server subproject. But I've never build it using linux.

[ghost]

Timing....
Just been playing with this in the last hour
As @kaso17 says the code is .NET Core compatible, but not much testing has happened and the updater has been implemented for .NET Core
From now on, .NET Core binaries will be created with each build, but will be 'hidden' until the updater is implemented. Will try and get to the updater in January, others most welcome (and appreciated) to have a go though first

@Jorman @dshanks01
You can try the .NET Core version from here, feedback welcomed
https://ci.appveyor.com/project/Jackett/jackett/build/artifacts
Download the .NET Core version for your OS and then to run ./JackettConsole (mono not needed)

[Jorman]

Thanks I'm trying the .NET core right now, let's see what happen.

Take a look to the VRAM, I noticed that sometime became in M, now is G and is red, but is impossible with my system to go at this level, lol!

[elliopitas]

Jackett Version 0.10.533.0
Linux Mint 19.1
Mono 5.18.0.225

Memory leaking seems pretty severe. After running for 10 hours overnight, the resident set size (memory allocated and in RAM) went from 126728KB to 335268KB and continues to grow. This is without the systemctl "low powered system" modifications suggested in the wiki. I'll try those and report back. I'll also try rolling mono back to 5.14 to see if that has any effect. There's also a mono branch (profiler-improvements in fork kumpera/mono) that implements a flag to profile and log memory allocation within mono. If I can get a binary or figure out how to build it I can post a log here (assuming Jackett's developers would find it useful).

While this may be a mono problem, it appears to more severe in Jackett than in other mono based daemons. I also have Sonarr and Radarr loaded. Over the same time period, Sonarr grew from 113668KB to 172272KB and Radarr grew from 119728KB to 254368KB. All three continue to grow.

For now, I'll probably have cron restart these daemons every 12 hours or so to keep them from completely taking over my computer.

yea me too

[dshanks01]

@flightlevel
I've been running the .Net Core version for the past several hours with no signs of memory leak. The RSS seems pretty big for what this program does (~200MB), but I've heard that C#/.NET programs tend to be pretty bloated.

The build process gives write-others permission to JackettConsole, which is odd and a potential security risk. The .so files also ended up with write-others permissions. A simple makefile could be used to automate the Linux build process including setting permissions appropriately and installing the daemon.

EDIT: Actually, VSZ is increasing while RSS is _decreasing_. VSZ (>21GB!) has gained 763MB in 8 hours. This seems like it probably indicates a defect in the program. But because it's increasing the program's address space without increasing the physical memory allocated, it doesn't seem to be a problem for now.

[ghost]

@Jorman @dshanks01
Thanks for your feedback and testing. Its good to know that resident memory use has been stable on the .NET Core version while you had issues on the Mono version.
I wouldn't be too concerned on the virtual memory size. Other .NET Core apps have reported the same symptoms to Microsoft so its not Jackett specific and they don't seem concerned e.g. https://github.com/tidusjar/Ombi/issues/2112

@dshanks01
Agree 200MB is large, but not terrible. .NET/C# does give you a lot out of the box which makes dev easier but you do pay for it a little in terms of size. In the future for .NET Core is https://github.com/dotnet/core/blob/master/samples/linker-instructions.md and AOT https://github.com/dotnet/corert which may help with size

We do have a build system called 'Cake' and its cross platform. However, when I first started writing it there was no plans for .NET Core, we build a Windows installer (InnoSetup) and Appveyor only supported Windows, so I only factored in Window usage. Now that AppVeyor supports Linux and we are heading towards .NET Core, I'll refactor it in the new year so that it support other OSes minus InnoSetup.
Cake (C# Make) is a cross-platform build automation system with a C# DSL for tasks such as compiling code, copying files and folders, running unit tests, compressing files and building NuGet packages. https://cakebuild.net/

Currently we set all files to 755 https://github.com/Jackett/Jackett/blob/37216bd0a9e11342e4d58800f6a3a747ac7bbc0b/build.cake#L292 What should it be?

[Jorman]

@flightlevel
So, we can stay with the .net core version?
Just to know, what about the new release of Jackett? There's a way to automatize all or for now we've to manually download the latest release from appveyor and apply by hand

[ghost]

@Jorman , yes, fine to stick with the .NET core version. If you have any problems please mention that you are running on Core build. Currently updates will need to be done manually. The updater will be upgraded soon and you will then get automatic updates for .NET core as well

[Jorman]

Ok, I'll sty with net core, let's see if some strange happen!

[edjalmobf]

I kind of solved this by adding this line in jackett systemd service
WatchdogSec=3000
This will make it restart after every 50 minutes.
But this is just a quick fix, but I think it’ll work!

[hadim]

I think I could be facing a similar issue: https://github.com/linuxserver/docker-jackett/issues/62#issuecomment-455021002

I am running Jackett on Docker on a QNAP NAS.

[Yellow1144]

@edjalmobf - Thanks for that idea. I set my service for 4 hours and now seem to be avoiding runaway memory consumption.

I also see that dotnet core version and solution above. It looks promising, but more complex than I have time to undertake at the moment. Will file this away for future reference.

[bruvv]

Random expierigns it now too on my nas, also on the raspberry :( hope the dotnet version gets released soon than
@hadim how did you get that overview? *im running on a synology and would love to have the same details.

[MrAmos123]

I've been fighting this issue for about 4 months now. It's annoying as hell because it ends up completely halting the servers as it runs entirely out of memory/swap.

Mono JIT compiler version 5.18.0.225 (tarball Wed Jan  2 21:20:16 UTC 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          yes(600)
        Suspend:       preemptive
        GC:            sgen (concurrent by default)

Jacket Version: v0.10.649

[SonGokussj4]

Hi. 1.85 GB on my Synology NAS (Docker) after 17 days. I've got only 4GB RAM. 3.5 GB already used. 1.85 GB is really huge...

After I stopped and run Jackett again, 133 MB.

[hadim]

Until the new version of Jackett without Mono, I ended up restarting my Jackett Docker image every 12h.

[bruvv]

@hadim van you explain how you did that ? Also please enlightening me about the stats that you posted earlier.

[hadim]

@d1slact0r for the automatic restart just make a bash script with an infinite loop that restarts the docker image with docker-compose restart jackett in my caseevery 12h using thesleep` command.

For monitoring memory usage please see the bash and Python scripts I used there: https://github.com/linuxserver/docker-jackett/issues/62#issuecomment-455272271

[XerionXavier]

Having the same issue. having cron run this gross little script seems to solve the problem.

!/bin/bash

RAM=$(free | grep Mem: | awk '{print $3}')

if [[ $RAM -gt 1400000 ]]; then
docker restart $(docker ps -a -q)
{ echo "*Docker Restart*" ; command date ; } > /var/log/docker/$(/bin/date +%R_%m.%d.%Y)_Docker_Restart

fi

[edjalmobf]

@Yellow1144 and other guys can you say if after updates v0.11.224, that reduces the cache size from a 7 day retention to just 1 hour, and the v0.11.230, that enable workstation garbage collection, got better?

Just reporting because I think these is a similar issues: https://github.com/Jackett/Jackett/issues/4966