Ombi: Memory Leak / High Utilization

[animosity22]

Ombi Version:

Application Version: 2.1.1336

Update Branch:

Stable

Media Sever:

Plex

Media Server Version:

Version 1.4.4.3495

Operating System:

root@plex:~# lsb_release  -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.7 (jessie)
Release:    8.7
Codename:   jessie
root@plex:~# uname -a
Linux plex 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07) x86_64 GNU/Linux

Mono Version (only if your not on windows)

root@plex:~# mono --version
Mono JIT compiler version 4.8.0 (Stable 4.8.0.520/8f6d0f6 Wed Mar 15 16:18:35 UTC 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
    TLS:           __thread
    SIGSEGV:       altstack
    Notifications: epoll
    Architecture:  amd64
    Disabled:      none
    Misc:          softdebug
    LLVM:          supported, not enabled.
    GC:            sgen

Applicable Logs (from /logs/ directory or the Admin page):

I really don't see much as my logging is currently at error level, but here is a clip of the last few errors.

System.NullReferenceException: Object reference not set to an instance of an object at Ombi.Services.Jobs.PlexUserChecker.Start () [0x0009c] in <7d30b485e04b4d81ad44f0b2cb2d3afb>:0    Ombi.Services.Jobs.PlexUserChecker  Error   3/15/2017 1:50:18 AM
System.InvalidOperationException: There is an error in XML document (0, 0). ---> System.Xml.XmlException: Root element is missing. at System.Xml.XmlTextReaderImpl.Throw (System.Exception e) [0x0002a] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo (System.String res) [0x00017] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlTextReaderImpl.ParseDocumentContent () [0x0041e] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlTextReaderImpl.Read () [0x00097] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlTextReader.Read () [0x00000] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlReader.MoveToContent () [0x0004b] in <611e64636ff745389933c69c817b2d4a>:0 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderPlexAccount.Read3_user () [0x00008] in <9088e4defa324cf4bd8cef5639e8ac39>:0 at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in :0 --- End of inner exception stack trace --- at System.Xml.Serialization.XmlSerializer.Deserialize (System.Xml.XmlReader xmlReader, System.String encodingStyle, System.Xml.Serialization.XmlDeserializationEvents events) [0x00139] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.Serialization.XmlSerializer.Deserialize (System.Xml.XmlReader xmlReader, System.String encodingStyle) [0x00000] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.Serialization.XmlSerializer.Deserialize (System.IO.TextReader textReader) [0x0001c] in <611e64636ff745389933c69c817b2d4a>:0 at Ombi.Api.ApiRequest.DeserializeXml[T] (System.String input) [0x00017] in <4291b6fb13504ddfb4bd3ad49e0f16a9>:0 Ombi.Api.ApiRequest Error   3/15/2017 1:50:18 AM
System.InvalidOperationException: There is an error in XML document (0, 0). ---> System.Xml.XmlException: Root element is missing. at System.Xml.XmlTextReaderImpl.Throw (System.Exception e) [0x0002a] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlTextReaderImpl.ThrowWithoutLineInfo (System.String res) [0x00017] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlTextReaderImpl.ParseDocumentContent () [0x0041e] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlTextReaderImpl.Read () [0x00097] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlTextReader.Read () [0x00000] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.XmlReader.MoveToContent () [0x0004b] in <611e64636ff745389933c69c817b2d4a>:0 at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderPlexFriends.Read5_MediaContainer () [0x00008] in <58b707d017324162833a1da7534f2f8b>:0 at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038] in :0 --- End of inner exception stack trace --- at System.Xml.Serialization.XmlSerializer.Deserialize (System.Xml.XmlReader xmlReader, System.String encodingStyle, System.Xml.Serialization.XmlDeserializationEvents events) [0x00139] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.Serialization.XmlSerializer.Deserialize (System.Xml.XmlReader xmlReader, System.String encodingStyle) [0x00000] in <611e64636ff745389933c69c817b2d4a>:0 at System.Xml.Serialization.XmlSerializer.Deserialize (System.IO.TextReader textReader) [0x0001c] in <611e64636ff745389933c69c817b2d4a>:0 at Ombi.Api.ApiRequest.DeserializeXml[T] (System.String input) [0x00017] in <4291b6fb13504ddfb4bd3ad49e0f16a9>:0

Problem Description:

My library size is as follows:
~1400 Movies, ~326 shows with ~13300 episodes.

I'm watching Ombi grow over the course it runs to almost consuming all the memory on my machine.

I initially thought it might be related to episode searching, but I disabled that a few days back and I still get the same pattern.

Process Snapshot
http://i.imgur.com/ekdWKHV.png

Using netdata, I capture the process and record/memory/cpu/etc and graph that in Grafana:

http://i.imgur.com/y99jLjU.png

You can see the drops each time I restart the process.

Reproduction Steps:

Restart the deamon and the memory will grow.

[Ombi-robot]

Hi!
Thanks for the issue report. Before a real human comes by, please make sure you used our bug report format.
Before posting make sure you also read our FAQ and known issues.
Make the title describe your issue. Having "not working" or "I get this bug" for 100 issues, isn't really helpful.
If we need more information or there is some progress we tag the issue or update the tag and keep you updated.
Cheers!
Ombi Support Team

[tidusjar]

Wow. Can you find the time memory usage started to rise and correlate it to a scheduled job?

[animosity22]

Based on the graph, it just seems a very gradual staircase moving upwards.

You can see when I do a restart:

http://i.imgur.com/XtLKjJj.png

[animosity22]

Hmm.

The only thing I could think of that I was doing is that I was hitting that login page for monitoring purposes to validate the app was actually up and running using Worldping.

Some examples look like:

[15/Mar/2017:06:36:03 -0400] "GET /userlogin HTTP/1.1" 200 6609 "-" "Go-http-client/1.1"

Which is part of the Apache log as I reverse proxy it in.

I turned that off and I was going to monitor over the next few hours.

[tidusjar]

Ok, keep me updated, the more information I have the easier it will be to identify and fix

[animosity22]

So I did more testing and super easy to reproduce.

I ran a quick loop to just curl the login page every 1 second and you can tell when I started the script.

 while true; do curl https://MYBOX/userlogin; sleep 1; done

I do a pretty straight forward Reverse Proxy in Apache:

ProxyPass / http://127.0.0.1:3579/
ProxyPassReverse / http://127.0.0.1:3579/

Here is when I started it:
http://i.imgur.com/o0iInx4.png

[tidusjar]

I have profiled Ombi a few times now and I cannot get this to happen. I think this must be a mono issue as on windows it's GC'ing correctly and releasing back the unmanaged memory (that is not stored in the heap)

You can see this here:

This is only over a 1 minute 30 seconds but yes, every time I refresh the page we use a small amount of memory (expected) but it get's garbage collected very soon after.

[animosity22]

Any suggestions on what I should do in terms of next steps?

[tidusjar]

The mono part is out of my control really...

You could hit the API instead:
http://localhost:port/api/version?apikey=OmbiApiKey

[erodewald]

I'm experiencing Ombi eating up a high amount of memory, and then getting [apparently] GC'd. At this point, it starts eating up a ton of CPU cycles. Check out this CPU/memory graph from my Linux container virtual environment. What can I do to help track down the issue?

_edit:_
At this point, the app has been unresponsive (won't load a page at all, but the service shows that it is up):

root@plexrequests:~# service ombi status
● ombi.service - LSB: Ombi
   Loaded: loaded (/etc/init.d/ombi; bad; vendor preset: enabled)
   Active: active (running) since Thu 2017-03-30 23:43:39 UTC; 1 weeks 5 days ago
     Docs: man:systemd-sysv-generator(8)
  Process: 27833 ExecStop=/etc/init.d/ombi stop (code=exited, status=0/SUCCESS)
  Process: 27895 ExecStart=/etc/init.d/ombi start (code=exited, status=0/SUCCESS)
    Tasks: 58 (limit: 512)
   CGroup: /system.slice/ombi.service
           └─27906 /usr/bin/mono Ombi.exe

Mar 30 23:43:39 plexrequests systemd[1]: Starting LSB: Ombi...
Mar 30 23:43:39 plexrequests ombi[27895]:  * Starting Ombi Ombi
Mar 30 23:43:39 plexrequests ombi[27895]:    ...done.
Mar 30 23:43:39 plexrequests systemd[1]: Started LSB: Ombi.

_edit2:_
Just saw you had a new version. I've installed it and will continue to monitor since it doesn't look like this issue was really resolved.

Application Version:    2.2.1676
OS: Mono
System Version: 4.6.2 (Stable 4.6.2.7/08fd525 Mon Nov 14 12:30:00 UTC 2016)
Branch: Early Access Preview
Log Level:  Error
Database Location:  /opt/ombi/PlexRequests.sqlite
Running Directory:  /opt/ombi

[tidusjar]

Do you have episode searching enabled?

Just to let you know I am working on rewriting Ombi currently to remove the mono dependency that I am constantly fighting with. It's going to be a while, but just to inform you. https://github.com/tidusjar/Ombi/issues/865

[erodewald]

@tidusjar I saw you make a reddit submission requesting info on .net core. I'm in the same boat, sort of-- trying to convert a webAPI 2.0 service to .net core. I certainly don't want to nag you given that you're working to move past the issues that come with Mono. I should be able to debug some issues I've seen soon enough.

To answer your question, I did have Episode Searching enabled for Plex. I will disable that and see if it has any impact.

[tidusjar]

Yeah the implementation of episode searching is not great.

Regarding .Net core, it's progressing nicely. It's just a complete rewrite since the framework in the stable version is using Nancy rather than MVC.

[animosity22]

It seems like one of the later versions of mono sorted this out as it's not happening anymore. I'll close it out as solved .

[fenichelar]

I still have this issue :(

[fenichelar]

That drop was when I shutdown Ombi. Ubuntu 16.04, Mono 4.8.1. 8GB seems a little high on memory usage, but I do have episode search on and a 20TB library...

[tidusjar]

I'm currently writing Ombi to remove the mono dependency

On Wed, 3 May 2017, 6:28 am Alec Fenichel, [e-mail] wrote:

[image: image]
https://cloud.githubusercontent.com/assets/3437075/25648997/adfc6d9e-2f9f-11e7-8e0c-532727b9cae8.png
That drop was when I shutdown Ombi. Ubuntu 16.04, Mono 4.8.1.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/tidusjar/Ombi/issues/1256#issuecomment-298825902, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGVaLDbEyc65Vbfs1fxzlpdbWUWA5ZQWks5r2BBtgaJpZM4MfTu7
.

[fenichelar]

Let me know if you want any Ubuntu testing help.

[harihoudini]

I've been having the same issue with Ombi on Unraid (Docker). As soon as I reset or enable the docker, the memory keeps creeping up until Unraid completely chokes and I need to Kill the process.

Application Version: 2.2.1678
OS: Mono
System Version: 4.6.2 (Stable 4.6.2.16/ac9e222 Tue Jan 3 11:48:26 UTC 2017)
Branch: Stable

Unraid : 6.3.4

Using either Linuxserver or Rogueosb docker produces same result.

I have just reinstalled Ombi docker from scratch and therefore do not have have the graph captures from last few days to demonstrate it.

Is it a possibility that the memory leak is due to Ombi continuously trying to process the default/Main account user of Plex and not completing the task?

[tidusjar]

So it's to do with our scheduling library with mono. Not much we have control over.

The rewrite too make it native on *nx systems is in progress and will be available for send alpha testing soon. See #865