Spksrc: [Update Request] Mono to 5.16 (current stable) - fixes Radarr/Sonarr memory leaks

Created on 28 Sep 2018  路  115Comments  路  Source: SynoCommunity/spksrc

Sonarr and Radarr packages are experiencing memory leaks, and I believe it was trace to Mono being the culprit.

Radarr Issue: https://github.com/Radarr/Radarr/issues/1580
Sonarr Issue: https://github.com/Sonarr/Sonarr/issues/2296

Mono Issue that addresses the memory leak in version 5.14: https://github.com/mono/mono/issues/7356

Can we please upgrade to Mono 5.16?

Most helpful comment

Some news: Finally managed to tweak the package and make it compile for qoriq (PowerPC) platforms. See here for build details:
https://travis-ci.org/m4tt075/spksrc/builds/499667955
New test builds here:
https://github.com/m4tt075/spksrc/releases/tag/mono-5.18-11
Happy testing and please report back if there are any issues.

@ymartin59 Let's observe download rates, but I think we should target merging this in the not-too-far future. The mono release cycles are short and I don't think it will get much better than this.

All 115 comments

5.16 stable is out - I suggest to update your issues's title to "[Update Request] Mono 5.16 (current stable) - fixes Radarr/Sonarr memory leaks"

5.16 stable is out - I suggest to update your issues's title to "[Update Request] Mono 5.16 (current stable) - fixes Radarr/Sonarr memory leaks"

Request updated. Thanks for the heads up.

I just checked the changelogs of mono and given we are currently on 5.8, updating mono will resolve another major and long outsdanding issue for Radarr and Sonarr.

From mono 5.12 release notes:
HttpWebRequest async handling has been rewritten. This resolves many long-standing and hard to reproduce issues involving requests cancellation or timeout. As HttpWebRequest is used as the underlying implementation by other types like HttpClient this should improve the reliability of a broad range of types.

@m4tt075 Hello. Do you have timeslot to update Mono package?

@ymartin59 Happy to look at it in the next couple of days. Will put it on my list. But I think there is some backlog in terms of open PRs which deserved attention first...

I've just tried to update to latest release but ran into a regression, which occurs with cross-compilations. See Mono Issue 9951 for details. I'm hesitant to trade one set of problems for another. To those of you, who have had memory leak issues with the current package: How bad is it?

@m4tt075 It's pretty bad. I'm on a smaller NAS with 1GB of RAM. If I don't restart Sonarr or Radarr once a week, the entire system is unusable. A hard reboot is required. If you check Radarr thread I linked, you'll see others have up to 4 GB of memory used by mono. Some have 2GB. So looks like it various depending on how much RAM your system has, which sounds right because the runtime will adjust itself to system RAM capacity.

@jasonla OK, I don't like to work around the regressed 5.16, but will try the 5.14 version instead. It should contain the fix already. Which NAS / platform are you running? Would like you to test it, when I'm done.

@m4tt075 I'm on DSM 6, and using a Synology DS214play, which I believe is the Evansport platform. Yes, I'm willing to test it.

OK chaps. DSM 6.1 and 6.2 test builds available here:
https://github.com/m4tt075/spksrc/releases/tag/mono-5.14-1
Please give it a try and report back whether this fixes your issue.

Of note: After the update the 88f6281 architecture does not compile the btls library anymore. I could simply exclude it, but not sure about the consequences. Does anyone know whether it is required, can be replaced or simply be omitted?

@m4tt075 I鈥檒l give it a try after I get home from work. If install works, it will probably take a few days of watching memory usage before I can give you any meaningful info.

Would it be ok for me to publicize this test build on Reddit? Just so we have people testing on different platforms and not just my 214play?

Edit: Installation went fine. Waiting and watching memory now.

@jasonla Thanks for testing. Yes, please feel free to link those builds via Reddit, if you haven't done that already. The more testers the better. If possible, please also re-post the question I've raised, as it affects ARM5 users, if we don't find a way forward for that platform...

Working great (tested Armada 370). Random HTTP socket issues seem to be resolved.

@matt075 just a question. would this update fix issues if radarr is in a docker container? and which one for Synology DS918+

Cheers

@Faltimore If it鈥檚 in Docker, then the Dockerfile controls which version of mono is installed.

Edit: And it also depends which docker repo you're pulling from. If it's the common linuxserver.io repo that a lot of people recommend (no idea if it's legit or not, I just see it recommended), then no this won't help your issues at all. This package update is only going to help if you're running the Radarr/Sonnar packages directly on DSM, not inside a Docker container.

I tracked down which version of mono the linuxserver.io Radarr images are built on, and it's 5.10.1.20. If you want the docker images updated, you need to talk to the owner/maintainer of those images.

Cheers.

Thanks for stepping in, @jasonla! 馃憤
Can you share some insights on your memory usage already or is it still too early to say?

I've noticed that the memory leak is much improved when using 5.14. However, it started climbing again today, but in very small increments, and not nearly as dramatic as it was under 5.8. I'm tempted to call this at least an improvement, but can we wait a couple more days to see if it gets worse?

Sure. No prob. It also seems there have only been 11 downloads of the test packages so far. Certainly not enough yet...

Reporting back. So far, so good. I've noticed that memory will get released, and it doesn't reach the levels I was seeing before.

Welp, Radarr has an internal upgrade mechanism, and after upgrading, I can't seem to run it anymore. I'm checking to see if it's related to the upgrade script being screwed up, or if Radarr is incompatible with this 5.14 build, or my DSM is screwed up.

FYI - I did the update and it still works fine here. So it's not related to the mono test pkg.

Thanks for the feedback @piejanssens! I did a little digging, and this is what I found.

1) Something was wrong with my install of Radarr. So I uninstalled it and tried to reinstall (and forgot to backup my config/database for Radarr. Oops, but that's not important.)

2) I have the test mono 5.14 package installed. When I tried to install Radarr via the SynoCommunity package using the built in package manager UI, it installs fine, but will not start.

3) There's a reason it doesn't start: Radarr/Sonarr/Jackett,etc were all using a version of NLog.dll that was not compatible mono 5.10 or above. See this thread: https://github.com/Radarr/Radarr/issues/2585

The error with NLog.dll has been fixed in subsequent versions of Radarr. The problem is, the version of Radarr that is installed by default in the SynoCommunity packages relies on mono 5.8, and users will experience the same start up issues I had if they install 5.14, and then try to install Radarr. Users can't use the self-updater built into Radarr since they can't get it to start.

Based on the thread I linked, I believe Sonarr and Jackett would experience this issue too if mono is at version 5.14 (or anything above 5.10).

If we want to release the 5.14 package, it would mean the you'd have to update the Sonarr/Radarr/Jackett packages that are installed by default in SynoCommunity packages. I don't know how much work is involved in that.

Edit: Further reference links:

https://forums.sonarr.tv/t/mono-5-10-update-broke-sonarr/17800/19
https://www.reddit.com/r/sonarr/comments/80ylur/new_sonarr_version_v2005183_and_mono_510_guide_if/

Thanks for looking into this and posting all the references, @jasonla. Very helpful and much appreciated. I just find it very difficult to advance from here: From what I've experienced in the last couple of weeks / months, it is very difficult to find any half-way recent stable mono version that does not create some kind of problem or regression. What I also observed is that the number of test package installations of my mono 5.14 is still very low (14 as of now). This usually indicates, that only few people actually have issues with our current package. I'd therefore struggle to push any update until all relevant mono, sonarr, radarr and jackett issues are fixed upstream.

@ymartin59, @Safihre (if you are still around) : Any advice / further insights from your side?

@m4tt075 I really agree with you... it make sense to "group" these applications deliveries because of publish process efforts. Hurry is a high risk for urgent support requests because of application disfunctions, and produce much more hurry then.

When compiling jackett from source it also seems it is using the wrong dll so it wont work on mono 5.8
Mono 5.16 does work..

https://github.com/Jackett/Jackett/issues/4232

Recent versions of Jackett are broken with the current Mono 5.8. Seems fixed with the latest 5.18 though.

https://github.com/Jackett/Jackett/issues/4369

@gotson Can you reproduce this on your Syno system? If so, please provide the corresponding jackett.log file.

@m4tt075 yes I can reproduce, here is the log file: https://gist.github.com/gotson/952ed43759cfa9a5a428ca4e1ff4b870

I have the problem both when starting the package from DSM UI, and from command line with synopkg start jackett.

Jackett has been self-upgrading to v0.10.550.0 (and failed to restart afterwards).
The Jackett Synology package installed is 0.8.490-6.
The Mono Synology package installed is 5.8.0.108-11.
I am running DSM 6.2.1-23824 Update 2 on a DS213j.

OK. Thanks.
@ymartin59 Pretty sure this will hit others soon. Will look at it...

@gotson I've produced an armada370 test build for the mono 5.18.0.225 pre-release version. You can download it and manually install it from here:
https://www.dropbox.com/s/3tcwkh2jy98z3jr/mono_armada370-6.1_5.18.0.225-12.spk?dl=0
Please test jackett, sonarr or whatever other mono-depending packages you use and report back what you have tested and what you found, ok?

Whoever else needs different test builds, just let me know what Syno model you have and which DSM version you are running. You will have to test and report your test results though... ;-)

@m4tt075 can i have a have a avoton build? so can test with radarr and sonarr?

@M0UL Here you go...

Currently available via...
~https://www.dropbox.com/sh/zl67clwy5rio8kn/AADpWJhkHQ6RgqaCbLxbNIJoa?dl=0~
https://github.com/m4tt075/spksrc/releases/tag/mono-5.18-2

  • 88f6281-6.1
  • armada370-6.1
  • ~avoton-6.1~ --> use x64-6.1 package instead
  • ~bromolow-6.1~ --> use x64-6.1 package instead
  • Edit: Remaining 6.1 and 6.2 packages available too

@gotson I've produced an armada370 test build for the mono 5.18.0.225 pre-release version. You can download it and manually install it from here:
https://www.dropbox.com/s/3tcwkh2jy98z3jr/mono_armada370-6.1_5.18.0.225-12.spk?dl=0
Please test jackett, sonarr or whatever other mono-depending packages you use and report back what you have tested and what you found, ok?

It's working great! Jacket works fine now.

Sonarr is fine:

Radarr also:

@m4tt075 I recommend you only build "x64" for multiple architectures as avaton==bromolow==x64

@ymartin59 I understand this for pure python packages, for instance, but otherwise I'm struggling to judge which builds are going to be interchangeable and which ones are not. I know the architecture group definitions in spksrc.common.mk (ARM5, ARM8, etc). Are they always interchangeable within a group? Are there cases when ALL, for instance, ARM architectures are interchangeable too? Are there any kind of "rules" you apply?
Also, do I have to amend the Makefiles or any other configurations to tell DSM that the package in question can be used for multiple architectures?

@m4tt075 In fact, there are only two "virtual" architectures which generates binaries compatible for multiple architectures:

Any other architecture (not listed in that variables, so arm v5, arm v7 and qoriq) requires a build from this own specific toolchain.

OK, here all 6.1 and 6.2 mono 5.18.0.225 (pre-release) packages for testing purposes:
[Link outdated, see below for more recent test packages]
Please continue to report successes and failures of all packages depending on mono.

Reporting back: It seems the 5.18 package caused massive memory leaks in my DS214 Play. It accumulates more memory that the 5.8 package, and faster. It's within 3 days that Radarr and Sonarr slow to a crawl, and it affects performance of the whole NAS (NZBGet, the DSM interface, file shares, etc).

@m4tt075 Would you mind reposting the 5.14 packages you created before?

@jasonla Well, mono 5.18.0.225 has just been declared as the new "stable release". Not sure what you experience is a regression or a new problem. Unfortunately, we cannot build random intermediate versions, to see which ones work in which scenarios and which ones don't. I think the only appropriate way forward is to report the problem upstream at https://github.com/mono/mono, add the necessary debugging information and asking for a fix there. Could you please do that?

In the meantime, here is your 5.14 evansport package to keep you going: https://www.dropbox.com/s/ix6sxz9dv9iduju/mono_evansport-6.1_5.14.0.177-12.spk?dl=0

@ymartin59: And that brings us back to square one, as all recent mono versions seem to exhibit one problem or another... sigh

Thank you, @m4tt075

I don't expect you to have to build random intermediate versions, at all. I'm just trying to see if switching back will fix the issue, or if it's something else causing the problems. I tried using the original 5.14 links you posted, but those look like they're gone, hence why I asked 馃槃

I'll report back my findings. Thank you very much for all of your work on this. It's truly appreciated.

OK. Cool. Yes, I've deleted those builds as I thought we could finally catch up with the latest release versions again.. :-S

Is this a known issue with this mono version? (mono_88f6281-6.1_5.18.0.225-12)

Starting Jackett as user sc-jackett
 rcall_membase R253 <- [R258 + 0xffffffbc] [System.IConvertible:ToSingle (System.IFormatProvider)] [arm_r0 <- R256] [arm_r1 <- R257] [arm_v5 <- R259] clobbers: c
* Assertion: should not be reached at decompose.c:1889

@m4tt075 Have you or anyone else attempted to build mono for Qoriq since the mono 5.0 release?
https://www.mono-project.com/docs/about-mono/releases/5.0.0/#bring-powerpc-back-to-life

@piejanssens I can't speak for anyone else obviously, but I've tried. And failed, unfortunately...
@jasonla Any news from your testing?

@m4tt075 Sorry to get back to you so late. The 5.14 release performed better for me than the 5.18 package. However, I noticed both Sonarr and Radarr released new versions since the last time I installed the 5.18 build. Specifically, these are the two interesting release notes:

Sonarr
Mono bug causing memory leakage when http connections use gzip
https://github.com/Sonarr/Sonarr/pull/2882

https://github.com/diveflo/Sonarr/commit/7ffc5154e7df668302908f2f30f355093431b57d

Radarr
https://github.com/Radarr/Radarr/releases/tag/v0.2.0.1293

It looks like the latest releases of both Sonarr and Radarr try to address memory leaks. I will reinstall 5.18 and use the new versions of Sonarr and Radarr to see if things improve.

@jasonla No worries. All fine. It seems, mono has had maintenance releases as well in the last couple of weeks. Assuming there are no major issues with those, I can update the mono test packages on the weekend as well. Then you could test the latest (stable) package combination...

OK, updated mono packages 5.18.0.240 available here:
https://github.com/m4tt075/spksrc/releases/tag/mono-5.18-3~~
Happy testing...
Edit: Newer builds below.

@Stanzilla No, I don't. I know there have been issues with the Jackett package, which is why we pushed an update recently (#3572). But not sure whether those issues are related to yours.

Our Jackett update has not been published yet. From your report above, I assume this should suit your platform: https://www.dropbox.com/s/0jyfr8lq4dbpjob/jackett_88f6281-6.1_0.8.490-6.spk?dl=0

If you want, give it a try and let us know.

Some news: Finally managed to tweak the package and make it compile for qoriq (PowerPC) platforms. See here for build details:
https://travis-ci.org/m4tt075/spksrc/builds/499667955
New test builds here:
https://github.com/m4tt075/spksrc/releases/tag/mono-5.18-11
Happy testing and please report back if there are any issues.

@ymartin59 Let's observe download rates, but I think we should target merging this in the not-too-far future. The mono release cycles are short and I don't think it will get much better than this.

@m4tt075 This is an amazing achievement. Thank you so much for your efforts.

@piejanssens Thanks. Just hope it will work, too. I have no way to test those builds. Only you guys can tell... ;-)

@m4tt075 OK I may publish this week-end. Are there any mono-based application that may require an update to run on this newer version?

Yes, I'm in the process of unlocking qoriq support for each of them and seeing whether they compile. Will add individual commits to this PR. Jackett seems to work already, but Lidarr, Radarr, Sonarr and my new WebGrab++ still need to compile successfully. I'll report the results here when I'm done (hopefully tomorrow).

OK, I've successfully cross-compiled Jackett, Lidarr, Radarr and Sonarr for qoriq-6.1. Test packages are available here:
https://github.com/m4tt075/spksrc/releases/tag/Mono-qoriq-deps
Of course the most recent Mono test packages from here
https://github.com/m4tt075/spksrc/releases/tag/mono-5.18-11
need to be be installed first.

We intend to merge and publish these changes in a couple of days, so any test reports (definitely not only on qoriq-platforms) will be very much appreciated. Thank You.

Thanks m4tt075/ymartin59 and others,

Small question, am I missing the avoton spk (for DS1815+) or should I use another one? Will test it in combination with Sonarr and Radarr.

Edit: my bad, just read the post about virtual architectures that avoton should be x64 (btw, naming conventions shows 5.2 (shouldn't it be 6.2?) in test builds)

@Dubbeltje If you are on DSM6.1 or 6.2, you will need the x64-6.1 version. If you are still on DSM5.2, you'll need that one.

@m4tt075 I've installed Mono succesfully, installation of Radarr also, but fails to run the package service.

Which log do I need to analyze to see what's going wrong?

OK. Please log into our NAS via ssh and look for hints in:

  • /usr/local/radarr/var/radarr_install.log
  • /usr/local/radarr/var/radarr.log
  • Anything else that looks like a log file in /usr/local/radarr/var
  • dmesg

radarr_install.log contains no errors

excerpt from radarr.log:
Debug info from gdb:

mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

Got a SIGILL while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application.

dmesg:
[97765.964969] init: pkgctl-radarr pre-start process (3369) terminated with status 1
[97816.999980] init: pkgctl-radarr pre-start process (6908) terminated with status 1
[98829.012728] init: pkgctl-radarr pre-start process (2180) terminated with status 1

I don't have any other files in var.

Hmh, there were problems with older Jackett versions, too. Installing newer ones seems to work, at least according to my limited tests. The Radarr version we are shipping is from 2017. What NAS architecture and DSM version do you have? I'll try to build a newer version to see whether that works for you...

@m4tt075 Did you run the mono testsuite on an qoriq cpu synology? Back in the day (<2015) that demonstrated the jit issues. So did running something simple like (long)Math.Round(20000000.0) (which returned 0 back in 2014).
But it's awesome news, we've been getting questions about support on qoriq devices over the years, to the point that people regretted buying the otherwise excellent device.

@Taloth Nope, I've just made it compile utilizing autogen, which apparently wasn't tried before. I don't own a qoriq platform though. To that extent I have absolutely no idea whether it actually works, which is why I'm looking for testers. I hope it will work, but please don't be disappointed if it doesn't...

Nope, I've just made it compile utilizing autogen, which apparently wasn't tried before.

Back in 2014 SynoCommunity had a qoriq package for Sonarr, it just didn't work coz mono didn't support the platform. The problem with qoriq was that it's actually more a ppc-e500 platform, rather than ppc64.

Shameless copy pasta:

From what I understand the DS213+ uses the Freescale QorIQ P1022 CPU. Which is based on the PowerPC E500 architecture. According to the spec a bunch of processor instructions emitted by the mono JIT-compiler are simply not supported on E500. Some of those unsupported instructions seem to work anyway, others don't.

It's possible that mono now detects the unsupported instructions and switched to software-based 64-bit instructions, but it's somewhat unlikely.
It'll be critical to validate it before pushing a qoriq package out officially.

@m4tt075 I have a DS-413, this one has a Freescale QoRIQ P1022 CPU and DSM 6.2.1-23824 Update 6 is installed.

@Taloth Thanks for the heads-up. I wasn't aware. And I fully agree, we still need to validate qoriq and everything else, before pushing anything...

@snurbnacnud You might be experiencing what @Taloth outlined above, but let's see. I've created a test build of the most recent Radarr release for you. Here you go:
https://www.dropbox.com/s/gv0feu4cyfo4x2n/radarr_qoriq-6.1_20190301-8.spk?dl=0
Please give it a try and report back what you find...

@Taloth I guess what you are referring to was before https://www.mono-project.com/docs/about-mono/releases/5.0.0/#bring-powerpc-back-to-life ?

Some of the basic jit tests can be found here, you only need to download the specific .cs files mentioned below.
You can compile the testcases with:
mcs basic-float.cs -unsafe TestDriver.cs
mcs basic-long.cs -unsafe TestDriver.cs
mcs basic-math.cs -unsafe TestDriver.cs
then run the tests with eg. mono basic-float.exe
There are over a dozen other tests, but if these 3 succeed then I'd be optimistic.

(mcs may or may not be part of the mono spk. However you can run it on any linux machine and copy the .exe over to the nas and then run the tests with mono.)

@piejanssens Yes, it was in 2014, but qoriq isn't ppc64le. (Also, bootstrapping isn't 'run on', it means mono should be able to compile itself on ppc64le instead of having to be cross-compiled on another platform. I fear you seriously misinterpreted that release note. But I'm happy to be proven wrong.)

@m4tt075
I uninstalled the earlier version and installed the one you supplied, but this one doesn't work either.
Looks like the same issue. Below are the results.

dsmesg:
[118914.961868] init: pkgctl-radarr pre-start process (574) terminated with status 1

radarr_install.log:
Sat Mar 2 00:10:02 CET 2019
===> Step preinst. USER=radarr GROUP=sc-download SHARE_PATH=
Sat Mar 2 00:10:06 CET 2019
===> Step postinst. USER=radarr GROUP=sc-download SHARE_PATH=
Installing service configuration /var/packages/radarr/conf/radarr.sc
Adding 'sc-radarr' to 'sc-download'
Group Name: [sc-download]
Group Type: [AUTH_LOCAL]
Group ID: [65536]
Group Members:
0:[sc-couchpotatoserver]
1:[sc-sabnzbd]
2:[sc-sickbeard-custom]
3:[sc-radarr]
Invoke service_postinst
Granting 'sc-radarr' unix ownership on /volume1/@appstore/radarr/var/.config

radarr.log (excerpt, maybe you need the entire stack trace?):
Debug info from gdb:
mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

Got a SIGILL while executing native code. This usually indicatesa fatal error in the mono runtime or one of the native libraries used by your application.

@m4tt075 Minor question: In the new builds, is "x64" being used as the catch-all for the newer Intel powered boxes like the 918+/1019+? I thought I remember seeing "apollolake" packages in previous builds. I ask because the 1019+ looks like my next purchase. 馃槃

@snurbnacnud Hmh, ok. Could you please download and run the tests @Taloth mentioned in his last post and report back what you find?

@jasonla Yes. The DS1019+ seems to be apollolake, which is binary compatible with the x64 package.

@m4tt075 I'd be happy to run the test, but am not sure how to compile the files on my NAS. Do you know if mcs is part of the spk as @Taloth mentioned? I do not have a Linux machine to compile the executables myself. Could you provide them for me?

@snurbnacnud Yes, mcs is part of the mono installation. You will find it in /usr/local/mono/bin/.
P.S: I'll be out until Tuesday. Please continue posting though. Maybe others can step in while I'm away...

@m4tt075 Thanks for your help so far. Will keep posting though!

@Taloth While running mcs /var/services/homes/xxx/CloudStation/basic-float.cs -unsafe TestDriver.cs from /user/local/mono/bin I receive an error -sh: mcs: command not found.
What am I doing wrong?

Check in /usr/local/mono/bin if mcs exists. It probably doesn't.
Also, if you do find mcs, run from /var/services/homes/xxx/CloudStation and use /usr/local/mono/bin/mcs ... as command instead.

@Taloth Thanks for your quick reply. mcs does exist and running the command from the directory where I had stored basic-float.cs did something. Still the same error though;

Debug info from gdb:

mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

Got a SIGILL while executing native code. This usually indicates a fatal error in the mono runtime or one of the native libraries used by your application.

Aborted

You got that error when running mcs or mono?
Also check mono --version to confirm which version you have, that's something mt4tt075 will need.

PS: SIGILL = Illegal Instruction

@Taloth I got that error while running the mcs command.

Checked mono --version, this is the output:
Mono JIT compiler version 5.18.0.240 (tarball Thu Feb 28 09:27:18 UTC 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: normal
Notifications: epoll
Architecture: ppc
Disabled: none
Misc: softdebug
Interpreter: yes
Suspend: preemptive
GC: sgen (concurrent by default)

Then imho this was a wild goose chase. But thank you for testing.

@m4tt075 When you re-add qoriq to the unsupported platforms list, can you please include a comment, so that the next one that attempts to create qoriq packages knows why he shouldn't?

Sorry for my late reply: just did a full swipe over my DS1815+ (DSM 6.2.1-23824 Update 6) for a clean install. For me, both Sonarr and Radarr packages do not run after installation. I see the following message in the radarr.log:

Sun Mar 3 13:23:34 CET 2019
Starting radarr command env PATH=/usr/local/mono/bin:/volume1/@appstore/radarr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/syno/sbin:/usr/syno/bin:/usr/local/sbin:/usr/local/bin HOME=/volume1/@appstore/radarr/var LD_LIBRARY_PATH=/volume1/@appstore/radarr/lib /usr/local/mono/bin/mono /volume1/@appstore/radarr/share/Radarr/Radarr.exe

Press enter to exit...

In radarr_install.log nothing strange:

Sun Mar 3 13:10:38 CET 2019
===> Step preinst. USER=radarr GROUP=sc-download SHARE_PATH=
Sun Mar 3 13:10:38 CET 2019
===> Step postinst. USER=radarr GROUP=sc-download SHARE_PATH=
Installing service configuration /var/packages/radarr/conf/radarr.sc
Adding 'sc-radarr' to 'sc-download'
Group Name: [sc-download]
Group Type: [AUTH_LOCAL]
Group ID: [65536]
Group Members:
0:[sc-nzbget]
1:[sc-nzbdrone]
2:[sc-radarr]
Invoke service_postinst
Granting 'sc-radarr' unix permissions on /volume1/@appstore/radarr/var/.config

sonarr logfile (nzbdrone.log) seems to be identical to the radarr.log file:

Sun Mar 3 13:19:44 CET 2019
Starting nzbdrone command env PATH=/usr/local/mono/bin:/volume1/@appstore/nzbdrone/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/syno/sbin:/usr/syno/bin:/usr/local/sbin:/usr/local/bin HOME=/volume1/@appstore/nzbdrone/var LD_LIBRARY_PATH=/volume1/@appstore/nzbdrone/lib /usr/local/mono/bin/mono /volume1/@appstore/nzbdrone/share/NzbDrone/NzbDrone.exe

Press enter to exit...

sonarr install log (nzbdrone_install.log) is almost identical to radarr_install.log:

Sun Mar 3 13:10:11 CET 2019
===> Step preinst. USER=nzbdrone GROUP=sc-download SHARE_PATH=
Sun Mar 3 13:10:11 CET 2019
===> Step postinst. USER=nzbdrone GROUP=sc-download SHARE_PATH=
Installing service configuration /var/packages/nzbdrone/conf/nzbdrone.sc
Adding 'sc-nzbdrone' to 'sc-download'
Group Name: [sc-download]
Group Type: [AUTH_LOCAL]
Group ID: [65536]
Group Members:
0:[sc-nzbget]
1:[sc-nzbdrone]
Invoke service_postinst
Granting 'sc-nzbdrone' unix permissions on /volume1/@appstore/nzbdrone/var/.config

I'm a rookie in software engineering but willing to help where possible.

Problem was explained bij @jasonla in an earlier reply https://github.com/SynoCommunity/spksrc/issues/3470#issuecomment-439185611

Will see if a manual installation of sonarr and radarr is possible

Edited to get layout of the reply correct

@Taloth Maybe you are right, but the last attempts apparently have been a long time ago. I'd be willing to inquire a bit more to see what can be done. @snurbnacnud I'd need your help though. Could you please ssh into your box again and install the synogear package? You'll find instructions on how to do that here: https://github.com/SynoCommunity/spksrc/wiki/FAQ-synogear
This will install gdb (and other things) on your Syno. Once this is done, please run mono or mcs again, just like you did before. I'd like you to reproduce the errors you've encountered, hoping that gdb will give us some hints on what actually goes wrong. Please post the error output here. Once we have that, we can inquire with the mono experts and ask for their help.

@Dubbeltje The log files actually look perfectly fine. Could it be that Radarr and Sonarr are actually running? I know from Jackett that it can take a couple of minutes until the package is fully up and running and a pid file is created. Only then it will be reported as "running" in the DSM. Please cross-check whether that is the case. You can also log into your NAS using ssh and execute the top comand. If Radarr and/or Sonarr are running, you'll see a line corresponding to those processes in the table.

If Radarr is indeed not running, I've cross-compiled the latest release for your platform (x64). Please download and manually install it from here:
https://www.dropbox.com/s/qiq14hfmb0a3dzd/radarr_x64-6.1_20190304-8.spk?dl=0

Please report back what you find.

@m4tt075 You're welcome to investigate further, but...
The technical problem is that qoriq is originally a signal processor, it does not have an classic ppc floating-point unit and does not support the ppc floating-point instructions as defined by PowerPC Book E, instead it has an extended instruction set specific to the e500 SPE (Signal Processing Engine) core. That SPE extension is tailored for signal processing, not general computing.
For reference material you can check the PPC e500 wiki page. To quote: "Support for SPESFP (Single Precision Embedded Scalar Floating Point). This is a new floating point unit that is distinct from the classic FPU, the latter of which is lacking in e500v1 and e500v2. SPESFP uses the integer register file. It is not completely IEEE754 compliant."
The normal FPU wasn't added back in till the e500mc core (Both syno models use the e500v2 core instead).
These cpu's were quite literally intended for use in printers, scanners and such due to their low idle power usage.

Given all that, the cpu requires a separate platform to deal with the different floating-point instruction set. Debian had a special port 'powerpcspe'. GCC dropped support for it from their compiler in late 2018.
Nobody is going to add support a 2010 chip, maybe back in 2014 but certainly not now. Adding support for it to mono just to make it work on two DiskStation models is kinda crazy, it requires in-depth knowledge of mono jit process as well as low level processor/assembly experience. Who would invest that kind of time? I'm not even sure if mono would accept a complete pull request for it, much less implement it themselves. Technically it remains possible, it just requires a level of expertise and time that I fear is not available.
I don't like being so negative, but imo it's not worth it for you to spend a large amount of time on a futility.

This is basically the same information I cobbled together back in 2014, but I simply didn't remember it until I revisited the e500 datasheet a couple of days ago. Funny how easily one forgets.

Regardless, if you wish to proceed I do wish you the best of luck.

@Taloth I truly appreciate your thoughts and advice. You certainly understand way more of the caveats and challenges associated with the task than I do. I was _hoping_ that the improvements to mono over the last couple of years could have changed the baseline situation, but from what you write, this is, if not impossible, at least extremely unlikely. As I'm lacking the necessary skills to tackle this and I agree with your assessment not to invest time without reasonable chances of success, I'll add all ppc platforms as unsupported again. Again, my sincere thanks for your guidance! I'll document the problems in the Makefiles as well as on our webpage.
@ymartin59 FYI.

@snurbnacnud @piejanssens Based on the above, no use to inquire further. Sorry to disappoint you, but I hope you'll understand...

Everybody else: Please keep testing! All of the above is only related to the qoriq platform. All other platforms should benefit from the mono update. Again, please keep testing and reporting...

@m4tt075 No problem, I was happy to contribute! Maybe I'll replace my device later this year for one with an architecture supported by mono

@m4tt075, Sonarr and Radarr where definitely not running as also the webUI was down. I waited for 15 minutes without any luck. It was also reported by @jasonla in an earlier reply.

I finally got Sonarr and Radarr working by just uninstalling both and mono 5.18.0.240, installing the current SynoCommunity mono version with Sonarr and Radarr, upgrading Sonarr and Radarr and then manually installing the new mono version (without removing the old one so basically doing an in place upgrade).

Im out now for a couple of days but when home I'll backup Radarr, remove it and manually install your test version.

Anyway, besides of the problem of not being able to start the current available SynoCommunity version of Sonarr and Radarr when the new mono version is installed first, both packages are running smoothly and I do not experience any problems with them after upgrading mono to the new test version (specifically using rarbg as indexer as the old mono versions had a problem connecting over SSL/TLS).

Will keep you posted.

@m4tt075, I uninstalled Radarr (which was updated to the last version) and installed Radarr again from the package center which resulted in not being able to run the package.

A manual installation of your Radarr version (which is the current latest version) runs immediately. So if mono is update in the package center to 5.18.0.240, Radarr has to be updated in the package center also. An old version of Radarr is not compatible with mono 5.18.0.240.

I assume the same happens with Sonarr but I can't test that as there is no version to install manually.

I tried Jackett from the package center but can't get it to run. Don't know if it is related to the mono-version. Will post the jackett.log below. Install log seems fine:

Wed Mar 6 21:09:45 CET 2019
Starting jackett command
Starting Jackett as user sc-jackett

Unhandled Exception:
System.ArgumentNullException: Value cannot be null.
Parameter name: context
at Autofac.ResolutionExtensions.ResolveService (Autofac.IComponentContext context, Autofac.Core.Service service, System.Collections.Generic.IEnumerable1[T] parameters) [0x00003] in <6b74cc51e42041dba63ed788f99f0cbb>:0 at Autofac.ResolutionExtensions.Resolve (Autofac.IComponentContext context, System.Type serviceType, System.Collections.Generic.IEnumerable1[T] parameters) [0x00007] in <6b74cc51e42041dba63ed788f99f0cbb>:0
at Autofac.ResolutionExtensions.Resolve[TService] (Autofac.IComponentContext context, System.Collections.Generic.IEnumerable1[T] parameters) [0x00000] in <6b74cc51e42041dba63ed788f99f0cbb>:0 at Autofac.ResolutionExtensions.Resolve[TService] (Autofac.IComponentContext context) [0x00000] in <6b74cc51e42041dba63ed788f99f0cbb>:0 at Jackett.Engine.get_Logger () [0x00000] in <8b8e6d5e35ce4adf8051460cf82927b5>:0 at JackettConsole.Program+<>c.<Main>b__0_1 (Jackett.Common.Models.Config.ConsoleOptions options) [0x00330] in <b2b006f567cb4b6f9a7fdc92916ee827>:0 at CommandLine.ParserResultExtensions.WithParsed[T] (CommandLine.ParserResult1[T] result, System.Action1[T] action) [0x00011] in <c500474b04314b778910cf182873b214>:0 at JackettConsole.Program.Main (System.String[] args) [0x0002f] in <b2b006f567cb4b6f9a7fdc92916ee827>:0 [ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentNullException: Value cannot be null. Parameter name: context at Autofac.ResolutionExtensions.ResolveService (Autofac.IComponentContext context, Autofac.Core.Service service, System.Collections.Generic.IEnumerable1[T] parameters) [0x00003] in <6b74cc51e42041dba63ed788f99f0cbb>:0
at Autofac.ResolutionExtensions.Resolve (Autofac.IComponentContext context, System.Type serviceType, System.Collections.Generic.IEnumerable1[T] parameters) [0x00007] in <6b74cc51e42041dba63ed788f99f0cbb>:0 at Autofac.ResolutionExtensions.Resolve[TService] (Autofac.IComponentContext context, System.Collections.Generic.IEnumerable1[T] parameters) [0x00000] in <6b74cc51e42041dba63ed788f99f0cbb>:0
at Autofac.ResolutionExtensions.Resolve[TService] (Autofac.IComponentContext context) [0x00000] in <6b74cc51e42041dba63ed788f99f0cbb>:0
at Jackett.Engine.get_Logger () [0x00000] in <8b8e6d5e35ce4adf8051460cf82927b5>:0
at JackettConsole.Program+<>c.

b__0_1 (Jackett.Common.Models.Config.ConsoleOptions options) [0x00330] in :0
at CommandLine.ParserResultExtensions.WithParsed[T] (CommandLine.ParserResult1[T] result, System.Action1[T] action) [0x00011] in :0
at JackettConsole.Program.Main (System.String[] args) [0x0002f] in :0

@Dubbeltje Thanks a lot for testing and reporting. I can confirm that Jackett requires an update too in order to work with mono-5.18. The update has been merged already (#3572), but is not yet published.
Would you mind testing Sonarr and Lidarr compatibility as well? I've just built test packages of the latest releases. You (and everyone else) will find them here:

For completeness:

@m4tt075 Removed Sonarr and install the version from package center to double check it will not run after installation (which indeed it didn't). Installed the version you provided and the service started immediately after installation. Restored a backup successfully and I'm up and running again. So far so good. Don't see any errors in nzbdrone_install.log. I do see some warnings in the nzbdrone.log for some poster files:

[Warn] MediaCoverMapper: File /volume1/@appstore/nzbdrone/var/.config/NzbDrone/MediaCover/9/poster.jpg not found
[Warn] MediaCoverMapper: File /volume1/@appstore/nzbdrone/var/.config/NzbDrone/MediaCover/12/poster.jpg not found
[Warn] MediaCoverMapper: File /volume1/@appstore/nzbdrone/var/.config/NzbDrone/MediaCover/11/poster.jpg not found

And some warning to set date of file but I doubt that is related to the new mono version:

[Warn] UpdateEpisodeFileService: Unable to set date of file [/volume1/tv shows/The Grand Tour (2016)/Season 3/The Grand Tour (2016) - S03E01 - Motown Funk [1080p-WEBDL] [QOQ].mkv]

[v2.0.0.5301] System.IO.IOException: Invalid parameter
at System.IO.File.SetLastWriteTime (System.String path, System.DateTime lastWriteTime) [0x0002a] in <30c1a991558c4064acefa7cee0fc6779>:0
at NzbDrone.Common.Disk.DiskProviderBase.FileSetLastWriteTime (System.String path, System.DateTime dateTime) [0x00048] in :0
at NzbDrone.Core.MediaFiles.UpdateEpisodeFileService.ChangeFileDateToLocalAirDate (System.String filePath, System.String fileDate, System.String fileTime) [0x00058] in :0

[Warn] UpdateEpisodeFileService: Unable to set date of file [/volume1/tv shows/The Grand Tour (2016)/Season 3/The Grand Tour (2016) - S03E02 - Colombia Special (1) [1080p-WEBDL] [METCON].mkv]

[v2.0.0.5301] System.IO.IOException: Invalid parameter
at System.IO.File.SetLastWriteTime (System.String path, System.DateTime lastWriteTime) [0x0002a] in <30c1a991558c4064acefa7cee0fc6779>:0
at NzbDrone.Common.Disk.DiskProviderBase.FileSetLastWriteTime (System.String path, System.DateTime dateTime) [0x00048] in :0
at NzbDrone.Core.MediaFiles.UpdateEpisodeFileService.ChangeFileDateToLocalAirDate (System.String filePath, System.String fileDate, System.String fileTime) [0x00058] in :0

Also installed the Lidarr version you provided. Installation went smoothly. No errors in lidarr_install.log. See just one warning in the lidarr.log about mono not running with --debug switch on:

[Warn] MonoDebugCheck: Mono is not running with --debug switch

Checked the sonarr.log again and see the same MonoDebugCheck warning so nothing serious I guess.

I don't use Lidarr myself so it is not setup or configured to do anything. Maybe someone else can do some more testing on that part.

I'll wait for a newer version of Jacket to test it.

@Dubbeltje I cannot judge the Sonarr warnings really, but it seems those poster files are not part of the installation package. At least I could not find them there. The date setting issue might(!) be related to premission setting. The Sonarr package belongs to the sc-download group, so that group should have RW permissions on your download folders. But otherwise, looking good.

Here test packages for Jackett: https://github.com/m4tt075/spksrc/releases/tag/Jackett-19

EDIT: Updated to newer Jackett-19 test builds.

@m4tt075 Thank you for the Jackett package. Installs and works fine with the new mono version.

Anyway, the warnings in Sonarr about the posters has to do with the restore in which the posters where not back upped or restored. An 'Update Library' fixed that.

The date time issue seems to be a known bug when running on mono according to https://forums.sonarr.tv/t/unable-to-set-date-of-file/4256. I just turned it off as advised.

@Dubbeltje Excellent. It seems we are fine, then. Thanks again for your support!

Everybody: Please keep testing and reporting on mono 5.18 in conjunction with the updated Sonarr, Radarr, Lidarr and Jackett packages. Again, here all the links to the test-builds:

EDIT: Jackett test builds updated above to account for issue #3653.
EDIT2: Sonarr test builds updated to respect and benefit from #3518.

@m4tt075
First of all. Good job! Everything seems to work. Here are my impressions of the first 30 minutes.

Mono:
Installed and started under a minute.
Jackett:
Installed and started under a minute. Successfully updated with restart to latest (0.11.46.0) version. Seems to be functioning normally.
Radarr:
Installed and started under a minute. Restored config.xml and nzbdrone.db via winscp and restarted successfully. Seems to be functioning normally.

I will continue testing and report back in a few days. Is there anything in particular you want me to watch out for?

So far so good. Again, congratulations and sincere thank you for your hard work.

@bolhaskutya Great. Many thanks for testing. Could you (or anybody else) please do me a favor and test Sonarr as well? I've just updated the package ("Sonarr-19") and the link above. Needed to rebase, so would like to make sure it (still) works.

@m4tt075
Sure thing, glad to help. Sonarr also installed and ran under a minute. Everything seems to be working fine at first glance. All the menus are responsive and load their respective pages. The log doesn't show any unusual entries. I don't use Sonarr, so I can't test it as readily and thoroughly as Radarr, but it seems to be good to go!

I'll report back again on all packages in a few days.

@bolhaskutya Excellent. Many thanks.

@m4tt075
I have found a minor bug. Jackett doesn't start if Mono is installed on another volume, due to the way Mono's path is called. This does NOT happen with Radarr/Sonarr.

Jackett's log:
"env: /volume2/@appstore/jackett/../mono/bin/mono: No such file or directory"
The way I see it, Jackett assumes that Mono is in its parent directory (relative path)

I'm no expert, but it can possibly be fixed following Radarr/Sonarr's example in this file:

jackett_x64-6.1_0.11.43-8.spk/scripts/service-setup (Line 34)
_Change this_
MONO_PATH="${SYNOPKG_PKGDEST}/../mono/bin"
_To this_
MONO_PATH="/usr/local/mono/bin"

I don't know if this is valid for other architectures or platforms.

Wow, that's a corner case really and never worked. Anyways, you are right and your proposal should fix the problem. @ymartin59 I'm torn and tired of producing more and more test packages. If you agree, I'd propose to merge and publish first and fixing this later. Or we just change it "at risk". Please advise...

I am publishing already... For MONO_PATH, I have preference for DSM-way: /var/packages/mono/target/bin

Awesome. Just reporting in after 4-5 days of usage. Jackett, Radarr, Sonarr are working great, their logs are clean, communication between Jackett and other packages is perfect. Thumbs up!

@ymartin59 Yes, fine as well. We just have to hardcode the links and not absolute paths to fix this corner case, but we'll keep it for the next round. And again, many thanks for building and publishing. I couldn't have done it with my equipment...
@bolhaskutya Thanks for the confirmation. I'm glad we finally found a package combination that seems to work...

@m4tt075 This PATH issue is a minor trouble as there is a simple work-around: install both packages on same volume.
I have all packages ready and most have been uploaded in spkrepo - may you please check to confirm none is lacking?
But mono packages are just too large and fails with timeout - I have CPUs and disks but my upload bandwidth is too low and so operations timed out. I already tried to tune nginx/uwsgi previously but not perfect yet - if unlucky in worst cases, I may take benefits of a better internet link this week-end.

@ymartin59 Yes. Many thanks. Just checked and it seems all the "mother" packages are complete for 6.1 platforms. Not sure there is still a need for 5.2 builds. Maybe not or only on demand...

W.r.t. your mono "troubles": I actually find it reassuring that I'm not the only one lacking "horsepower" in one way or the other. Good thing is, we can team up: I have 12Mbit/s upload rate on offer. If you fancy a visit to Germany and you bring your machine, more than welcome... :-)

@m4tt075 That is my idea. I will publish 5.2 only on demand.
Have you noticed each mono package is 140 MiB large? 20 MiB more than previous version... Isn't there a way to get it lighter?

I see that the new packages have appeared on the synocommunity.com, but in my synology's package center I only see the old versions (Jackett 0.10.589-7, Radarr 20180303-6) Is that normal?

Yes, they have been uploaded but not yet activated. Mono is still missing because of the problems @ymartin59 mentioned above. All packages need to be activated at once.

Great! Thanks for the info. I installed the latest Jackett and Radarr manually by downloading from synocommunity.com, they both work flawlessly with Mono previously linked in this thread.

I don't know if this is the correct place form my issue below, but after searching for a while I don't see other options.

My DS112+ worked fine until the last update of mono. At this moment Radarr and Sonarr are not working. I've tried several attemps of uninstall / install. Ended on this page and tried the latest packages (mono_88f6281-6.1_5.18.0.240-12 / nzbdrone_88f6281-6.1_20190315-15 and radarr_88f6281-6.1_20190304-8). The result is the same. Neither Sonarr or Radarr starts. The radarr.log contains the following error:
Any idea??

Native stacktrace:

    /usr/local/mono/bin/mono() [0xf1574]
    /usr/local/mono/bin/mono() [0xb30d8]
    /lib/libc.so.6(__default_rt_sa_restorer_v2+0) [0x40179370]
    /lib/libc.so.6(gsignal+0x40) [0x40177fa0]
    /lib/libc.so.6(abort+0x1b0) [0x4017bea8]
    /usr/local/mono/bin/mono() [0x27f53c]
    /usr/local/mono/bin/mono() [0x279fcc]
    /usr/local/mono/bin/mono() [0x294fdc]
    /usr/local/mono/bin/mono(monoeg_assertion_message+0x20) [0x29522c]
    /usr/local/mono/bin/mono() [0x7db1c]
    /usr/local/mono/bin/mono() [0xf7bf8]
    /usr/local/mono/bin/mono() [0xf8488]
    /usr/local/mono/bin/mono() [0x38e50]
    /usr/local/mono/bin/mono() [0xb67ec]
    /usr/local/mono/bin/mono() [0xb6e78]

Pkilling 0x41d56450 from 0x400217c0
Entering thread summarizer pause from 0x400217c0
Finished thread summarizer pause from 0x400217c0.

Waiting for dumping threads to resume

Debug info from gdb:

mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.

@Jayjo4 Not the right place. Please open a new issue. However, before doing this, please do the following:

  1. ssh into your NAS as "admin"
  2. gain root priviledge via "sudo su -" (using the admin password again)
  3. Execute "synogear install" and wait
  4. Reproduce the error above and then post the (hopefully extended) output in the new issue.

Memory leak here was at almost 7GB of ram used. Randomly discovered this. Updated mono to the latest version and not Radarr won't start.

@stormtrooper298 What NAS architecture and firmware do you run?

Hello, any updates available for DS118 NAS ?

Was this page helpful?
0 / 5 - 0 ratings