dietpi-software reinstall 147
dietpi-bugreport
ID | sed -n 5p /DietPi/dietpi/.hw_model
: 3974a20e-b6ca-4207-bf41-bb0f7c9d4f8f@ezekini
Thanks for your report.
I will check your bug report.
@ezekini
Sorry for late reply.
I didn't find the bug report, could you retry and send a (new) report:
systemctl start jackett
dietpi-bugreport
Otherwise the interesting outputs are:
journalctl -u jackett
or journalctl -t jackett
Hi @MichaIng. Here is the bugreport again:
Bug report sent, reference code: 3974a20e-b6ca-4207-bf41-bb0f7c9d4f8f.
BTW, journalctl -u jackett
is very long (I gues you will see it the report), journalctl -t jackett
has no entries.
@ezekini
The last lines of journalctl -u jackett
after crash/service start failure should be sufficient. However will check the bug report now.
jackett 9407 89.5 3.2 53908 22736 ? Rsl 16:39 0:01 /usr/bin/mono -O=-aot /opt/jackett/JackettConsole.exe
β jackett.service - Jackett Daemon (DietPi)
Loaded: loaded (/etc/systemd/system/jackett.service; disabled; vendor preset: enabled)
Active: active (running) since Sat 2019-03-02 16:39:09 -03; 2s ago
Main PID: 9407 (mono)
CGroup: /system.slice/jackett.service
ββ9407 /usr/bin/mono -O=-aot /opt/jackett/JackettConsole.exe
Mar 02 16:39:09 DietPi-rpi systemd[1]: Started Jackett Daemon (DietPi).
[ OK ] DietPi-Services | jackett active (running) since Sat 2019-03-02 16:39:09 -03; 4s ago
Your initial error contains (auto-restart)
. So yeah Jackett cashed randomly and due to systemd unit settings it is attempted to restart automatically. However this obviously did not occur in the session when you sent the bug report.
What I could think of is a termination by the system due to full memory. Please paste free -m
output.
@MichaIng, memory seems fine. Output is:
total used free shared buff/cache available
Mem: 685 323 71 52 290 259
Swap: 1071 0 1071
This behaviour with jackett was not a one time event. If I reboot, the same thing happens. If I restart the service, the same. Right now it is basically not functioning.
@ezekini
Hmm could you send a bug report right after the error occurred? As said in the bug report I checked there was no error visible.
Also interesting that DietPi-Services shows [ OK ]
even that it shows an exit code π€:
[ OK ] DietPi-Services | jackett activating (auto-restart) (Result: exit-code) since Tue 2019-02-26 11:28:36 -03; 4s ago
Ah because systemctl status jackett
does not show "failed" string: https://github.com/MichaIng/DietPi/blob/dev/dietpi/dietpi-services#L590
I have reinstalled using dietpi-software reinstall appID and it is working now. If the error reappears, we can investigate further, for now we can close this so you can dedicate time to more important stuffπ .
@ezekini
Ah great, jep perhaps simply a faulty/failed install the first time.
@MichaIng
This issue and #2328 could be related to Jackett's built-in updater leaving the service in a broken state. I say that since every time an update is released, Jackett fails to start on my machine. This would also explain why these issues are difficult to reproduce and that reinstalls are fixing them.
This is my error from journalctl -u jackett.service
, which started today around the time of the release of Jackett v0.11.295. Note that the version in the logs is v0.11.279.0
.
May 07 21:17:33 DietPi mono[7821]: 05-07 21:17:33 Info Starting Jackett v0.11.279.0
May 07 21:17:35 DietPi mono[7821]: 05-07 21:17:35 Error System.MissingMethodException: Method not found: AutoMapper.IMappingExpression`2<!0, !1> AutoMapper.IMappingExpression`2.AfterMap(System.Action`2<!0, !1>)
May 07 21:17:35 DietPi mono[7821]: at AutoMapper.MapperConfiguration.Build (System.Action`1[T] configure) [0x00006] in <8645827e567f4e9e927a7dd252e7ec3a>:0
May 07 21:17:35 DietPi mono[7821]: at AutoMapper.MapperConfiguration..ctor (System.Action`1[T] configure) [0x00000] in <8645827e567f4e9e927a7dd252e7ec3a>:0
May 07 21:17:35 DietPi mono[7821]: at AutoMapper.Mapper.Initialize (System.Action`1[T] config) [0x00000] in <8645827e567f4e9e927a7dd252e7ec3a>:0
May 07 21:17:35 DietPi mono[7821]: at Jackett.Server.Helper.InitAutomapper () [0x00009] in <9c1b87b5e9e74c5086bc89d8846d979e>:0
May 07 21:17:35 DietPi mono[7821]: at Jackett.Server.Helper.Initialize () [0x00007] in <9c1b87b5e9e74c5086bc89d8846d979e>:0
May 07 21:17:35 DietPi mono[7821]: at Jackett.Server.Startup.ConfigureServices (Microsoft.Extensions.DependencyInjection.IServiceCollection services) [0x00171] in <9c1b87b5e9e74c5086bc89d8846d979e>:0
May 07 21:17:35 DietPi mono[7821]: at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
May 07 21:17:35 DietPi mono[7821]: at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <6649516e5b3542319fb262b421af0adb>:0
I've correlated Jackett breaking every time a new version is released, but the error is not always the same. Before today I would usually get an error about a missing file such as in this issue: https://github.com/Jackett/Jackett/issues/3635. Checking /opt/jackett/
for the file would usually show that the file existed, but with a filesize of 0.
Remember that I was originally running v0.11.279.0 and the service is still reportedly trying to start v0.11.279.0. /opt/jackett/.config/Jackett/updater.txt
shows this update attempt to v0.11.295.0 from earlier today:
2019-05-07 18:40:20.8800 Info Jackett Updater v0.11.295.0
2019-05-07 18:40:20.9515 Info Options "--Path" "/opt/jackett" "--Type" "Console" "--Args" " JackettConsole.exe " "--KillPids" "6973"
2019-05-07 18:40:20.9573 Info Jackett variant: Mono
2019-05-07 18:40:21.2065 Info Finding files in: /tmp/JackettUpdate-v0.11.295-636928512143410940/Jackett/
2019-05-07 18:40:21.2455 Info Attempting to copy AngleSharp.dll from source: /tmp/JackettUpdate-v0.11.295-636928512143410940/Jackett/AngleSharp.dll to destination: /opt/jackett/AngleSharp.dll
2019-05-07 18:40:21.2493 Info Copied AngleSharp.dll
...
2019-05-07 18:40:21.4265 Info Attempting to copy classix.yml from source: /tmp/JackettUpdate-v0.11.295-636928512143410940/Jackett/Definitions/classix.yml to destination: /opt/jackett/Definitions/classix.yml
2019-05-07 18:40:21.4265 Info Copied classix.yml
2019-05-07 18:40:21.4265 Info Attempting to copy concen.yml from source: /tmp/JackettUpdate-v0.11.295-636928512143410940/Jackett/Definitions/concen.yml to destination: /opt/jackett/Definitions/concen.yml
2019-05-07 18:40:21.4265 Error System.UnauthorizedAccessException: Access to the path "/tmp/JackettUpdate-v0.11.295-636928512143410940/Jackett/Definitions/concen.yml" or "/opt/jackett/Definitions/concen.yml" is denied.
at System.IO.File.Copy (System.String sourceFileName, System.String destFileName, System.Boolean overwrite) [0x00192] in <6649516e5b3542319fb262b421af0adb>:0
at Jackett.Updater.Program.CopyUpdateFile (System.String jackettDestinationDirectory, System.String fullSourceFilePath, System.String updateSourceDirectory, System.Boolean previousAttemptFailed) [0x00162] in <a241fc80b1fb463bb7$
at System.IO.File.Copy (System.String sourceFileName, System.String destFileName, System.Boolean overwrite) [0x00192] in <6649516e5b3542319fb262b421af0adb>:0
at Jackett.Updater.Program.CopyUpdateFile (System.String jackettDestinationDirectory, System.String fullSourceFilePath, System.String updateSourceDirectory, System.Boolean previousAttemptFailed) [0x00162] in <a241fc80b1fb463bb7$
2019-05-07 18:40:21.4445 Info Attempting to copy concen.yml from source: /tmp/JackettUpdate-v0.11.295-636928512143410940/Jackett/Definitions/concen.yml to destination: /opt/jackett/Definitions/concen.yml
2019-05-07 18:40:21.4445 Info The first attempt copying file: concen.ymlfailed. Retrying and will delete old file first
2019-05-07 18:40:21.4445 Info /opt/jackett/Definitions/concen.yml was found
Notice that the log ends with the attempt on concen.yml
, but there should be many more files needing to be copied.
My assumption here is that perhaps the Jackett service is restarting before the updater has a chance to install the new version. Otherwise, perhaps the updater is just broken and this should be an issue on the Jackett repo.
I tried to have a look through the Jackett repo for clues and the only difference I can find is that the recommended service runs Jackett with /usr/bin/mono --debug /opt/jackett/JackettConsole.exe --NoRestart
(link), versus DietPi's /usr/bin/mono -O=-aot /opt/jackett/JackettConsole.exe
(link). Perhaps --NoRestart
is important here for the updater?
Another interesting thing I've noticed is that sometimes the /var/log
tmpfs gets full (currently allocated 50MB) - namely with Sonarr logs (172KB after deleting Sonarr logs and restarting the Sonarr service). Perhaps this has an effect on the updater, though I have no evidence to support this claim.
@DeathIsUnknown
Many thanks for report and investigation.
The failing part is
Error System.UnauthorizedAccessException: Access to the path "/tmp/JackettUpdate-v0.11.295-636928512143410940/Jackett/Definitions/concen.yml" or "/opt/jackett/Definitions/concen.yml" is denied.
which is strange since the other files just work well. Since everything runs as "jackett" user all files should be owned by it including the existing Jackett install dir as well as the downloaded update, which is proven until concen.yml
π€.
However could you please verify permission:
ls -al /opt/jackett /tmp/JackettUpdate*
After failure it tries to copy concen.yml a second time but then stops. The question is, if the daemon stops and then the systemd unit stops because of this, of indeed the updater restarts the daemon, which is expected to fail, since the systemd unit stops when the main process exits, preventing it from starting up again.
This docs btw contain -NoRestart as well: https://www.htpcguides.com/install-jackett-ubuntu-15-x-for-custom-torrents-in-sonarr/
And here: https://github.com/Jackett/Jackett/wiki/systemd-service
Okay I will do some test install to verify the option and update behaviour.
.... takes a while... mono compiling takes forever on VM π
Hmm another topic:
Standalone version of Jackett is now available - Mono not required
To upgrade to the standalone version of Jackett, click here for install instructions. Upgrading is straight forward, simply install the standalone version and your indexers/configuration will carry over. Benefits include: increased performance, improved stability and no dependency on Mono.
--NoRestart Don't restart after update
Makes sense, although Restart=always
in systemd should have it restarted automatically. And it says "after update", so the update should have already been finished, which fails as above.
@DeathIsUnknown
Dumb question: Where do I find the internal updater (in web UI)? Or is it a separate binary that needs to be executed manually?
I can check for updates and the logs reports it available, but can't find any "update now" button. There is an auto-update option, so does this mean it does it as scheduled maintenance job but there is no way to force it now?
Indeed there seems to be no way to force an update via UI currently: https://github.com/Jackett/Jackett/issues/4321
I added the --NoRestart
option which is also reported on other GitHub issues to be required with systemd: https://github.com/MichaIng/DietPi/commit/df2177a0dbf9675544349f255b83d7276a3a4d50
Made some other enhancements as well, e.g. sandboxing the service. Requires some testing but should work fine.
But I am still waiting for the auto update to occur, to verify the error and reason, as well as verify the --NoRestart
indeed fixes this.
Testing the standalone install (non-mono version) now as well. If this works well, we will migrate existing installs, since configs migration is stated to be safe.
Okay, the new standalone version (available since Jackett v0.11) will be installed on non-ARMv6 devices. Sadly there is no ARMv6 binary available, so these need to stay with mono: https://github.com/MichaIng/DietPi/pull/2773
Iβd be interested to know what effect /usr/bin/mono --optimize=all
(Recommended in your link https://github.com/Jackett/Jackett/wiki/systemd-service) has on performance in comparison to -O=-aot
.
I just remember we had this once but it caused issues. However with the standalone binary it doesn't matter anymore, as it runs without mono.
I'm unable to run ls -al /opt/jackett /tmp/JackettUpdate*
on my broken instance since I've fixed the issue and tmp
has been cleared. Sorry! Thanks for looking into this - hopefully we've solved it. π
@DeathIsUnknown
I could still not verify that using the internal updater does not break things since I couldn't find a way to use/force it.
I ran one instance for > one hour with auto updater enabled and it finds/logs a new version when checking via web UI but how do I force it to apply the update?
@MichaIng
"Check for updates" from the web UI seemed to trigger the update for me. Here's journalctl:
May 11 00:14:58 DietPi mono[30435]: 05-11 00:14:58 Info Jackett variant: Mono
May 11 00:14:59 DietPi mono[30435]: 05-11 00:14:59 Info New release found. Current: v0.11.295 New: v0.11.316
May 11 00:15:24 DietPi mono[30435]: 05-11 00:15:24 Info Starting updater: mono /tmp/JackettUpdate-v0.11.316-636931305191255460/Jackett/JackettUpdater.exe --Path "/opt/jackett" --Type "Console" --Args " JackettConsole.exe " --KillPids "30435"
May 11 00:15:24 DietPi mono[30435]: 05-11 00:15:24 Info Updater started process id: 31246
May 11 00:15:24 DietPi mono[30435]: 05-11 00:15:24 Info Exiting Jackett..
May 11 00:15:25 DietPi mono[30435]: Application is shutting down...
May 11 00:15:30 DietPi systemd[1]: jackett.service: Service hold-off time over, scheduling restart.
May 11 00:15:30 DietPi systemd[1]: Stopped Jackett Daemon (DietPi).
May 11 00:15:30 DietPi systemd[1]: Started Jackett Daemon (DietPi).
May 11 00:15:32 DietPi mono[31264]: 05-11 00:15:32 Info Starting Jackett v0.11.295.0
May 11 00:15:36 DietPi mono[31264]: 05-11 00:15:36 Info Environment version: 4.0.30319.42000 (/usr/lib/mono/4.5)
May 11 00:15:36 DietPi mono[31264]: 05-11 00:15:36 Info OS version: Unix 5.0.0.1105 (64bit OS) (64bit process)
May 11 00:15:36 DietPi mono[31264]: 05-11 00:15:36 Info Jackett variant: Mono
May 11 00:15:36 DietPi mono[31264]: 05-11 00:15:36 Info ThreadPool MaxThreads: 600 workerThreads, 200 completionPortThreads
May 11 00:15:36 DietPi mono[31264]: 05-11 00:15:36 Info App config/log directory: /opt/jackett/.config/Jackett
May 11 00:15:36 DietPi mono[31264]: 05-11 00:15:36 Info issue: Debian GNU/Linux 9 \n \l
May 11 00:15:36 DietPi mono[31264]: 05-11 00:15:36 Info mono version: 5.20.1.19 (tarball Thu Apr 11 09:09:09 UTC 2019)
May 11 00:15:36 DietPi mono[31264]: 05-11 00:15:36 Info TrustedRootCertificates count: 160
May 11 00:15:36 DietPi mono[31264]: 05-11 00:15:36 Info Using HTTP Client: HttpWebClient
May 11 00:15:38 DietPi mono[31264]: 05-11 00:15:38 Info Loading Cardigann definitions from: /opt/jackett/.config/cardigann/definitions/, /etc/xdg/cardigan/definitions/, /opt/jackett/Definitions
May 11 00:15:38 DietPi mono[31264]: 05-11 00:15:38 Info Loading Cardigann definition /opt/jackett/Definitions/1337x.yml
May 11 00:15:38 DietPi mono[31264]: 05-11 00:15:38 Info Loading Cardigann definition /opt/jackett/Definitions/2fast4you.yml
...
May 11 00:15:48 DietPi mono[31264]: 05-11 00:15:48 Info Adding aggregate indexer
May 11 00:15:48 DietPi mono[31264]: 05-11 00:15:48 Info Deleting JackettUpdate temp files from /tmp/JackettUpdate-v0.11.316-636931305191255460
May 11 00:15:50 DietPi mono[31264]: Hosting environment: Production
May 11 00:15:50 DietPi mono[31264]: Content root path: /opt/jackett/Content
May 11 00:15:50 DietPi mono[31264]: Now listening on: http://[::]:9117
May 11 00:15:50 DietPi mono[31264]: Application started. Press Ctrl+C to shut down.
After the updater was triggered, Jackett started back up again with the version number still reporting 0.11.295.0 instead of the downloaded 0.11.316.0.
After stopping Jackett, adding --NoRestart
to the service, starting the service, then checking for updates (twice, whoops):
May 11 00:25:48 DietPi mono[31840]: 05-11 00:25:48 Info New release found. Current: v0.11.295 New: v0.11.316
May 11 00:26:16 DietPi mono[31840]: 05-11 00:26:16 Info Starting updater: mono /tmp/JackettUpdate-v0.11.316-636931311706428280/Jackett/JackettUpdater.exe --Path "/opt/jackett" --Type "Console" --Args " JackettConsole.exe --NoRestart" --KillPids "31840" --NoRestart
May 11 00:26:16 DietPi mono[31840]: 05-11 00:26:16 Info Updater started process id: 31911
May 11 00:26:16 DietPi mono[31840]: 05-11 00:26:16 Info Jackett variant: Mono
May 11 00:26:17 DietPi mono[31840]: 05-11 00:26:17 Info Jackett Updater v0.11.316.0
May 11 00:26:17 DietPi mono[31840]: 05-11 00:26:17 Info Options "--Path" "/opt/jackett" "--Type" "Console" "--Args" " JackettConsole.exe --NoRestart" "--KillPids" "31840" "--NoRestart"
May 11 00:26:17 DietPi mono[31840]: 05-11 00:26:17 Info Jackett variant: Mono
May 11 00:26:17 DietPi mono[31840]: 05-11 00:26:17 Info New release found. Current: v0.11.295 New: v0.11.316
May 11 00:26:17 DietPi mono[31840]: 05-11 00:26:17 Info Finding files in: /tmp/JackettUpdate-v0.11.316-636931311706428280/Jackett/
May 11 00:26:18 DietPi mono[31840]: 05-11 00:26:18 Info Attempting to copy AngleSharp.dll from source: /tmp/JackettUpdate-v0.11.316-636931311706428280/Jackett/AngleSharp.dll to destination: /opt/jackett/AngleSharp.dll
May 11 00:26:18 DietPi mono[31840]: 05-11 00:26:18 Info Copied AngleSharp.dll
...
May 11 00:26:18 DietPi mono[31840]: 05-11 00:26:18 Info Attempting to copy System.Net.WebSockets.Client.dll from source: /tmp/JackettUpdate-v0.11.316-636931311706428280/Jackett/System.Net.WebSockets.Client.dll to destination: /opt/jackett/System.Net.WebSockets.Client.dll
May 11 00:26:18 DietPi mono[31840]: 05-11 00:26:18 Info Copied System.Net.WebSockets.Client.dll
May 11 00:26:24 DietPi systemd[1]: jackett.service: Service hold-off time over, scheduling restart.
May 11 00:26:24 DietPi systemd[1]: Stopped Jackett Daemon (DietPi).
May 11 00:26:24 DietPi systemd[1]: Started Jackett Daemon (DietPi).
@DeathIsUnknown
Strange, I think my instance didn't trigger the update. I will recheck.
However interesting that the service is still restarted with --NoRestart
. Was the update at least successful then?
jackett.service: Service hold-off time over, scheduling restart.
Hmm, some timeout is reached triggering systemd to restart the service. If this makes the update still break, possibly we need to increase some timeouts there and/or report the issue to Jackett to be fixed, respectively their example systemd units to be updated.
@MichaIng
The update was successful this time, though I'm not sure if I should attribute it to restarting the service and re-triggering the update or to adding --NoRestart
.
@MichaIng
Thanks for following this issue, but it seems to still exist in the latest DietPi update. I am going to assume this is on Jackett's side since users on their repo are having the same issue. See https://github.com/Jackett/Jackett/issues/5285 and https://github.com/Jackett/Jackett/issues/5208.
However interesting that the service is still restarted with
--NoRestart
. Was the update at least successful then?jackett.service: Service hold-off time over, scheduling restart.
Hmm, some timeout is reached triggering systemd to restart the service. If this makes the update still break, possibly we need to increase some timeouts there and/or report the issue to Jackett to be fixed, respectively their example systemd units to be updated.
I would assume these lines in jackett.service
to be the culprit, but they are also recommended by the Jackett docs.
Restart=always
RestartSec=5
I'll follow the issue on the other repo and update here if a solution is found.
@DeathIsUnknown
Damn thing. Yeah actually it makes somehow sense:
Restart=always
Restart=always
does not really solve it, because systemd then still thinks that the service is dead and will not recognise another process being spinned up (most probably). So even the Jackett would be up then, the systemd service will report a failure. This also should be the reason why --NoRestart
is required and then Restart=always
. So after update, Jackett kills itself (not triggering a restart itself) and systemd recognises process down and restarts the service.According to your logs, at least I cannot see anything not working about that
May 11 00:15:24 DietPi mono[30435]: 05-11 00:15:24 Info Updater started process id: 31246
May 11 00:15:24 DietPi mono[30435]: 05-11 00:15:24 Info Exiting Jackett..
May 11 00:15:25 DietPi mono[30435]: Application is shutting down...
May 11 00:15:30 DietPi systemd[1]: jackett.service: Service hold-off time over, scheduling restart.
May 11 00:15:30 DietPi systemd[1]: Stopped Jackett Daemon (DietPi).
May 11 00:15:30 DietPi systemd[1]: Started Jackett Daemon (DietPi).
May 11 00:15:32 DietPi mono[31264]: 05-11 00:15:32 Info Starting Jackett v0.11.295.0
However the update still results in a failing service? Is there any second mono/jackett process up? pgrep -f 'jackett'
Ref: https://dietpi.com/phpbb/viewtopic.php?p=18240#p18240
"Waiting for user reply" = Waiting for Jackett devs to fix this or find a systemd unit that works well with the internal updater: https://github.com/Jackett/Jackett/issues/5208#issuecomment-499315248
Is this fixed? I uninstalled jackett because it restart itself every second, temps were high
@Dr0bac
As of the linked issue on Jackett repo, nope, not fixed. Please disable the internal updater for now.
Referencing all related issues:
Seems to not being fixed soon. We need to find a way to disable the internal updater code-wise for now and prepare our install code to provide a reliable update method via dietpi-software reinstall 147
which can be used by our users on a regular basis.
@MichaIng Test the official fix => https://github.com/Jackett/Jackett/issues/5208#issuecomment-547709864
@ngosang
Jep already did and suggested some minor enhancements. Will add it with next DietPi release as well which will be very soon as kinda hotfix release, reasonable due to some other issues our side.
Note to self: mono-xsp4
(webserver) sufficient instead of mono-complete
.
Implemented workaround: https://github.com/MichaIng/DietPi/commit/9a543c7210dcdeabb5f382f261a6d27efcdc8f6d
Reinstall with next update to fix for existing installs: https://github.com/MichaIng/DietPi/commit/c58e87cc7e3150561af054dce1a178c5ad2848db
Changelog: https://github.com/MichaIng/DietPi/commit/61d00eacde50836f90757994cfff3718d1c03a15
I've experienced this bug twice now. (reinstalled the software the first time) Not sure how to help, but I sent in debug info - reference code: 9fe2513d-da5f-4b3d-9cdb-1226033186aa
@EdEstes
Note that the workaround has been applied with v6.27. Apply the dev
branch for the fix: https://github.com/MichaIng/DietPi/blob/master/BRANCH_SYSTEM.md
Otherwise beta will be released most likely tomorrow and release until end of week.
Most helpful comment
@EdEstes
Note that the workaround has been applied with v6.27. Apply the
dev
branch for the fix: https://github.com/MichaIng/DietPi/blob/master/BRANCH_SYSTEM.mdOtherwise beta will be released most likely tomorrow and release until end of week.