DietPi-Software | Sonarr import to FS without UNIX-permission fails, regardless of mount options

[shlatchz]

Creating a bug report/issue

Required Information

• DietPi version:

!/bin/bash

G_DIETPI_VERSION_CORE=6
G_DIETPI_VERSION_SUB=26
G_DIETPI_VERSION_RC=3
G_GITBRANCH='master'
G_GITOWNER='MichaIng'

• Distro version:

stretch
9.11

• Kernel version:

Linux DietPi 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64 GNU/Linux

• SBC device:

Native PC (x86_64)

Additional Information (if applicable)

Bug Report ID: 3b0cd5c5-8000-40c6-ac83-14972cc6877b

Steps to reproduce

Download anything using Deluge + Jackett

Actual behaviour

When Sonarr tries to fetch the downloaded data, I receive:

Couldn't import episode /mnt/PIHDD/deluge/downloads/XXXX/XXXX.mkv: Access to the path is denied.

System.UnauthorizedAccessException: Access to the path is denied. ---> System.IO.IOException: Operation not permitted
--- End of inner exception stack trace ---
at Interop.ThrowExceptionForIoErrno (Interop+ErrorInfo errorInfo, System.String path, System.Boolean isDirectory, System.Func2[T,TResult] errorRewriter) [0x00014] in <285579f54af44a2ca048dad6be20e190>:0 at Interop.CheckIo (System.Int64 result, System.String path, System.Boolean isDirectory, System.Func2[T,TResult] errorRewriter) [0x0000a] in <285579f54af44a2ca048dad6be20e190>:0
at Interop.CheckIo (System.Int32 result, System.String path, System.Boolean isDirectory, System.Func2[T,TResult] errorRewriter) [0x00000] in <285579f54af44a2ca048dad6be20e190>:0 at System.IO.FileSystem.CopyFile (System.String sourceFullPath, System.String destFullPath, System.Boolean overwrite) [0x0005c] in <285579f54af44a2ca048dad6be20e190>:0 at System.IO.File.Copy (System.String sourceFileName, System.String destFileName, System.Boolean overwrite) [0x00062] in <285579f54af44a2ca048dad6be20e190>:0 at NzbDrone.Common.Disk.DiskProviderBase.CopyFileInternal (System.String source, System.String destination, System.Boolean overwrite) [0x00000] in <4d680ffa10414c3ab7e1372769e11d73>:0 at NzbDrone.Mono.Disk.DiskProvider.CopyFileInternal (System.String source, System.String destination, System.Boolean overwrite) [0x00076] in <9c1bc51989834744a8646618373e03e7>:0 at NzbDrone.Common.Disk.DiskProviderBase.CopyFile (System.String source, System.String destination, System.Boolean overwrite) [0x000bc] in <4d680ffa10414c3ab7e1372769e11d73>:0 at NzbDrone.Common.Disk.DiskTransferService.TryCopyFileTransactional (System.String sourcePath, System.String targetPath, System.Int64 originalSize) [0x00108] in <4d680ffa10414c3ab7e1372769e11d73>:0 at NzbDrone.Common.Disk.DiskTransferService.TransferFile (System.String sourcePath, System.String targetPath, NzbDrone.Common.Disk.TransferMode mode, System.Boolean overwrite, NzbDrone.Common.Disk.DiskTransferVerificationMode verificationMode) [0x0034a] in <4d680ffa10414c3ab7e1372769e11d73>:0 at NzbDrone.Common.Disk.DiskTransferService.TransferFile (System.String sourcePath, System.String targetPath, NzbDrone.Common.Disk.TransferMode mode, System.Boolean overwrite, System.Boolean verified) [0x0000e] in <4d680ffa10414c3ab7e1372769e11d73>:0 at NzbDrone.Core.MediaFiles.EpisodeFileMovingService.TransferFile (NzbDrone.Core.MediaFiles.EpisodeFile episodeFile, NzbDrone.Core.Tv.Series series, System.Collections.Generic.List1[T] episodes, System.String destinationFilePath, NzbDrone.Common.Disk.TransferMode mode) [0x0012c] in <96f52d5e5b95442a89ae3bbf62e59664>:0
at NzbDrone.Core.MediaFiles.EpisodeFileMovingService.CopyEpisodeFile (NzbDrone.Core.MediaFiles.EpisodeFile episodeFile, NzbDrone.Core.Parser.Model.LocalEpisode localEpisode) [0x000a6] in <96f52d5e5b95442a89ae3bbf62e59664>:0
at NzbDrone.Core.MediaFiles.UpgradeMediaFileService.UpgradeEpisodeFile (NzbDrone.Core.MediaFiles.EpisodeFile episodeFile, NzbDrone.Core.Parser.Model.LocalEpisode localEpisode, System.Boolean copyOnly) [0x00167] in <96f52d5e5b95442a89ae3bbf62e59664>:0
at NzbDrone.Core.MediaFiles.EpisodeImport.ImportApprovedEpisodes.Import (System.Collections.Generic.List`1[T] decisions, System.Boolean newDownload, NzbDrone.Core.Download.DownloadClientItem downloadClientItem, NzbDrone.Core.MediaFiles.EpisodeImport.ImportMode importMode) [0x00281] in <96f52d5e5b95442a89ae3bbf62e59664>:0

Extra details

• ...

[MichaIng]

@shlatchz
Many thanks for your report.

Looks like a permissions issue. Please assure that the Deluge downloads are created with group "dietpi" group and 002 umask, so the "sonarr" user (which is "dietpi" group member) has write permissions to import them.

Deluge created file permissions and Sonarr group can be derived via:

systemctl cat deluged
systemctl cat sonarr

[shlatchz]

@MichaIng files have root:root as owner and group
All the files are on an external HD mount

[MichaIng]

@shlatchz
That explains Sonarr access issues then. If those are on external drive, which file system is it? In case UNIX permissions are not supported, root:root would be hardcoded and no chance then to get write access for any non-root user. You would need to edit the fstab then to mount the drive e.g. as dietpi:dietpi with mode=775 or such.

But just to rule service issues our, could you paste output of the two commands above?

[shlatchz]

root@DietPi:~# systemctl cat deluged

# /etc/systemd/system/deluged.service
[Unit]
Description=Deluge Daemon (DietPi)
Documentation=man:deluged

[Service]
User=debian-deluged
Group=dietpi
UMask=002
ExecStart=/usr/bin/deluged -d -l /var/log/deluged/daemon.log -L warning
Restart=on-failure
TimeoutStopSec=300

[Install]
WantedBy=multi-user.target

root@DietPi:~# systemctl cat sonarr

# /etc/systemd/system/sonarr.service
[Unit]
Description=Sonarr (NzbDrone) Daemon (DietPi)
After=network.target

[Service]
User=sonarr
Group=dietpi
ExecStart=/usr/bin/mono -O=-aot /opt/NzbDrone/NzbDrone.exe -nobrowser -data=/mnt/dietpi_userdata/sonarr

[Install]
WantedBy=multi-user.target

As for what you said, it doesn't explain why it worked perfectly before the update.
Nothing has changed with my setup. It always worked with an external HD mount.

cat /etc/fstab

UUID=0ECF-1050 /mnt/PIHDD auto defaults,noatime,rw,nofail,x-systemd.automount 0 0

blkid

/dev/sda2: LABEL="Multimedia" UUID="0ECF-1050" TYPE="exfat" PARTLABEL="Basic data partition" PARTUUID="21a98c36-0f68-4db6-b1b0-d98f8790a176"

fdisk -l

Disk /dev/sda: 7.3 TiB, 8001563221504 bytes, 15628053167 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 8069607F-EEA7-4D53-AC30-862E94964FAC

Device      Start         End     Sectors  Size Type
/dev/sda1      34      262177      262144  128M Microsoft reserved
/dev/sda2  264192 15628052479 15627788288  7.3T Microsoft basic data

[shlatchz]

I changed fstab to

UUID=0ECF-1050 /mnt/PIHDD auto defaults,noatime,rw,nofail,x-systemd.automount,uid=1001,gid=1001,mode=775 0 0

and I still get access denied even though the files have dietpi:dietpi owner and group

-rwxrwxrwx 1 dietpi dietpi 1450813771 Oct 19 12:59 [HorribleSubs] Boku no Hero Academia - 65 [1080p].mkv

[shlatchz]

@MichaIng any ideas?

[MichaIng]

@shlatchz
Hmm, now I am also out of ideas. Btw. shouldn't be dietpi user and group both UID 1000? However the name is shown correctly 🤔. You need to have the drive exFAT to allow access from Windows, right? If it's only this, you could try NTFS instead, where UNIX permissions are emulated by the ntfs-3g package. I am not sure whether Deluge and/or Sonarr are trying to chown/chmod files, which then still fails on exFAT.

Just as last try with exFAT: apt install exfat-fuse exfat-utils
Running Sonarr and Deluge as root user, as last resort test, to be sure initial access must succeed:

sed -Ei 's/^(User=|Group=)/#\1/' /etc/systemd/system/deluged.service
sed -Ei 's/^(User=|Group=)/#\1/' /etc/systemd/system/sonarr.service
systemctl daemon-reload
systemctl restart deluge
systemctl restart sonarr

If that still fails, the issue must be failing chmod/chown support by exFAT. Still strange that it worked well before, probably some kernel and/or exFAT package upgrade broke it e.g. with stricter rules or such 🤔.

[shlatchz]

@MichaIng

sed -Ei 's/^(User=|Group=)/#\1/' /etc/systemd/system/deluged.service

caused errors

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/deluge/core/rpcserver.py", line 262, in dispatch
    ret = component.get("AuthManager").authorize(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/deluge/core/authmanager.py", line 95, in authorize
    raise BadLoginError("Password does not match")
BadLoginError: Password does not match
[ERROR   ] 19:38:58 rpcserver:268 Password does not match

so I skipped it
I only changed sonarr.service and it worked!~

[MichaIng]

@shlatchz
Okay, I am not yet sure why Sonarr fails to access as "sonarr" user and "dietpi" group, while Deluge can with "debian-deluged" user and "dietpi" group. I see no difference from permissions point of view 🤔. However, glad it works now.

BadLoginError: Password does not match
[ERROR ] 19:38:58 rpcserver:268 Password does not match

Ah I am pretty sure this is due to changed user home dir, hence new configs are created and web interface RPC server credentials do not match anymore. So for this the HOME env var would need to be overridden by systemd unit to /mnt/dietpi_userdata/deluge.

Regardless, if it works now, no need to experiment, using root is anyway worst case workaround.

---

I'm trying to replicate. Everything went well on ext4 file system. exFAT btw produces a lot of additional CPU usage... really no good choice on Linux if possible to avoid... However Deluge download works well, import to Sonarr on first attempt indeed has not been done, although I am not 100% sure if it was due to removal and re-search of the same file, which is then ignored or such? Will retry tomorrow after fresh VM restart.

[MichaIng]

Okay did another test today with a new file to download. Everything went very well with Deluge downloading to external exFAT partition and Sonarr grabbing it, moving to internal ext4 partition. Now I want to try exFAT to exFAT.

EDIT

Verified:

[v2.0.0.5338] System.UnauthorizedAccessException: Access to the path is denied.

• Very strange since the whole mount point has 777 permissions by default.
• Sonarr was even able to create the series/season directory, but not to import the file. So write permissions for Sonarr process are there.
• Import to ext4 file system with dietpi:dietpi 775 permissions worked as expected.

---

🈯️ Mounted drive with sonarr:dietpi (777) => Import succeeds
🈴 Mounted drive with dietpi:dietpi (777) => Import fails, what matches @shlatchz case with same + explicit 775 permissions.

• So finally it seems to only work, if Sonarr runs as root, or the exFAT file system is mounted with Sonarr run user.
• Both is no acceptable as default that we can ship, either for security or for usability reasons.
• Since the user itself definitely has full R/W access (can be also verified via sudo -u sonarr eval 'echo test > /mnt/exfat/test'), it is some Mono and/or Sonarr internal bug (or strange authentication design), which should be reported to Sonarr devs in the first place.

---

Reported: https://github.com/Sonarr/Sonarr/issues/3364
Lets see if the guys over there have some idea 🙂.

[MichaIng]

Seems to be a mono v6 specific issue indeed: https://github.com/mono/mono/issues/16573

[Taloth]

Stick with mono 5.20 and steer clear from mono 6.x

[MichaIng]

@Taloth
Not an option. Sticking to older versions just makes us run into future deprecations, users asking to ship current version, running into security issues when support fades etc. IMO, as long as its anyhow possible, going forward and fix issues on current versions is the only way we can go.

And aside from that, FAT fs issues are really not something we should base our priorities on. This is still a Linux distro with native support for file systems that support UNIX permissions and do not have this issue. Linux => Windows data transfer should not be done by plugging the drive around, but via network drives instead 😉.

[Taloth]

The problem is that since mono 6.0 they ported over code from dotnet core. And broke Copy/Move operations for any situation where file permissions cannot be changed. Not just exfat, anything. It also applies to overwriting files of which you're not the owner, but in the right group (However apps are less likely to do copy-overwrite, so it's less of an issue)
Essentially they changed from doing cp to cp --preserve=mode. They have to fix that.

There's little the user can do to work around it other than _for now_ sticking with 5.20. Unless the specific fuse filesystem has an option to swallow chmod without errors.

I hope that explains why this is happening.

[MichaIng]

@Taloth
That explains everything very well, many thanks for your research. IMO this should be definitely handled as bug on Mono side since one cannot expect that a file system has UNIX permissions, nor that the user is allowed to change/set permissions, even if the file system would support. In question, as well as a security or admin intention, the given/set/forced permissions of the directory/mount must be kept by times.

I think we'll add the info as "known issue" here and to our online docs for Mono-runtime software and possible solutions.

Best solution is to avoid FAT/exFAT file systems. They are anyway poorly supported by Linux, not only due to missing UNIX permissions but as well high CPU usage when doing R/W access by the background fuse process on exFAT. On NTFS, adding "permissions" mount option (default with dietpi-drive_manager) simulates/enables UNIX permissions, but the CPU usage of ntfs-3g process is still not beautiful. FAT32/vfat does not have that issue, but no UNIX permissions and 4 GiB limitation is anyway reason enough to avoid it where possible.
To transfer data between Linux and Windows, network file servers/shares are anyway a more comfortable solution, so the drives can stay plugged on the same machine. E.g. NFS: https://graspingtech.com/mount-nfs-share-windows-10/

If there is a strict need to use exFAT or FAT32 or any other non-UNIX-permissions file system:

On Buster, mono 5.18 can be pulled from Debian repo: https://packages.debian.org/buster/mono-complete

rm /etc/apt/sources.list.d/mono-xamarin.list
apt clean
apt update
apt install --reinstall mono-complete

On Stretch this results in a very old version. Since there is only an official mono v4 repo but none for mono v5, the packages must be downloaded manually, AFAIK? This is a bid annoying due to the large amount of dependencies which must all be pulled and installed manually as well :thinking:. Or is there an easy way to downgrade mono with official repo applied? The list files only contain the most current version: https://download.mono-project.com/repo/debian/dists/stretch/main/binary-amd64/Packages

[Taloth]

Yes, I think it's a good idea to simply qualify it as a known issue and provide instructions for affected users on how to downgrade, rather than doing that for everyone.

Or is there an easy way to downgrade mono with official repo applied

Quite doable, but I wouldn't qualify it as easy since the existing version _must_ be uninstalled first.

Mono has repository 'snapshots' of many of the versions, you can find them here for debian stretch https://download.mono-project.com/repo/debian/dists/stretch/snapshots/
There's a short instruction here in the mono docs.

Please note that we might add some workaround to Sonarr v3 (not v2) that detects affected mono versions and falls back to a (slower) custom copy/move process rather than relying on mono to do it.
But I'm hoping to hear back from mono, coz if they can fix it for the mono 6.6 release we can avoid all that pain.

[MichaIng]

@Taloth

Quite doable, but I wouldn't qualify it as easy since the existing version must be uninstalled first.

Good to know!

Mono has repository 'snapshots' of many of the versions

As long as v5 does not get stability/security updates, this looks like the best solution/workaround, at least for Stretch.

Please note that we might add some workaround to Sonarr v3 (not v2) that detects affected mono versions and falls back to a (slower) custom copy/move process rather than relying on mono to do it.
But I'm hoping to hear back from mono, coz if they can fix it for the mono 6.6 release we can avoid all that pain.

Sounds reasonable. Currently we ship Sonarr v2 with DietPi but will switch to Sonarr v3 once released as stable. I'll keep an eye on this issue to remove downgrade/workaround hints from our docs, once a workaround or fix has been applied.

For completeness as note to myself:

• Mounting the drive with sonarr user works as well, even sonarr:dietpi which with umask 002 allows downloaders and media players to access as well, when installed via dietpi-software or added to dietpi group.

[MichaIng]

There is a PR open which should solve the issue: https://github.com/mono/mono/pull/17870

[Taloth]

@MichaIng the mono fix was canceled because they wanted to port over the dotnet core fix, the dotnet core fix was postponed to net 5, so dotnet core 3 never got a fix so mono didn't port it over.
Like two weeks ago we added a workaround in Sonarr that detects the exception and then attempts to create a new file and copy the contents. It should be live in Sonarr v3 beta 3.0.3.731 and above.
I can't say whether mono 6.x runs sonarr entirely stable, but this particular issue should be ok now.

PS: A similar fix was made to Radarr.

[MichaIng]

@Taloth
Many thanks for this information. Seems we should move to Sonarr v3 better earlier then later.

[Taloth]

Ah, didn't think about v3 vs v2 for this issue. v2 doesn't have the fix.

I should talk to markus about doing an official rc1 for v3.

[MichaIng]

@Taloth
That would be great, if it is considered stable enough of course. I am just not sure if I find the time to migrate with DietPi v6.29 already. There are two other time consuming API changes I want to implement and want to push v6.29 to beta then, as it contains quite some urgent fixes meanwhile.

[Taloth]

There's no priority whatsoever, the mono version you're using now is stable and works fine with Sonarr. So pick your priorities, I just wanted you to know about the workaround.

[MichaIng]

Samba/SMB/CIFS mounts are affected as well: #3529

[Joulinar]

Guys,

does anybody know if the issue still persist with mono 6.10?

[MichaIng]

I put it onto the list for testing later.