Sonarr: Symlink/Hardlink/Copy/Move

[CochinoChingon]

Currently Only copy and hardlink are supported. I need symlink before switching over from sickrage. I'm pretty sure I'm not the only one holding out for symlink. the time I used Sonarr was great for adding shows quite simply, but over time my need for symlink trumps ease.

I'm not the only one who needs this feature and I'm sure someone else could more eloquently explain a situations where it is needed.

[kayone]

I'm curious what's the scenario that symlinks would be useful?

also, I assume the original file would stay in download directory and symlink would be in the series folder? or the other way around?

[Megagator]

I'm running into this exact issue. I need torrents to continue to seed from the host machine, yet my collection lives on external drives/NAS. Previously, SickRage would copy the completed download to the destination drive and symlink that back to a seeding folder where it could continue to seed, keeping the actual bits off of my computer's primary drive.

Hard links cannot be made between two different file systems, so Sonarr falls back to coping the data instead.

Mono.Unix.UnixIOException: Cross-device link [EXDEV].
  at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00000] in /private/tmp/source-mono-mac-4.2.0-branch/bockbuild-mono-4.2.0-branch/profiles/mono-mac-xamarin/build-root/mono-4.2.1/mcs/class/Mono.Posix/Mono.Unix/UnixMarshal.cs:460 
  at Mono.Unix.UnixMarshal.ThrowExceptionForLastErrorIf (Int32 retval) [0x00007] in /private/tmp/source-mono-mac-4.2.0-branch/bockbuild-mono-4.2.0-branch/profiles/mono-mac-xamarin/build-root/mono-4.2.1/mcs/class/Mono.Posix/Mono.Unix/UnixMarshal.cs:473 
  at Mono.Unix.UnixFileSystemInfo.CreateLink (System.String path) [0x00000] in /private/tmp/source-mono-mac-4.2.0-branch/bockbuild-mono-4.2.0-branch/profiles/mono-mac-xamarin/build-root/mono-4.2.1/mcs/class/Mono.Posix/Mono.Unix/UnixFileSystemInfo.cs:277 
  at NzbDrone.Mono.DiskProvider.TryCreateHardLink (System.String source, System.String destination) [0x00000] in m:\BuildAgent\work\6c3239faf2b92630\src\NzbDrone.Mono\DiskProvider.cs:160

Ideally, symbolic links would replace hard links as they are more flexible and make it clear where the original file exists.

[CochinoChingon]

the original file would MOVE and the symlink would reside in the original download location for the BT client to "see" and continue seeding. currently the "dirty" way I have found a solution is have sonarr snatch downloads to Blackhole (watch folder) and have SickRage watch said folder and handle all the post processing.

However I would suggest keeping HARDLINK in ADDITION to symlink and Move (as copy already exists). I don't know of a personal scenario where I would need hardlink. Presumably if there are no complaints about Hardlink then it likely serves someone.

[msh100]

I am also curious if the option to specify how a "copy" is done would also be possible?

My scenario: I am copying into an encrypted filesystem from the torrent client, and I would like to copy at a limited speed (probably with rsync) so that it doesn't put so much load on my CPU when copying occurs.

[markus101]

Hardlink is better than symlink because both files can be deleted independently and the other is not effected, is easier to achieve without having to pause/stop the torrent, move the file then setup the link, but is limited by the filesystem and being on the same drive.

I don't see us implementing a custom copy system to rate limit the copy, we'll just call the function and let the framework and OS handle it, it adds a lot of complexity and the issues that come with it.

[Megagator]

No need to pause seeding to symlink. You can just copy the file to the destination and then replace the orginal with the symlink. As long as the file name is the same, bittorrent clients will find the data fine without a re-scan or anything. The downside to symlink is if the source file moves, links to it break, but if your library resides off of your primary disk, it's your only option.

[markus101]

You definitely need to in Windows, file locks will prevent the deletion, regardless of still be weary if it was actively seeding.

Deleting files for upgrades would also cause issues since you'd need to restore the original file instead of simply deleting it or you end up with a broken link.

[Megagator]

Ahh, Windows...

Upgrades is definitely an issue, with symlink and move. Maybe for those, use the Sonarr recycle folder and move the lower quality files there?

[CochinoChingon]

You can't hardlink across volumes/drives.

I have a drive to download to, once complete TV shows are symlinked the "TV" drive where the downloaded file is copied to and the original download location has the link with original filename.

Movies are moved to a larger movie drive and symlinked as well with Couch Potato.

Finally music is symlinked as well to a Music drive.

It IS a useful/needed feature and the only thing holding me back from switching completely from sickrage

[JigSawFr]

Same problem as @elgallo .
UNIX or Linux does not allow creating hard links across file system boundaries.
So it's problematic to jus have Hardlink option.
Please consider adding Symlink too

[thadius856]

Same problem as @elgallo here as well.

With SickRage, I was able to have it move the files and create symlinks in the torrent download location. This allowed me to perma-seed things in my library.

Hardlinks don't work in my case because my library resides on a different volume then my torrents. Hardlinks can't span multiple volumes.

Symlinks allowed me to have one zpool that I care enough about to build redundancy into (11x 3TB in a RAIDZ3 + hot spare) for long-term storage, while the seeding I do just for the sake of seeding and ratio-building can remain on the zpool I care much less about (4x 5TB in RAIDZ1).

Right now with Sonarr, I'm unable to perma-seed without having to keep two copies of the data around -- one on my library pool, one on my seeding pool.

[SirSilent]

Couldn't this be done with a custom Python script executed on completion? Have Sonarr set to Copy so it copies the physical file to your desired drive, then use the the EpisodeFile_Path and EpisodeFile_SourcePath envorinment variables to create your symlink.

If you were looking to avoid file locks, you could probably loop it if it fails, or put a wait timer on it to avoid the initial surge.

[CochinoChingon]

yes it could be done via python, as well as Filebot and Flexget, However personally, though I would guess that @JigSawFr and @thadius856 are with me on this, we'd like an all in one solution, for the time being I've switched to SickGear [more stable than Sickrage(tv) ] and has symlink/copy/move built in.

[thadius856]

@elgallo Truth be told, I ended up re-organizing my pools so I could hardlink. It took a pretty big restructuring of my folders and a lot of moving across vols, but now I'm rocking along with the hardlinks.

It's not what I originally wanted, but its workable in my case.

Oh god was it a lot of work setting everything back up tho...

[CochinoChingon]

Precisely considering the fact that I have to run a secondary app to search to rent providers that other automatic show downloaded have in box AND symlink, I'm gonna stick with that.

[SirSilent]

I just tested this Python script, with Sonarr executing the custom script on download. It worked fine with Sonarr set to copy. I'm sure there are better ways to do it, like loop on failure until it works etc.

import logging
import os
import sys
import time

# Set up the log file
logging.basicConfig(filename='sonarrsymlink.log',level=logging.INFO)

# Get the paths from the environment variables
permpath = os.environ.get('Sonarr_EpisodeFile_Path')
linkpath = os.environ.get('Sonarr_EpisodeFile_SourcePath')

# Sleep 1 minute to avoid the surge
time.sleep(60)

# Delete the original file
try:
  os.remove(linkpath)
except OSError:
  pass

# Create the symlink
os.symlink(permpath, linkpath)

# Log it if level is set to debug
logging.debug('Created link from symlink location: [%s] to permenant location: [%s]', linkpath, permpath)

Obviously the ideal scenario is Sonarr supports the different link methods with flip switches, but once I move from my seedbox to in-house next month, I'll be using this or something very close to this until it's possible.

[Megagator]

@SirSilent This is close, but it doesn't work when renaming is on and the On Rename event doesn't have the SourcePath environment variable like On Download.

[ctrlbreak]

Can I add my own +1 to adding the ability to symlink.

My use case is similar to those above:

1) I use unraid to create virtual shared mounts across multiple disks (it uses the fuse fs under the hood). This means that hardlinks are out for me.
2) I have a large array - around 20TB, so I like to permaseed
3) copying files around on my array is relatively slow
4) my preferred approach, therefore, is just to create a symlink in the target folder (i.e. the TV shows media folder, including all the clever renaming/restructuring), back to the original file in the download area.

I realise this is the other way round from the copy and symlink approach, but it has been my (manual) approach until now.

This optional/advanced feature would make sonarr an absolute winner for me!

Thanks.

[zanechua]

Took Inspiration from @SirSilent with his Python Script.

Converted it to shell and added some extra checks.

This does not support File Renames and frankly I don't need file renames.

There is a bug with the script where if you download something and your torrent client is still seeding, this script might fail. I am looking at seeing if I can work something around that

#!/bin/bash

LOGFILE="/opt/NzbDrone/Logs/sonarrsymlink.log"
PERMPATH="$sonarr_episodefile_path"
LINKPATH="$sonarr_episodefile_sourcepath"

if [ -f "$LINKPATH" ];
then
        echo "Normal Queued Import Invoked, Continuing Script" >> $LOGFILE
else
        echo "Manual Import Invoked, Exiting Script" >> $LOGFILE
        exit 1
fi

ORIGFILESIZE=$(stat -c%s "$LINKPATH")
PERMFILESIZE=$(stat -c%s "$PERMPATH")

env > /opt/NzbDrone/Logs/env.log

echo "Hello. I am going for sleepy time for 30 seconds" >> $LOGFILE
sleep 30

echo "Checking if Sonarr is still copying the file" >> $LOGFILE

while [ $PERMFILESIZE != $ORIGFILESIZE ]
do
        echo "Sonarr is still copying the file, wait for 1 minute and try again" >> $LOGFILE
        sleep 60
        PERMFILESIZE=$(stat -c%s "$PERMPATH")
done

echo "Entering Final File Size Check Phase" >> $LOGFILE

if [ $PERMFILESIZE == $ORIGFILESIZE ]
then
        echo "Removing File" >> $LOGFILE
        rm "$LINKPATH"
        echo "Creating symlink" >> $LOGFILE
        ln -s "$PERMPATH" "$LINKPATH"
fi

echo "Created symlink from source location: $PERMPATH to symlink location $LINKPATH" >> $LOGFILE

[Ardakilic]

Another +1 for symlink as an option to choose from. I use Plex with only one drive using Debian server, and this method just doubles up my spare free space.

[Taloth]

Euh, one drive on debian? So why no hardlinks?

[Taloth]

symlinks are an absolute headache to manage, that's why we haven't done it.

[arette]

First of all - Sonarr rocks! Thank you to all the developers who devote their time to this awesome product. I need this one - same situation with multiple volumes on a NAS where hardlinks are not possible. My NAS is filling up at a rate of 2X for torrents that remain seeding indefinitely. Please consider implementing this one! I would be willing to help if someone could give me a quick orientation to how to go about implementing it (note that I've already pulled the Git repo down).

[CochinoChingon]

@arette go with SickGear supports symlinks out the box, more trackers than sonarr (without Jackett). It's nowhere near as pretty, or easy to add multiple TV Shows, but it helps save a TON of space with symlinks.

[huanga]

Would like to add yet another +1 on this request.

Have Sonarr finish processing the renaming first, then build the symlink to the destination. If the user chooses to rename things again after the fact, that is user's own accord and it is the user's responsibility to make sure things work in their other clients (e.g.: Transmission/Deluge).

Duplicating data and thus taking up twice the space is cost prohibitive, and not user friendly.

[pando85]

None option? I don't need copy, move or hardlink. My deluge is managing that.

[mihaiturcu]

+1 for the requirement of symlinks.
I just bought a secondary hdd that i added to my hp microserver. Since my raid controller doesn't allow me to extend the currently existing array, the logical volume can't be extended and thus my os will see the 2nd drive i just bought as another disk mounted on the system.
I've decided to simply configure the torrent client to dump downloads to the new disk (2nd disk) i bought + i'd need sonar to place symlink on my first disk so that Plex will find the newly downloaded tv show :)

[bjeanes]

Any thoughts to supporting reflinks? I'm using Sonarr on a BTRFS file-system and have downloads and my series in different subvolumes. Subvolumes do not share inode space so hard links fail, but it is possible to issue cp --reflink[={auto|always}] which will use create a Copy-on-Write version of the file. This means the operation is instant and won't increase disk usage.

A useful comparison between hard/sym/ref links: http://www.pixelbeat.org/docs/unix_links.html

Obviously this is a bit niche, but BTRFS is growing in usage and the default recommendation on Synology systems now.

@Taloth @markus101 I'm going to peek at the hard link code now to see if implementing this seems prohibitively complicated but let me know if this is something that you outright don't see Sonarr including.

[bjeanes]

It looks like this would be non-trivial without implementing a custom low-level file copy. I followed Sonarr's code (which was pleasantly easy!) and realise that in the case where we actually perform a copy, it'll defer to Mono's CopyFile implementation in C, which if you then follow it to write_file seems to do the copying itself by reading and writing to file descriptors. I don't spend much time in C (and none in C#) so I may have misinterpreted things here, but it's probably too big to tackle, certainly for me.

[Taloth]

I'd welcome a copy-on-write approach, assuming it can be implemented transparently and reliably. Personally I think filesystems should support copy-on-write by default.... in-fact, filesystems like zfs already do copy-on-write under the hood whenever you modify a file to prevent in-place mutation and make snapshots possible. (like btrfs)

I haven't looked at mono file copy code, I assumed mono called the platform api to do it. If it really does it by itself then it explains why it fucks up when CIFS times out (different issue).

I suggest you follow the hardlink code instead: https://github.com/Sonarr/Sonarr/blob/0bdc1370937a35377925bb2a07136ab9b8533acf/src/NzbDrone.Mono/Disk/DiskProvider.cs#L120
https://github.com/mono/mono/blob/0bcbe39b148bb498742fc68416f8293ccd350fb6/mcs/class/Mono.Posix/Mono.Unix/UnixFileSystemInfo.cs#L277
https://github.com/mono/mono/blob/0bcbe39b148bb498742fc68416f8293ccd350fb6/mcs/class/Mono.Posix/Mono.Posix/Syscall.cs#L245

I couldn't find an syscall for clone_file, but something in that direction would be the way to go. It's definitely non-trivial, the function doesn't exist in mono.

[Elegant996]

This is the solution more or less, both Windows and mono support symlinks so there's really nothing stopping one from adding the support. I did however implement this on Radarr (Sonarr but for movies).

The solution is in my last commit found here.

[mihaiturcu]

Wow, there's hope for my storage space. @Taloth the solution proposed by @Elegant996 seems pretty promising.

[bjeanes]

@Elegant996 the problem with symlinking is that if you don't move _then_ symlink, removing a file from download client could break the symlink in your library. Hardlinks (or reflinks) mean that a delete on either end has no affect on the other. I glanced at your commit but not being familiar with the codebase couldn't readily determine which direction your symlinking. Moving file into library then symlinking it back to download directory _could_ be OK but some clients may have issue with this.

[Elegant996]

I thought about that today actually. I'll make some adjustments, this is not difficult to do. You're basically trying to piggy back off of the Move function that already exists and add an extra check at the end to symlink.

I'll work on this a bit more; to be clear, I have not used Sonarr or Radarr beyond a few minutes and was immediately turned off by the lack of this feature (I ran back to SickRage, which I also had to add symlink fixes to). If I can somehow incorporate this feature with little development time I will do so. I have not programmed in ages but this code is far from complicated.

[markus101]

Renamed/deleted files would cause problems, the only reliable way to handle that would be to track the files and update the symlinks if the files were renamed (which would work if the file is in the series folder and symlinked back to the download client), if you did a reverse symlink (I think that's what other apps have called them) and symlink to the series folder then deleting the downloading file would be a bad experience and Sonarr would need to be aware the download was symlinked into the series folder so it would prevent automatic cleanup of those files if Remove Completed Downloads was true.

The commit is a bit hard to follow because github is telling me entire files have changed when they shouldn't have, but one thing I noticed was a new option for symlinking, which means that you could have hardlink instead of copy and symlink instead of copy, but there isn't an obvious winner between symlinking and hardlinking.

[Elegant996]

There is no real winner for anything here. This is only to enable one use case and to leave the other intact (eg. multiple drives/shares; can't hardlink). My Radarr implementation would be a "reverse symlink" which is why I am changing it, as conceptually it makes no sense to me.

In addition, this shouldn't be part of the copy operation but the move operation (this will require an additional step so seeding continues). It makes a lot more sense to move the file to it's final destination and then symlink it back to the download with the old file name.

I am working on this under a Sonarr fork in my repo instead of a Radarr fork as my earlier source was. You can see my idea much more clearly in my latest commit but this will more than likely require some more testing. Again, I am not familiar with running this app on Windows so when I see that generic{ "message": "NotFound" }I move right along.

I really just want to get the ball rolling, ultimately I agree with the idea of reflinks but that is more than likely beyond what I want to commit as a time investment as that's a non-trivial solution.

[markus101]

Again, I am not familiar with running this app on Windows so when I see that generic { "message": "NotFound" } I move right along.

You need to compile the UI so Sonarr has something to show: https://github.com/Sonarr/Sonarr/blob/develop/CONTRIBUTING.md#getting-started

[shrouski]

@Elegant996 There is only one winner and that would be to either add a symlink field to the EpisodeFiles table or create a dedicated table which contains the Episode ID and physical path of the symlink so that it can properly be monitored (including rename, upgrade, etc.). Ultimately hardlinks are too limited with the file sizes of today and there are plenty of us want to be able to easily permaseed.

[Taloth]

Ultimately hardlinks are too limited with the file sizes of today and there are plenty of us want to be able to easily permaseed.

I don't see what filesize has to do with it. There are filesystems that tackle all of that transparently.

More ideal with be btrfs reflink mechanism since it's copy-on-write and thus the space-saving of hardlinks without the nasty parts. but there is no standardized api (ioctl) for reflink. In fact, it would be so much better if btrfs had a fstab flag to make all copy actions behave as reflink so it works transparently, but afaik there isn't.
zfs doesn't have a similar mechanism either.

[shrouski]

Filesize because an 8TB drive is simply not big enough when storing media, so a lot of people will be using more than one HDD and something like DrivePool on Windows or MergerFS on Unix to pool the drives.

[mihaiturcu]

Agree with what shrouski said. The ordinary consumer doesn't have his NAS fully stuffed with disks and he will most likely upgrade drive by drive depending on budget.
Sonarr already has 3mil docker pulls, that's a very huge user base for your project. Providing them with the option of using symlinks as to not double the space usage after sonar sorts and renames will most likely lead to an even larger user base for Sonarr. Some of my friends who own mediaservers preffer to stay with Sickrage just becase of the symlink & reverse symlink functionality.

[togbabe]

My issue has been that I have a virtual linux machine running sonarr and utorrent. But the file system for Plex etc is on the host’s windows files system. So – obviously – Sonarr’s hardlink solution is useless in this case. What I wanted was to let sonarr do its copying – which it does very well – and then continue seeding. My solution is heavily based on @zanechua's script. This way, the ‘torrent client’ can continue seeding through a symlink, and I don’t have to keep the same file in two places. Copying the file and leaving the original to seed is – soon – going to take up a lot of unnecessary space. I don’t get Sonarr’s reasoning for creating the hardlink in the ‘media library’ and keeping the original file in the ‘torrent download location.’ To me it makes much better sense to copy the media into the media library and put the hard link into the ‘torrent client’s’ directory. Anyway, I have use zanechua’s script and just improved on it a bit by adding a bit more logging, so that the script can also be run from the command line, with the arguments from the $FROMTOFILE if – for some reason – it didn’t work automatically, and you need to do it later. But, more importantly, I added some error handling for the original file ‘still being in use’ after sonarr has done it’s copying. Torrent clients don’t keep a permanent lock on the original media file, and will eventually leave it alone long enough to create the symlink. All I have done is added some code that checks to see if the original media file is in use, and – as soon as it is not – it creates the symlink. I have found that the torrent client just keeps rocking on with the symlink as if nothing has changed. This way, I get to continue seeding without keeping two copies. I also added some ‘times stamping’ in the log, and redirected errors to the ‘standard out’ so that you can see what happened later if things didn’t work, and you needed to use the script from the command line to fix them.
Obviously, you would have to change the values of $LOGFILE, $FROMTOFILE and $TORRENTCLIENT to match your environment. Keep in mind that the value of $TORRENTCLIENT is the actual name of the ‘torrent client’s’ process you would see if you did a ‘top command,’ and not necessarily the friendly name of the ‘torrent client.’ For example, Uttorrent’s ‘process name’ is ‘utserver.’ You'll have to change the 'file extension' to '.sh.'

GoSymlinks.txt

[ChaosTherum]

I'm going to have to plus 1 for symlink it's the only thing keeping me with sickrage though I'm thinking of using sonarr in conjunction with filebot just to get symlinking.

[togbabe]

@ChaosTherum, have a go at the script I did. It has been running floorlessly, and you end up with a nice symlink in your seeding directory, and the file safely copied to your 'media library.' It works a treat.

[bjeanes]

I don’t get Sonarr’s reasoning for creating the hardlink in the ‘media library’ and keeping the original file in the ‘torrent download location.’ To me it makes much better sense to copy the media into the media library and put the hard link into the ‘torrent client’s’ directory.

@togbabe that's not how hard-links work. There isn't an "original file", unlike a symlink. Both files (original and new) are identical. They are both "links" and they both point to the same underlying data. There is 0 difference in end result based on which file is there first. For _symlinks_, there is indeed a difference. Now, moving/copying the file from download location to the media location and _then_ symlinking so that the download directory contains a symlink might make sense. However, moving/replacing a file that is open in another program (the download client) is buggy at best and risky at worse. If there was a reliable way to ask a download client to temporarily pause the file and close its file handle, this would work pretty well. That isn't the case though. It sounds like you've written your own scripts to try to work around this exact issue. The problem is that such a solution isn't necessarily generalisable to any torrent client someone might use.

If anyone is on the same filesystem (or can structure their network/docker/whatever mountpoints so that they are) then using hard link is basically always going to be the best option. The download client and Sonarr can read/rename/move the files as they please without affecting each other (as long as neither _writes_ to file content — in that case, a reflink would be the best option, but they are not as widely supported).

[Magic815]

Just adding a +1 to symlinking.

I mainly use torrents over usenet, and have a very large collection of permaseeded torrents that fills up about 90% of a 6TB hard drive. Right now, it seems like I only have two options:

1. Put my Sonarr folder on another 6TB hard drive that I have, and let Sonarr copy it to there. But then I'm effectively having to store every show twice on my hard drives. (Huge waste of space).
2. Keep my Sonarr folder on the almost full 6TB hard drive that holds the torrents, and let Sonarr hardlink it. But then I will quickly run out of space on that hard drive.

Symlinking is the only answer that will allow me to save as much space as possible, and let me "share" my torrent files across drives, so that none of them get completely full.

[jtpavlock]

+1 here. Using a pool with many disks where hardlinks are not an option. Symlinks could prevent duplicated files and save disk space.

[Taloth]

Reminder: Use the 'add your reaction' button instead of a +1 comment if you have nothing substantial to add to the discussion. Thank you.

[SirSilent]

@Taloth I think we just need a link db table and manage it properly. Most of us are with private trackers and want to permaseed.

[skeptibat]

I also am needing Symlink capability, as my long-term storage is on a different volume than my torrent download folder. I tried simply changing the code to symlink instead of hardlink, but I guess more is needed due to file locks?

When a file is done downloading in Deluge, I first need to pause the torrent, then move and rename, then drop symlink with the original downloaded file name in the downloaded folder, then resume the torrent, and things work, seeding continues.

If the destination file is replaced or upgraded, I have no qualms about the download client complaining that the original file is missing or incorrect, at which point I would just remove the torrent manually.

However, I'm not exactly sure how to properly script this, and would like to see it as a feature, at some point.

[CochinoChingon]

@skeptibat get sickRage or sickGear, both offer symlinks AND you don't need a secondary webapp (like jackett) to use most trackers

[skeptibat]

I enjoy the Sonarr interface, I find it pretty, easy to use, and lets me
know what's upcoming. Does the sickRage/sickGear suggestion work in tandem
with Sonarr or replace it entirely? If replacing, I'd like a smooth way to
transfer my show list over, and I'd probably want to do the same with my
movies (currently I'm using Radarr, a Sonarr fork for movies.)

Thanks for the suggestion!

edit: I checked out the Sickrage UI, pretty clean and nice! Though, this is probably tangential to this github issue thread, so we can take this offline.

[togbabe]

@bjeanes and @skeptibat I have literally been using the script I posted as an answer to these issues for 7 months straight without a single issue. It is rock solid, and allows the file to reside on another volume without any issues at all. All my torrents are still perma-seeded. No issues, no problems.

[skeptibat]

@togbabe yeah I saw that, looks like an elegant solution. However, as my sonarr runs on windows, bash scripts would require some extra framework that I'm not really interested in using.

Further, with windows file locks, I would need a scripty way to pause the downloader, then perform the symlink, then unpause the downloader.

I'm certainly not opposed to this solution, but some research into how to interface with the downloader would be required.

[bjeanes]

I have literally been using the script I posted as an answer to these issues for 7 months straight without a single issue.

That's great. However, as I said last time:

The problem is that such a solution isn't necessarily generalisable to any torrent client someone might use.

I've moved files that a torrent client was using before and it usefully moved the torrent into the error state (at least, if it was actively reading/writing). It sounds like maybe you're moving the file within the same filesystem. In unix, that's generally going to be safe for any applications with have a file handle already open.

However, the only real reason to use symlinks (vs hardlinks) is because you're moving _between_ filesystems. In that case, I'm positive this is not a resilient solution and that some if not many torrent clients will not expect a file to disappear. Who knows, perhaps most now have added error code to try re-opening the file after a few seconds before marking it as failed and that's why it's OK for you.

In any case, I personally use a single 30TB filesystem, so I have no need to use symlinks as hard links are infinitely better in this case. I'm only involved in this conversation because I wanted to see if we could also get reflinks support (which has many of the benefits of hardlinks I like without risking corruption when one client makes a modification and would work BTRFS subvolumes)

[lattedesu]

Please implement this feature. Lack of symlinks just wastes half of my hard drive space.

[Ardakilic]

I have found @zanechua 's script quite useful, and modified a bit. I both use bittorrent and usenet protocols, so I made an extra step to check whether the download is from usenet protocol by a simple string checking (a unique string containing for the usenet folder's name). So this way, if the protocol is usenet, only moving will be done and this symlinking process will not work. Also, I sourced from the download client to target folder, instead of adding symlink to download folder. In case there's an I/O error, I can always fix the movie by hash-checking from download client, re-download broken/missing parts and fix it. You can easily change these paths anyways. Last of all, I halved the waiting times. Here's the modified script:

#!/bin/bash

# Sonarr symlink post-process script
# Based on https://github.com/Sonarr/Sonarr/issues/967#issuecomment-228599878
# With some tiny modifications

# make syre Logs folder and file exists 
LOGFILE="/opt/NzbDrone/Logs/sonarrsymlink.log"
PERMPATH="$sonarr_episodefile_path"
SOURCEPATH="$sonarr_episodefile_sourcepath"


if [ -f "$SOURCEPATH" ];
then
     echo "Normal Queued Import Invoked, Continuing Script" >> $LOGFILE
else
    echo "Manual Import Invoked, Exiting Script" >> $LOGFILE
    exit 1
fi

# Usenet-specific download folder to ignore symlink process because usenet files are moved
USENET_PATH="nzbget" #a bunch of string, which usenet download folder contains but bittorrent folder doesn't
if [[ $SOURCEPATH = *$USENET_PATH* ]];
then
    echo "Import ignored, because download source is Usenet and file will be moved" >> $LOGFILE
    exit 1;
fi
# Usenet-specific download folder to ignore symlink process because usenet files are moved

ORIGFILESIZE=$(stat -c%s "$SOURCEPATH")
PERMFILESIZE=$(stat -c%s "$PERMPATH")

#env > /opt/NzbDrone/Logs/env.log

echo "Hello. I am going for sleepy time for 15 seconds" >> $LOGFILE
sleep 15

echo "Checking if Sonarr is still copying the file" >> $LOGFILE

while [ $PERMFILESIZE != $ORIGFILESIZE ]
do
    echo "Sonarr is still copying the file, wait for 30 seconds and try again" >> $LOGFILE
    sleep 30
    PERMFILESIZE=$(stat -c%s "$PERMPATH")
done

echo "Entering Final File Size Check Phase" >> $LOGFILE

if [ $PERMFILESIZE == $ORIGFILESIZE ]
then
    echo "Removing Hardlinked File" >> $LOGFILE
    rm "$PERMPATH"
    echo "Creating symlink" >> $LOGFILE
    ln -s "$SOURCEPATH" "$PERMPATH"
fi

echo "Created symlink from source location: $SOURCEPATH to symlink location $PERMPATH" >> $LOGFILE

You can also access from here:
https://gist.github.com/Ardakilic/f9670cfa90522509c80ed3f390d5b661

[devster31]

I'm putting in my preference for symlinking, this is a significant missing feature for many torrent users, as already mentioned, and integrating it into Sonarr itself is a lot better for docker users.
As mentioned in the linked Radarr issue there's work from @Elegant996 here
https://github.com/Elegant996/Radarr/commit/15bdaa8f11f134ea141c8af001abc3199f7cd180 which has it implemented.
This, in addition to #300 are some of the many features missing from Sonarr before it can be considered fully functional in my opinion.

[threesquared]

For me this is also a missing feature that prevents me from moving from sickrage. My data directory is a mounted NFS share so hard links will not work.

[NablaMetal]

Same here, built-in symlinks (and reverse symlinks) option would be very helpful...

[philk27]

Same here, the use case for docker is already there, but also REFS is coming mainstream and it does not support hardlinks.

[Elegant996]

I settled for hard links, it was an easier solution all thing considered as it prevents broken links. Just download your files to a .seed folder in your media share (NFS or otherwise) and then import them as usual. This makes it so you don't have to view the folder when browsing as .seed would be hidden.

[funkyb0i]

Suggestion:
"Use hardlink instead of copy" checkbox be changed to a select drop-down with the following options:

• Hardlink
• Copy
• Move

Reasoning:
Seedbox users with remote mappings do not need to preserve the original downloaded files on their local storage. Currently, files are copied & renamed. Ideally, files would be moved & renamed. Otherwise, downloaded files need to be deleted from the downloads directory after they are copied & renamed. (/downloads is separate drive from /tv)

[2fst4u]

This is kinda related: can this also be a seperate setting for NZB and torrents? I need torrents to be hardlinked but I obviously don't need NZBs to.

[markus101]

Sonarr already moves NZBs because there isn't anything to seed, it'll also move torrents that have finished seeding.

[2fst4u]

Under the "importing" part in media management settings there's a setting
to enable hardlinks. I have this enabled because I want files to be "moved"
immediately, not once they finish seeding.

Does this setting only apply to torrents? Or doesn't it affect NZBs and
torrents?

On Sun., 3 Feb. 2019, 10:08 Mark McDowall <[email protected] wrote:

Sonarr already moves NZBs because there isn't anything to seed, it'll also
move torrents that have finished seeding.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Sonarr/Sonarr/issues/967#issuecomment-459999255, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AOjRRoYNY0QHYZ5xWIA4VgNngzrw-O0Gks5vJf42gaJpZM4GrM5p
.

[markus101]

Use Hardlinks when trying to copy files from torrents that are still being seeded

That's right from the UI.

If you have more questions about this functionality please open a thread on the forums/subreddit 30+ people don't need notifications about questions on the behaviour as it is right now.

[SirSilent]

Use Hardlinks when trying to copy files from torrents that are still being seeded

That's right from the UI.

If you have more questions about this functionality please open a thread on the forums/subreddit 30+ people don't need notifications about questions on the behaviour as it is right now.

I've raised it numerous times on the forum and was the one that did the simple symlink python script a couple of years ago. Hardlinks don't work over different NTFS hard drives, so for those using Sonarr for it's main purpose; collecting, organising and storing media using a Windows platform, we're limited to symlinks.

[CochinoChingon]

I do not get why anyone (who uses private trackers) bothers with sonarr, to use certain sites you need to install a second app, jackett for connectivity to other trackers like MTV, Shazbat and many more.

personally I have gone almost completely CLI, Filebot (with some tinkering) works as needed, for moving downloaded files into Plex readble. Auto-DL/irssi works great for instantly snatching files. Neither are easy to setup, But Filebot dev answers questions/issues very quickly. (though he will not help with setup issues anymore).

I get Sonarr is "pretty" but does not work out of the box for nearly every need as SickRage/Gear/TV/Beard.

[churlkun]

I've been looking to move off of Sonarr for this very reason.

Without symlink support, everything is just too messy to get working properly. As much as I love Sonarr, it's just not worth the headache until this gets implemented :(

[zanechua]

I am not sure but have you guys tried the script I wrote? I have been using it for a few months now and that solves the problem for me honestly. Perhaps this issue should be locked till the developers actually wanna triage this or someone submits a PR for it.

[popeofmope]

I had separate shares for downloads and media on the same file system in my OMV NAS and can't hardlink so symlink would have been nice. Sonarr/Radarr/Lidarr/Transmission on a server writing to downloads share and Kodi accessing media share for viewing. To get around the no hardlink issue I just set Sonarr to save to a sub directory in the media share, and created a .nomedia file so Kodi won't scan it.

Before:

/downloads/sonarr
       ../radarr
/media/tv
   ../movies

After:

/media/tv
   ../.nomedia
  ../Show A

[elboletaire]

Couldn't this be a sonarr feature like the one flexget has which moves the files notifying the downloads client for the new location?

[eeverett91]

Adding my +1 to this for the reasons others have mentioned. Being able to have both a permanently seeding torrent and a well organized/renamed library often requires symlinks as you're crossing file systems unless you want to double your used storage. Please please please consider adding this. It's THE biggest missing feature I'm looking for have from sickrage/sickchill.

I'm not sure what the concerns with adding them are outside of them possibly breaking, but that's on the user to sort out. I think Sonarr just needs to create them, not maintain them.

[pyr0ball]

Adding my +1 to this as well

Main reasons for me is ability to continue seeding while also having organized media library for Kodi/Plex

Usage for me would be original file stays with torrent program location, symlink lives in media library folder. The reason for this is for easy re-add of torrents if the client gets broken, and qbittorrent doesn't seem to react well to symlinks in that scenario

[stuart475898]

A vote from me for move - I've a Linux based seedbox + a Windows based media server with Sonarr. I use Resilio + remote path mappings to transfer the files from the Seedbox to the media server (I've tried some SSH drives - hit and miss), ready for import. Being able to move the files would be nice, rather than me periodically checking Sonarr that all is well and then deleting manually. I haven't looked at the source, but given the manual import has a move options I'm guessing this wouldn't be difficult to implement?

[CochinoChingon]

A vote from me for move - I've a Linux based seedbox + a Windows based media server with Sonarr. I use Resilio + remote path mappings to transfer the files from the Seedbox to the media server (I've tried some SSH drives - hit and miss), ready for import. Being able to move the files would be nice, rather than me periodically checking Sonarr that all is well and then deleting manually. I haven't looked at the source, but given the manual import has a move options I'm guessing this wouldn't be difficult to implement?

I'd recommend switching to pyMedusa. You get move/copy/link and you don't need Jackett for unsupported trackers. Though sonarr has done better to include more trackers since it first started, as I see it. It's just prettier than other tv show PVRs, but certainly not as capable

[sotiris-bos]

Hardlinks don't work between ZFS datasets.

Having to use a single massive dataset for movies, series, music etc makes it really difficult when you have to move it around, balance the pool or apply changes that cannot happen in place.

[a10kiloham]

i love Medusa but the pace of Sonarr development seems to have picked up and frankly the various Sickbeard forks have fragmented the community too much IMO (medusa is the best of them but has the least recognition).
also maybe just my opinion but medusa feels to have gotten really slow - the interstitial pages between pages indicated a JS framework that's just too fat for my liking. Sonarr feels better in many ways, but a backlink via Symlink doesn't seem that difficult or out of the ordinary. I can't see why any torrenter who has a separate NAS wouldn't use that methodology.
for now my solution is just a much larger drive on the seedbox and deleting from there after enogh seeding. i'd seed permanently if the NAS solution worked but needs be.

[a10kiloham]

I am not sure but have you guys tried the script I wrote? I have been using it for a few months now and that solves the problem for me honestly. Perhaps this issue should be locked till the developers actually wanna triage this or someone submits a PR for it.

can you reshare please? i can't see it

[a10kiloham]

Took Inspiration from @SirSilent with his Python Script.

Converted it to shell and added some extra checks.

This does not support File Renames and frankly I don't need file renames.

There is a bug with the script where if you download something and your torrent client is still seeding, this script might fail. I am looking at seeing if I can work something around that

#!/bin/bash

LOGFILE="/opt/NzbDrone/Logs/sonarrsymlink.log"
PERMPATH="$sonarr_episodefile_path"
LINKPATH="$sonarr_episodefile_sourcepath"

if [ -f "$LINKPATH" ];
then
        echo "Normal Queued Import Invoked, Continuing Script" >> $LOGFILE
else
        echo "Manual Import Invoked, Exiting Script" >> $LOGFILE
        exit 1
fi

ORIGFILESIZE=$(stat -c%s "$LINKPATH")
PERMFILESIZE=$(stat -c%s "$PERMPATH")

env > /opt/NzbDrone/Logs/env.log

echo "Hello. I am going for sleepy time for 30 seconds" >> $LOGFILE
sleep 30

echo "Checking if Sonarr is still copying the file" >> $LOGFILE

while [ $PERMFILESIZE != $ORIGFILESIZE ]
do
        echo "Sonarr is still copying the file, wait for 1 minute and try again" >> $LOGFILE
        sleep 60
        PERMFILESIZE=$(stat -c%s "$PERMPATH")
done

echo "Entering Final File Size Check Phase" >> $LOGFILE

if [ $PERMFILESIZE == $ORIGFILESIZE ]
then
        echo "Removing File" >> $LOGFILE
        rm "$LINKPATH"
        echo "Creating symlink" >> $LOGFILE
        ln -s "$PERMPATH" "$LINKPATH"
fi

echo "Created symlink from source location: $PERMPATH to symlink location $LINKPATH" >> $LOGFILE

ah i see it - where is this called from? renaming is key for me too to keep my NAS orderly...

[zanechua]

This is called via connect and you set it to on download and on upgrade. Last I checked this work well for radarr. They even put the script in the wiki. Recently however I've been noticing that the script doesn't work properly on my sonarr configuration but haven't had the time to debug it to see if it was a script issue or my mount just not supporting symlinks.

[a10kiloham]

This is called via connect and you set it to on download and on upgrade. Last I checked this work well for radarr. They even put the script in the wiki. Recently however I've been noticing that the script doesn't work properly on my sonarr configuration but haven't had the time to debug it to see if it was a script issue or my mount just not supporting symlinks.

Cool thanks I'll try that

[a10kiloham]

This works well thanks @zanechua ! I updated a bit to exit properly so that may be why yours is failing
https://github.com/Sonarr/Sonarr/wiki/Reverse-symlink-script-for-Connect

I'd consider this solved now...

[SerialVelocity]

@a10kiloham I don't think this should be considered solved if we have to use a script. Especially since it needs a copy to succeed first. If the secondary location is NAS then you are going to saturate the network link temporarily (or CPU depending on your NAS setup).

My use case is I have multiple hosts each hosting a subset of my tv shows in a shared directory. Currently, I run multiple Sonarr instances, one on each host, but with symlinks, I can have a one Sonarr instance that uses multiple download clients, and then have a folder with all symlinks.

[a10kiloham]

Well sure it'd be nice if it was included in the main codebase but the script should work for 99% of users. Would love to see this mainlined into the official release...

[bakerboy448]

No plans at this time....someone posted a script.