Flood: Problems when rtorrent is set to move completed download using event.download.finished

[randomnonsense]

With vanilla rtorrent flood is great. One adds a torrent and it gets saved wherever the user chooses. The problem arises when rtorrent is configured in a more advanced fashion to sort completed folders based on their respective watch folders.

The old way to do this (using _"watch"_, _"incoming"_ and _"done"_ folders)

schedule = watch,5,5,"load_start=watch/*.torrent,d.set_directory=incoming/,d.set_custom1=done/"
method.set_key = event.download.finished,move_complete,"d.set_directory=$d.get_custom1= ;execute=mv,-n,$d.get_base_path=,$d.get_custom1="

A modern and more accurate way (using only _"watch"_ and _"done"_, _"incoming"_ set by _directory.default.set_ is the same for all). This is also the method officially described and remmended by rtorrent, described here:https://github.com/rakshasa/rtorrent/wiki/Common-Tasks-in-rTorrent#id11

schedule2 = watch,5,5,"load.start=watch/*.torrent,d.custom1.set=done"
method.insert = d.get_data_full_path,simple,"branch=((d.is_multi_file)),((cat,(d.directory))),((cat,(d.directory),/,(d.name)))"
method.insert = d.move_to_complete,simple,"d.directory.set=$argument.1=; execute=mkdir,-p,$argument.1=; execute=mv,-n,$argument.0=,$argument.1=; d.save_full_session="
method.set_key = event.download.finished,move_complete,"d.move_to_complete=$d.get_data_full_path=,$d.custom1="

Now, this causes problems when a torrent is added via flood. Let's assume the default download location for flood is _"flood/"_ (replacing "watch" from the examples above), which can be easily set under Settings > Resources > Default Download Directory. The same is mirrored in .rtorrent.rc. When one adds a torrent, it is initially saved into _"flood/"_ until its completion, but then it is erroneously moved out of _"flood/"_ into user's home directory. It is not moved into "done/" or kept in _"flood/"_. rtorrent reports an error in processing of event.download.finshed and the torrent is moved out of _"flood/"_ into user's home directory (and flood displays its "Base path" as the torrent's folder name, so without any preceding path at all).

Rtorrent's log indicates:
mv: rename /home/rtorrent/flood/<The torrent's folder> to : No such file or directory

Looks like there is a conflict when event.download.finished is processed. Would you be kind enough to find a moment to look into this? So far, apart from mediainfo (which is not important), this is the only thing not working.

Excellent job, btw. I fucking love flood. Exactly what I was looking for.

[jfurrow]

Are you adding files via URL or uploading the actual torrent file?

It's interesting that Flood has any affect on this at all. It makes me think Flood is incorrectly assigning a value to one of the properties in those scripts, but I don't know for sure.

If you're adding via uploaded torrent files, I suspect the problem is somewhere in this range: https://github.com/jfurrow/flood/blob/master/server/models/ClientRequest.js#L130-L141

If you're adding via URLs, the problem is probably in this range: https://github.com/jfurrow/flood/blob/master/server/models/ClientRequest.js#L165-L175

I honestly don't remember what those properties even do (probably should have left some comments), so I'll have to dive into this when I have more time.

Thanks for the issue and kind words!

[randomnonsense]

Actually... I don't think this is Flood's bug at all. According to (messy and incoherent as they are) rtorrent docs, every d.* command takes an info hash as the first argument, so in case of using _schedule2_ torrents loaded this way get assigned their _d.custom1_ directories, so the destination for completed data to be moved upon completion. Now, when loading torrents directly, outside schedulers which assign them appropriate _d.custom1_ variables, rtorrent simply doesn't know where to copy completed data (and as global event.download.finished tells it to use _d.custom1_... which at this point is empty), and effectively the data doesn't go anywhere.

I loaded a torrent manually directly into rtorrent, and the effect was the same. Data ended in user's home dir. Mystery solved.

The practical problem remains. Every config using schedulers and sorting completed data based on their watch dirs, and using Flood, will run into this problem. Anything loaded directly via Flood will end up mismanaged upon completion. This is especially likely as the "sort based on watch dirs" is included in the official rtorrent docs, i.e. here: https://github.com/rakshasa/rtorrent/wiki/Common-Tasks-in-rTorrent#move-completed-torrents-to-different-directory-depending-on-watch-directory

To solve this ultimately:

.rtorrent.rc would need to include a conditional definition of a method.insert call defining _d.custom1_ if it is not set or if it's empty, in a fashion: "if custom1 is empty, set default value Y". Also this needs to be defined so it doesn't interfere with _schedule2_ type schedulers. I am not really sure if this is possible.

OR

Flood needs to be told to assign the torrents it loads via GUI or URL appropriate _d.custom1_ variables, which would mimic _schedule2_'s behaviour. So basically rtorrent needs to see what Flood does like Flood was a watch dir. Torrents loaded via Flood get their _d.custom1_ like they were loaded via a watch dir, i.e.:

_schedule2 = flood,5,5,"load.start=flood/*.torrent,d.custom1.set=flood-done"_

what according to rtorrent docs means: "Each watch directory adds torrents with the final destination stored in the custom1 string of the download". Flood would need to do the same.

[blacksponge]

Hi,
I don't know if it can help you but that is the config I used in my .rtorrentrc to get the final directory of a torrent :

method.insert = d.get_finished_dir,simple,\
        "if=(d.get_custom1),\
        (cat, /data/downloads/, (d.get_custom1), /),\
        (cat, /data/downloads/unsorted/)"

I haven't tested it with flood yet but I guess it could work.

[randomnonsense]

FUCKING A! This does it, man. This is exactly what I spoke of before, just didn't have time to research this further. This is the conditioning that was needed.

This way all .torrent files tied to watch directories end up completed in appropriate locations, and what's not tied, ends up in "_done":

Example:
/home/rtorrent/_incoming = default download directory, shared
/home/rtorrent/_unsorted = default completed directory, if no watch dir/scheduler was used
/home/rtorrent/test-watch = example watch directory
/home/rtorrent/test = example completed directory tied to _watch/

session.path.set = /home/rtorrent/session/

method.insert = cfg.basedir,private|const|string,(cat,"/home/rtorrent/")
directory.default.set = (cat,"_incoming")

schedule2 = test,5,5,"load.start=test-watch/*.torrent,d.custom1.set=test"

method.insert = d.get_data_full_path,simple,"branch=((d.is_multi_file)),((cat,(d.directory))),((cat,(d.directory),/,(d.name)))"
method.insert = d.move_to_complete,simple,"d.directory.set=$argument.1=; execute=mkdir,-p,$argument.1=; execute=mv,-n,$argument.0=,$argument.1=; d.save_full_session="
method.insert = d.get_finished_dir,simple,"if=(d.get_custom1),(cat, (cfg.basedir), (d.get_custom1), /),(cat, (cfg.basedir), _unsorted/)"
method.set_key = event.download.finished,move_complete,"d.move_to_complete=$d.get_data_full_path=,$d.get_finished_dir="

[randomnonsense]

This needs to be included in the future Flood docs, as a case study.

@jfurrow this "bug" can be marked as solved ;-)

[randomnonsense]

Also, since we are at this, for perfect compatibility, if somebody wants to use a specific temporary directory for data being downloaded, or if completed torrents need not to be moved to a specific directory, add a _d.directory.set_ to a schedule2, e.g.

schedule2 = movies,5,5,"load.start=watch/*.torrent,d.directory.set=movies,d.custom1.set=movies"

This will create torrent files in _movies_ and then keep the data after completion there.

This makes sense when you are the original seeder of a torrent. Put complete data into "movies", rehash and start seeding. The torrent will start and stay there without any further shenanigans.

[randomnonsense]

Okay this is solved. Perhaps should be marked "to be documented".

[Maelstromeous]

This discussion has helped me a lot. Thank you.

[Laynezilla]

I'm not sure if I completely understand but I don't think this is solved.

For more complex watch schedules, custom variables do not get set when adding a torrent URL through the Flood UI. For example:

schedule2 = watch_add_episode, 13, 10, ((load.start_verbose, (cat, (cfg.watch), "television/episode/*.torrent"), "d.custom.set=type,television", "d.custom.set=subtype,episode", (cat, "d.custom.set=addtime," (system.time))))
schedule2 = watch_add_season, 13, 10, ((load.start_verbose, (cat, (cfg.watch), "television/season/*.torrent"), "d.custom.set=type,television", "d.custom.set=subtype,season", (cat, "d.custom.set=addtime," (system.time))))
schedule2 = watch_add_special, 13, 10, ((load.start_verbose, (cat, (cfg.watch), "television/special/*.torrent"), "d.custom.set=type,television", "d.custom.set=subtype,special", (cat, "d.custom.set=addtime," (system.time))))

This sets 2 variables for each watch directory. Post download processing fails because these variables do not get set and are blank.

The workaround seems to be to set a fallback value like blacksponge's example but that doesn't actually solve the real issue. For a configuration like mine, variables are too complex to have a single fallback value.

Why are post load commands not being executed?