Luigi: Recursively resolve dependencies

[LeiShi]

I have just started using Luigi, and enjoyed the pythonic style and the beautiful web GUI.

However, I find that by default, Luigi doesn't detect the indirect dependencies, i.e. Luigi only checks the directly required tasks status to determine the readiness for the current task, but does not know if a deeper task has been renewed. For example, if I have a dependency map like follows:

A <- B <- C

After running it once, if I delete the output from A, and rerun it, Luigi will say task C has already done. I need to delete all outputs from A,B and C to rerun them.

This is not a very intuitive way of solving dependencies, I would assume that a task scheduler would know if one task is renewed, and all the tasks that depend on it should be renewed as well.

Is there an official solution to this? I wonder if Luigi already had the recursive feature somewhere?

Thanks!

[chrispalmer]

Luigi's model of task execution and completeness is primarily based on the logic of:

A task is complete if and only if it's output exists.

Some dependency graphs can include hundreds or thousands of tasks, and checking them for completeness is a non trivial process. Repeating all those checks time and again could get very costly.

I think the first question is why did you delete the output from A? In a production environment assuming that A, B and C ran correctly there shouldn't be a need to delete the output.

However if you are still developing a pipeline (or your production tasks are broken), then yes there can be situations where you will find that A didn't run correctly and thus the output from A, B and C is incorrect. That can certainly be annoying to deal with but the only real solution with standard Luigi is to just delete the output from all three, and then run it again.

Beyond that you could write your own custom Task subclass and overwrite the complete() method to handle whatever conditions you want.

Chris

[LeiShi]

Hi Chris,

Thanks for the comment. Right now, I'm overriding the complete method and checking the dependencies recursively. As you pointed out, this task is indeed non-trivial in terms of the efficiency. What I have now is an O(n!) complexity way of doing it, and the overhead can be significant when the number of tasks gets large. I wonder how GNU Make deals with this, maybe it just assumes no source code should contain millions of source files...

However, I think this is a very useful feature. Even in the production runs, if I accidentally passed in one wrong input data file to one task, I certainly don't want to re-do all the tasks, but rather re-run the "affected" tasks. However, to manually find out and delete the invalid outputs is a very annoying thing. And handling all these dependencies and update the out-of-date outputs is one of the most important features for a task manager like Luigi.

This feature may not be turned on as default, but I would like to see this as an option.

Lei

[mayou36]

I agree that for production use this is usually not required. Although for scientific purposes it is crucial. Changing one parameter in a row is the application of pipeline tools. I would definitely vote for a (by default deactivated) feature like that.

@LeiShi did you come up with a nice solution already? If so, would you mind to share it somehow?

[LeiShi]

Hi @mayou36 ,

I have asked this on StackOverFlow a while ago, and accepted the answer.
https://stackoverflow.com/questions/42465710/can-python-task-scheduler-luigi-detect-indirect-dependencies

As mentioned in the comment section of the above link, the answer may not be the most efficient solution. I haven't put much more consideration on this issue since then because, in my case, efficiency is not a big problem. Maybe other Luigi users can come up with a more elegant solution.

[mayou36]

@LeiShi thanks for that! Sure, but I guess we are using it for similar cases ;) I think as long as the dependencies are rather deep then broad (as it is in the case of a data analysis pipeline where you have a dataset and basically apply 10-20 steps involving only a couple of different datasets), the efficiency of this approach ain't a problem.

[dckc]

I ran into this same issue, but overriding complete() to do belt-and-suspenders checking (as noted in #issuecomment-283505738) was reasonably straightforward.

A mention in the docs would be nice, but otherwise, I suggest you close this as not-a-bug or wontfix or whatever. I'd prefer that to leaving this open indefinitely.

[dckc]

p.s. I sorta forgot how complete, requires and output and task_id relate today... if I depend on more tasks, does my id change? (answer: no)

The fact that the docs link right to the source and the source of each of those methods is just a handful of lines worked just fine for me.

[mayou36]

Well, I would leave it open, as it is a reasonable feature request. Probably someone starts implementing something at some point.

[Tarrasch]

@mayou36, in favor of less maintinership burden, we try to close eagerly, and let anyone who's willing to do the work to pick it up. :)

---

I'm closing this issue.

• [x] It has been inactive for +4 months.
• [ ] It's not about luigi core, so not as many users are affected about this.
• [ ] The change seems quite big, it's unlikely to be sporadically picked up.
• [ ] The owner haven't responded or disappeared.
• [ ] I don't understand what this issue is about.
• [ ] There exists a reasonable workaround for this.
• [ ] We need to check if this hasn't been fixed by now (for old issues).
• [x] This is kind of by design and not a bug.
• [ ] Resolving this would probably add a lot of complexity.
• [ ] Cannot reproduce
• [ ] This is a question not an issue. Use SO or the mailing list.

Every open issue adds some clutter, and we try to make the issues fewer and make it easier for new collaborators to find. Currently we try to close any issue that meets the first checkbox + one other.

Feel free to reopen this issue at any point if you have the intent to continue to work this. :)

[dckc]

I support closing this as by design, but from the summary views, all one sees is the closed status and the feature request label, so it looks like it's a feature that's been added. I suggest a not-a-bug or by-design or wontfix label.

Also, it's not clear to me how to reopen an issue. I suspect only authorized collaborators can do that.

[dlstadther]

@dckc Only repo owners (or those with write access) and issue owners can reopen.