Luigi: Add support for (recursively) ignoring task failure

[Alex-picwell]

The Reason

We have an existing pipeline that parses a file, validates, and then uses that output in the rest of the pipeline. We are looking to refactor the parsing and validation portion of the pipeline by running the new refactored code alongside the old, and then manually validating the new output matches the old output, maybe running some checks automatically to compare the two, before replacing the old path with new.

The refactoring effort would then look like the following:

|WrapperTask|---->|OldParser|---->|OldValidator|--------->[RestOfPipeline|
                       \                                                                                      /
                         ---->|NewParser|---->|NewValidator|-----------

But currently if the NewPath fails in some way, the whole pipeline will say it failed, which is a bit more extreme then we are hoping for. We would rather be able to say "It's ok if the new path fails, the real pipeline hasn't."

Proposed Change

Just a simple change as an optional param to the Task class critical would be sufficient, other possibilities are an OptionalTask class, but I think that may be a bit overkill. Ideally, this would be intelligent and recursively apply it to all Tasks that follow a non-critical Task

If this is already a supported feature in some way, I did a good deal of digging around so maybe just adding a note on the documentation making this a bit more obvious would be great.

[dlstadther]

This is an interesting (and reasonable) feature.

Let me restate to verify my understanding....

You'd like to add a property of a Task to denote if its failure is critical. By default, all Tasks are critical. But you can optionally change this (similar to setting the priority of a Task) such that its failure still allows the whole pipeline to say it succeeded?

If this is true, how would you denote the summary output? Would it still say all critical Tasks succeeded, but x number optional tasks (or non-critical) passed/failed?

[Alex-picwell]

That is accurate, and I think you are right that the output should mention which optional tasks passed/failed in addition to whether the pipeline failed, however that format is open to suggestions.

If I myself were to suggest, I would say an optional section listing passed/failed next to task name would suffice, simple is always nice.

[dlstadther]

I think I'm good with this feature addition.

I'd expect that default behavior is that all Tasks must succeed in order to receive a pipeline success.

I'm happy to review a code submission! Be sure to include plenty of tests :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.

[DrPyser]

Has anything been done related to this feature?

I guess the only way to deal with this currently, is to swallow all exceptions raised from the optional task's run method, to make sure the task is successful. A wrapper task using delegates could be used. The only issue is then to not simply ignore failures, using a target implementation aware of the failure.