Airflow: 2016 Roadmap Item : Allow depends_on_past=True for Skipped tasks

[r39132]

There is a problem with tasks that have been skipped and have depends_on_past=True. Future instances of the task won't run. This does not make sense. Instead, if a previous instance of a task has been skipped and depends_on_past=True, schedule the task anyway -- in essence, treat a SKIPPED as a SUCCESS.

That is what this issue address.

There are many cases where would like to mark a task/DAG as skipped

• Tasks skipped during the execution of ShortCircuit or Branch python operators
• DAGs or Tasks that we would like to manually skip as described in https://github.com/airbnb/airflow/issues/262
• DAGs that specific only_run_latest as in https://github.com/airbnb/airflow/issues/59. In these cases, we want to skip DAG Runs except for the latest. Unfortunately, if we skip all but the latest, the latest will not run if depends_on_past=True

In order to tackle all of these, we need to modify the system to treat Skipped as Success. Additionally, we may want to describe each of these skipped differently (e.g. skipped_condtional, skipped_data_issue, skipped_only_run_latest)

cc @mistercrunch @bolkedebruin @criccomini

[griffinqiu]

+10086

[griffinqiu]

For my case:
Some of tasks run as hourly and some tasks only run as daily in one DAG. The reason of set depends_on_past to True BCS the today's DAGRUN depend on yesterday result.

914

[r39132]

This issue is now fixed. I will be submitting a PR soon.

[r39132]

@griffinqiu Pull the latest and let me know if you have other issues.

[pkexcellent]

Hi, I really need to use the only_run_latest feature, but I do not know how to use is in my code. Is this a feature has not beed developed or I just missed it? Can you give me a example of task declaration ? If this feature is not developed, how can I implement my code to realize the only_run_latest feature? Thanks!