Airflow: Calendar Based Scheduling Pattern - Scheduler Feature in DAG

[abdul-786]

The current airflow scheduler architecture can scan the DAG, understand the cron schedule defined in the DAG and submit a task to a local/remote executor. This works best for most of use cases to automate any flavor of workflows. The cron schedule can fit most of workflow patterns but there are few complex business scheduling requirements as specified below.

1. Run the workflow on every alternative week ends and don't run on holidays
2. Run the workflow on every week and include few holidays except Christmas and New Year.

These 2 are sample examples.

Use case / motivation

Most of licensed scheduler have calendar based scheduling feature where users can pick or customize schedule according to pre-defined calendar for each workflow.

*Feature Request * :

1. Have a configurable option to use customized calendar based schedule or cron based
   schedule for each DAG.
2. Scheduler can read the option at each DAG level schedule option and act according to calendar based schedule or cron based schedule
3. Webserver DAG Graph view can display the calendar based schedule or cron based schedule.

Note: This feature request is based on usage of airflow for more than 3 to 4 years and felt it would be a great feature to incorporate in the airflow architecture.

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[turbaszek]

Dev list thread: https://lists.apache.org/thread.html/rd683c05321c2d6bddc1a667e6bf24eb26279810d0e0936da528a0a2f%40%3Cdev.airflow.apache.org%3E

[hlokired]

Hi Mil-laj,

I can see that the request #9703 has been closed saying it is duplicate. But when I go through #9096. I don't see the exact question or solution for the question.
Could you please let us know will these be a future request. if Yes, then by when can we get a solution.
Because we have a business requirement and need to respond to the business.

Thanks.

Regards,
Harish

[mik-laj]

@hlokired Apache Airflow is an Open Source project, so the community manages it. Each feature that is added to this project is the result of volunteer work or was done by other companies that decided to share their changes. We don't have any paid employees who write code that other people expect, because it's free software. For this reason, you shouldn't expect specific dates when the feature will be finished. Each ticket will be completed when the community gets consensus on the design, and then other contributors find time to implement this change. After a positive review, it will be merged.

There is also a positive side - you can make any change to the source code to meet the requirements of your business. If you want, you can also share your patch with the community. When it is reviewed positively, it will be merged into the project.

You can also participate in the mailing list discussion to increase the chance of solving this problem faster. Each voice is important for discussion because it allows you to find a solution that meets the requirements of all our users.

If your business expects you to require specific changes, you can also hire other companies that deal with the development of Airflow. Such a company can prepare modifications for you. I work for one of them - Polidea.

I hope that the solutions presented help to solve your problem.

[hlokired]

Hi,

Can you please provide the mailer list, when we can discuss with in the group.

Thanks,

[mik-laj]

@hlokired Here is thread: https://github.com/apache/airflow/issues/9096#issuecomment-637506595

[hlokired]

Hi Team,

Could you please help us on this requirement, #9703.

Regards,
Harish Reddy L

[hlokired]

Hi Team,

Could anyone help us on the request/requirement please?

Regards,
Harish

[elwinarens]

@hlokired Could you please be more respectful and understand this is a community of volunteers? Jeez.

We'll be investigating extending scheduling with calendar features, but considering the current fundamental cron based/opionated nature, this might justify an AIP, just to be sure the right design is in place (extending cron syntax? impact? feasibility? etc.)