Airflow: Add DAG level cluster policy

[turbaszek]

Description

Currently we have only task level cluster policy:
https://github.com/apache/airflow/blob/7dfb3980cecfafdb7d6b79d17455b08971cec7d4/airflow/models/dagbag.py#L386-L387

This is an amazing tool that allows users to validate their tasks by either skipping DAG or setting default values (queues, owners, emails etc). I would like to proposes same mechanism for DAGs.

Some may argue that users already can implement it by accessing task.dag attribute in the policy, however this has the following drawbacks:

• it's not "explicit over implicit" see https://github.com/apache/airflow/issues/12086#issuecomment-722986766
• the "DAG policy" is executed for every task (if there's 1000 tasks in DAG, the DAG policy will be checked 1000 times)

Use case / motivation

Create explicit mechanism for creating "DAG level" cluster policy that will be optimised for this task (check only once per dag not for every task) and will not required users to implement this custom optimisation logic (for example by monkey patching or changes in their fork).

Related Issues

10282

[turbaszek]

CC: @mik-laj @jaketf @swalkowski @JavierLopezT @kaxil

[potiuk]

Good idea !

[mik-laj]

+1

[JavierLopezT]

Nice feature. Thanks for informing me