Airflow: Add ability to run some tasks on LocalExecutor but rest on Celery/DaskExecutor

[tooptoop4]

is there a way to set executor per task? ie I want all tasks (except task named abc) to use celeryexecutor/daskexecutor but I want task named abc to use local executor (due to firewall only opened from master)

https://github.com/apache/airflow/pull/6725 has some relevant code, hoping it can be merged!

[potiuk]

I once added a POC for multiple executors based on queue - would you find it useful ?

https://github.com/PolideaInternal/airflow/commit/c87e87a5e7673e67e78d4ce39d5a15a8cc494e56

[tooptoop4]

can u pls merge to apache airflow repo?

[potiuk]

Well. I could resurrect it yeah. But only after I finish some of the most urgent things with CI/Kubernetes/prod image - but If you would like to pick it up till then - feel free @tooptoop4 :)

[joshowen]

Hi @potiuk I do think this is worth resurrecting. Having a path to migrate between executors is helpful with a larger deployments, especially now that the Kube executor is a thing. I assume I could build a custom executor to handle the executor instance routing, but making it easily configurable by dag/task is definitely preferable.