Spark-on-k8s-operator: Airflow Integration

[ashishwrt]

Has anyone use this operator with Airflow? ie Schedule jobs via airflow instead of built in Cron?
Any Suggestions on how to go about doing it?

Only way I could find to rerun an existing job (SparkApplication CRD) is to do:
1) kubectl delete -f
2) kubectl apply -f

[mrow4a]

Just out of curiosity, what would be the use-case / benefit of using workflow engine instead of cron?

[ashishwrt]

1) If this spark job is part of multistage workflow, may be across different tools/environments, then it is easier to use something like Airflow to compose and manage the whole pipeline.
2) Better management UI, alerting. logs etc.
(If it is a standalone spark job then probably not much to gain except for much nicer management tools/UI.)

[liyinan926]

I tried to build a custom Airflow operator/executor for creating SparkApplication resources a long time ago but never completed it. The draft code is here https://github.com/liyinan926/incubator-airflow/commit/d9e8fd7e68ed88d942a7e16473462d14886ae5ea. Let me know if this is close to what you were thinking of.

[GrigorievNick]

Please look on this.
https://github.com/nikkatalnikov/bigkube/tree/master/airflow/operators
I use this code, in produciton.
Ok not exactly this one, but very similar

[kodelint]

@GrigorievNick will you be able to share a sample dag using the operator. Are you passing the entrie YAML to the python_operator to submit it sparkoperator? I am wondering that if it is even feasible to write a dag as well as the yaml to run the job.

[roitvt]

we've written airflow integration including kubernetes_hook, airflow operator and sensor.
I opened a Jira issue at the airflow project https://issues.apache.org/jira/browse/AIRFLOW-6542 in order to contribute it to airflow contrib.
I'll be glad if you'll try it and share your thoughts.
@liyinan926 can you support me at the Jira issue? thanks

[mik-laj]

Hello,

I am a committer in the Apache Airflow project.
How can I help you?

Best regards,
Kamil

PS. Airflow is used by Google Cloud Composer, so if a team from this project contacts the other team they can get support from Airflow experts.

[liyinan926]

@roitvt sure, I would be glad to support. Thank you for the contribution!

[roitvt]

Hey, I wanted to let you know that my effort to create a native Airflow integration for Spark-on-K8s operator (https://issues.apache.org/jira/browse/AIRFLOW-6542) was just merged, and will be a part of the next Airflow version!
This will allow you to migrate your Airflow-based Spark jobs to K8s, with minimal changes to the DAGs.
Thanks to Airflow committers who helped with this effort - @mik-laj , @kaxil, @ashb

[liyinan926]

@roitvt Wow, this is awesome! Thank you for the effort!

[roitvt]

can we add this to the docs?

[liyinan926]

Sure, can you create a PR for the doc changes?

[roitvt]

for sure will start working on it tomorrow