Airflow: Add Dataproc Job Sensor

Created on 11 Aug 2020  路  3Comments  路  Source: apache/airflow

Description

The Dataproc hook now has separate methods for submitting and waiting for job completion.
We should improve this further by adding a method to simply poke the status of the job.
Finally implement a DataprocJobSensor so we can use reschedule mode for jobs known to run for a long time.

Use case / motivation

For Long running dataproc jobs that are known to run for a long time (e.g. > 1 hr) we should not block a slot for the entire duration of the job.

Related Issues

N/A

* Other notes *
@varundhussa is currently working on this for a customer with this use case.

providers good first issue feature Google
Source
picture jaketf
👍3

Most helpful comment

Thanks @rootcss and @mik-laj
I have completed the sensor and am completing the test cases. I will raise a PR soon.

picture varundhussa on 24 Aug 2020
👍3 ❤1

All 3 comments

I'd like to work on this. Can you please assign this to me @mik-laj

rootcss picture rootcss on 22 Aug 2020

@varundhussa is currently working on this for a customer with this use case.

mik-laj picture mik-laj on 22 Aug 2020
👍2

Thanks @rootcss and @mik-laj
I have completed the sensor and am completing the test cases. I will raise a PR soon.

varundhussa picture varundhussa on 24 Aug 2020
👍3 ❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

xu-xiang picture xu-xiang  路  3Comments
turbaszek picture turbaszek  路  3Comments
ryanahamilton picture ryanahamilton  路  3Comments
mik-laj picture mik-laj  路  4Comments
d-lee picture d-lee  路  4Comments