Luigi: Feature: Integration with Databricks API

Created on 4 Aug 2017 · 7Comments · Source: spotify/luigi

Databricks (Spark-as-a-service) recently added native support to Airflow for their Jobs API (https://docs.databricks.com/api/latest/jobs.html, https://databricks.com/blog/2017/07/19/integrating-apache-airflow-with-databricks.html).

I'd like to work on adding a similar functionality to Luigi as a new contrib submodule. Still thinking through what targets/outputs would be, will add thoughts here as I start working through it.
Any thoughts or comments?

wontfix

Source

colemanja91

👍3

Most helpful comment

Are there still any plans to add this functionallity? I see a stale PR here that started to get it to work...

wkeithvan on 9 Apr 2020

👍3

All 7 comments

I think a DatabricksSparkSubmitTask using Databricks' Jobs API and another module that could kick-off a cluster using Databricks' Cluster API would be very useful.

a1xt06 on 6 Aug 2017

@a1xt06 I hadn't thought about kicking off a cluster! Might take a pass at that too. Was also wondering if having a set of DBFS targets would be helpful.

colemanja91 on 6 Aug 2017

DBFS targets might be useful, but as a first pass, using Luigi's S3Target for targets would probably be enough, especially considering most DBFS endpoints will be associated with mounted s3 buckets. I'm guessing doing DBFS endpoints would require using Databricks' DBFS API, and I haven't looked into that API.

At the same time, it might be nice to have DBFS endpoints as a part of a roadmap, considering Databricks' is trying to add Azure support (maybe?) later this year, so the S3Target might not be enough for Luigi on Microsoft's cloud.

a1xt06 on 6 Aug 2017

@a1xt06 I just added a pull request (took me long enough, eh); if you have the ability to do some light review or testing would appreciate your input.

And since now Databricks does have Azure support (with DBFS pointing to Azure Blob Storage) it may be more helpful to have DBFS targets defined. I might be able to get started on this over the holiday break.

colemanja91 on 3 Dec 2017

See https://github.com/spotify/luigi/pull/2312 for an Azure Data Lake file storage option. I may write a module to support Azure Blob Storage as well.

kabirkhan on 14 Dec 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.