Delta: Pandas support for Delta Lake

[jackwellsxyz]

Hi Delta Lake community, I had asked the Pandas team to investigate adding Delta Lake support since I bounce between Pandas and Spark, and they said they wouldn't support it since it looks like Delta Lake seems tied to Spark for its APIs (see pandas issue 35017). Is this true?

The only stable public APIs, currently provided by Delta Lake, are through the DataFrameReader/Writer (i.e. spark.read, df.write, spark.readStream and df.writeStream). Options to these APIs will remain stable within a major release of Delta Lake (e.g., 1.x.x).

All other interfaces in this library are considered internal, and they are subject to change across minor/patch releases.

I personally would love to start using Delta Lake more extensively in my machine learning projects, but I'd be nervous if I had to give up on being able to read it in to my laptop for exploratory data analysis. Thanks for considering!

[brkyvz]

@jackwellsxyz Can you use koalas? https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.read_delta.html

[jackwellsxyz]

Thanks @brkyvz for the feedback! Ultimately, I'm afraid that if I store data in Delta Lake format, then I tie myself to only analyzing it using Spark, as koalas only runs on Spark if I'm not mistaken. It would be nice to be able to open the data in pandas in a Jupyter notebook on my local laptop as well. Are there any architectural limitations of the Delta Lake file format that would prevent it from being opened on a local machine with pandas? For example, parquet and json formats are both well supported in pandas via the Apache Arrow and fastparquet libraries.

[manucohenyashar]

I agree. It makes no sense to bind Delta to Spark. Delta is a storage format while Spark is an execution engine. Modern data management tend to separate storage from compute (unlike to old days of Hadoop). Exposing Delta to Pandas will open the doors to countless use cases.

[vmesel]

I agree with @manucohenyashar! In the company I work for, we need to use the delta formats on the gold data layer to enable us keep versioning and continue improving our pipeline.

[philMarius]

Is there an update on this?

[houqp]

For those who need access to delta lake natively outside of JVM, i recommend giving https://github.com/delta-io/delta-rs a try. Its python binding already supports reading delta tables using pandas, see examples at https://github.com/delta-io/delta-rs/tree/main/python#usage.

I think it's best to leave the official reference implementation depend on spark because that's the only way to process large scale delta tables.

[Sutyke]

Pandas support already parquet. Pandas contributors would implement delta lake if databricks creates the same pyarrow connector as for parquet. Can databricks help?

Pandas developers are excellent with simplifications. While parquet works the best with spark for data that doesn't fit to one machine, parquet is used a lot with pandas for data that fits to single machine. It will be great expansion for pandas to have deltalake. This will makes delta lake and databricks more popular and it will brig more customers to databrikcs for users that data doesn't fit to single machine.

Below is pandas contributor reply:
https://github.com/pandas-dev/pandas/issues/40573

[houqp]

@Sutyke see https://github.com/delta-io/delta-rs/tree/main/python#usage, we already support processing delta tables with native pandas.

[manucohenyashar]

The idea is not only to read delta tables to a data frame but also to
persist a data frame to a delta table.
I don't see that supported today.
In short, it should be possible to work with delta tables without using
Spark anywhere in the pipeline.

On Thu, Mar 25, 2021 at 2:34 AM QP Hou @.*> wrote:

@Sutyke https://github.com/Sutyke see
https://github.com/delta-io/delta-rs/tree/main/python#usage, we already
support processing delta tables with native pandas.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/delta-io/delta/issues/467#issuecomment-806401006, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ABULBRY7IM64N7UINXD7ADDTFLKPFANCNFSM4OK2ZJAA
.

[houqp]

@manucohenyashar that's our medium term goal, we will be using delta-rs in https://github.com/delta-io/kafka-delta-ingest to perform delta table write for streaming input, then process them in spark and other ETL pipelines downstream. Delta-rs currently already have append delta table write working in a PoC, we are working on making it production ready.