Delta: Support Delta Lake as Apache Beam I/O
Is there plans to integrate delta.io with Apache Beam?
For example - ParquetIO is supported: https://beam.apache.org/documentation/io/built-in/
Delta being open-source storage layer having Apache Beam I/O can help us integrate directly into beam pipelines..
ChethanUK
All 3 comments
This isn't currently on our roadmap, but we'd be happy to provide support if someone from the community wanted to build this.
If Beam has any general-purpose connectors that use Spark's DataFrameWriter/DataFrameReader APIs, one option is to just use those. You can read from Delta by following the Delta protocol (https://github.com/delta-io/delta/blob/master/PROTOCOL.md), though we're also working on a general-purpose Delta Lake reader that will help with that. Writing to Delta is a bit trickier, as right now there's not really a supported way to do that outside of Spark.
mukulmurthy
on 11 Nov 2019
245 tracks the work to build a library for querying delta metadata.
marmbrus
on 11 Nov 2019
It would be really good to develop read/write interface outside of spark. Specailly beam would be really ideal. In general which component of delta is tied to spark? I do see another project for connectors that also only works for reads no writes. What would be option for people who are building large scale data pipe lines? I'm looking to build a massive data pipeline which moves billions of rows. I believe that is where apache beam really make sense.
kesavkolla
on 10 Dec 2020
Related issues
ashitabh
路
4Comments
marmbrus
路
7Comments
tdas
路
6Comments
xctom
路
4Comments
gaurav8297
路
3Comments
Most helpful comment
It would be really good to develop read/write interface outside of spark. Specailly beam would be really ideal. In general which component of delta is tied to spark? I do see another project for connectors that also only works for reads no writes. What would be option for people who are building large scale data pipe lines? I'm looking to build a massive data pipeline which moves billions of rows. I believe that is where apache beam really make sense.