Delta: Delta Lake in Azure is restricted as of now to Blob Storage and Data Lake Gen1 ?
On the official Data Lake Gen2 documentation, it is stated that the Hadoop support is only for versions 3.2+.
On Delta IO the documentation only refers Azure Data Lake Storage Gen1, whose requirements are: Spark 2.4.2 and above built with Hadoop version 2.9.1 and above (not 3.x)
As far as i can tell, Databricks Runtime 5.5 supports Delta Lake Gen2, but Delta Lake open source does not.
At this time, Delta Lake open-source only supports Azure Data Lake Storage Gen1 and Blob Storage? Is this correct?
Can we expect Azure Data Lake Gen2 support anytime soon?
Thank you!
Nuno
nfplay
>All comments
Spark 3.0 will come with Hadoop 3.2 support, therefore it will be possible to support Gen2 when Spark 3.0 is released with open source Delta Lake.
brkyvz
on 27 Dec 2019
👍4
Was this page helpful?
0 / 5 - 0 ratings
Related issues
marmbrus
路
7Comments
ramizmehran
路
4Comments
tdas
路
6Comments
xctom
路
4Comments
changpingc
路
4Comments
Most helpful comment
Spark 3.0 will come with Hadoop 3.2 support, therefore it will be possible to support Gen2 when Spark 3.0 is released with open source Delta Lake.