Delta: Delta Lake is not compatible with pyspark 2.4.3

Created on 31 May 2019 · 4Comments · Source: delta-io/delta

Hi Delta Lake team,

I tried to setup for Delta Lake by following this instruction and got the same error as described in this issue. I also left comment about the error and environment in that issue.

Today I tried to downgrade my pyspark version to 2.4.2 and it worked:

pyspark --packages io.delta:delta-core_2.12:0.1.0
Python 2.7.16 (default, Apr 12 2019, 15:32:40)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Ivy Default Cache set to: /Users/xuc/.ivy2/cache
The jars for the packages stored in: /Users/xuc/.ivy2/jars
:: loading settings :: url = jar:file:/usr/local/lib/python2.7/site-packages/pyspark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
io.delta#delta-core_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-bbcee9fd-e9b9-444d-88ac-389b062944c6;1.0
    confs: [default]
    found io.delta#delta-core_2.12;0.1.0 in central
:: resolution report :: resolve 195ms :: artifacts dl 2ms
    :: modules in use:
    io.delta#delta-core_2.12;0.1.0 from central in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   1   |   0   |   0   |   0   ||   1   |   0   |
    ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-bbcee9fd-e9b9-444d-88ac-389b062944c6
    confs: [default]
    0 artifacts copied, 1 already retrieved (0kB/6ms)
19/05/31 12:05:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.2
      /_/

Using Python version 2.7.16 (default, Apr 12 2019 15:32:40)
SparkSession available as 'spark'.
>>> data = spark.range(5, 10)
>>> data.write.format("delta").mode("overwrite").save("/tmp/delta-table")
>>> exit()

Wonder if there is some compatibility issue?

Source

xctom

Most helpful comment

Hi Chen,

There was a bug where Spark 2.4.2 was built with Scala 2.12 instead of 2.11. So if you're using Spark 2.4.2, you need to use Delta Lake 2.12, but for 2.4.3 (and future 2.4.x versions) use Delta Lake 2.11.