Delta: NoSuchMethodError in LogicalPlan

[kkr78]

We are having the following issue while upgrading from EMR 5.28 to 5.29. Both EMR versions support Spark 2.4.4. We rolled back due to this issue. is this issue specific to EMR the way packaging spark jars? Any idea?

: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Lorg/apache/spark/sql/catalyst/plans/logical/LogicalPlan;)Lorg/apache/spark/sql/catalyst/plans/logical/LogicalPlan;
at org.apache.spark.sql.delta.util.AnalysisHelper$class.tryResolveReferences(AnalysisHelper.scala:33)
at io.delta.tables.DeltaMergeBuilder.tryResolveReferences(DeltaMergeBuilder.scala:121)
at io.delta.tables.DeltaMergeBuilder$$anonfun$1.apply(DeltaMergeBuilder.scala:228)
at io.delta.tables.DeltaMergeBuilder$$anonfun$1.apply(DeltaMergeBuilder.scala:228)

[zsxwing]

This looks like a EMR specific issue. This method is in Apache Spark 2.4.4: https://github.com/apache/spark/blob/v2.4.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L118

[guyneedhamcanva]

We are also experiencing this issue with EMR 5.29. When writing to S3, merge operations fail with that exception. Overwrite operations are not impacted.

[marmbrus]

This is unfortunately something you will probably have to raise with the EMR team. As they often don't upstream their changes to Spark, it is not really possible for other projects to figure out what changes they are making internally. As @zsxwing points out, it does look like the missing API is part of Apache Spark 2.4.4. I'm not sure why EMR is choosing to remove / change it.

[jsaurav]

We are also experiencing the same issue with EMR version 5.29.0, and after downgrading to version 5.28.0 everything started working as expected.

[kkr78]

I just got an update from the EMR team, it will be fixed in EMR 5.30.0. There is no ETA when that will be released. I think this issue can be closed.

[tdas]

@kkr78 thank you for the update!

[Antauri]

Also a problem in EMR 6.0.0 for the interested.

[Zhujun-Vungle]

Double confirm, still exists in EMR 6.0.0.

[ejono]

Hi, I'm the engineer on EMR who actually caused and fixed this problem. (So sorry!) Unfortunately, emr-6.0.0 was already released by the time we became aware of this issue in emr-5.29.0 and fixed it for emr-5.30.0, so this is why it is still present in emr-6.0.0. Rest assured that it will be fixed in both emr-5.30.0 and emr-6.1.0 when they come out.

[Antauri]

No problems. ETAs for both versions?

[tdas]

@ejono Thank you very much for confirming the fix versions. Any ETA would be great!

[ejono]

Sorry, but I'm unable to provide ETAs.

[Antauri]

5.30 seems to be out, for the interested. Will test and see.

[guyneedhamcanva]

Have tested this on 5.30, the merge operation now works 🥳

[tdas]

@Antauri @guyneedhamcanva thank you for confirming!