Delta: Getting event logging listener exception

[ramizmehran]

Delta version: 2.11:0.6.0
EMR Version: 5.30

Exception:
20/05/28 08:45:00 ERROR AsyncEventQueue: Listener EventLoggingListener threw an exception
java.lang.ClassCastException: java.util.Collections$SynchronizedSet cannot be cast to java.util.List
at org.apache.spark.util.JsonProtocol$.accumValueToJson(JsonProtocol.scala:348)
at org.apache.spark.util.JsonProtocol$$anonfun$accumulableInfoToJson$3.apply(JsonProtocol.scala:324)
at org.apache.spark.util.JsonProtocol$$anonfun$accumulableInfoToJson$3.apply(JsonProtocol.scala:324)
at scala.Option.map(Option.scala:146)

Facing this while using Delta Merge operation

Found a similar error raised here:
https://stackoverflow.com/questions/61799041/strange-non-critical-exception-when-using-spark-2-4-3-emr-5-25-0-with-delta-la

[zsxwing]

Thanks for reporting this. This is caused by https://github.com/delta-io/delta/commit/a1f7976b9c32849c605cfdce8d97254298555f33 : We changed the accumulator used by Merge to an internal accumulator. But Spark event log serialization doesn't handle internal accumulators whose type is SetAccumulator.

This error doesn't impact anything of your jobs. Except it may impact the debugging when you look at the Spark UI on Spark History Server, because the error will cause Spark not log the task events for your merge query and the Spark UI won't be able to render the Spark jobs and stages for the merge query properly. The live Spark UI when your application is running is not impacted.

However, we cannot revert the commit because it's a pretty important fix for the stability of merge queries. We noticed that a non-internal accumulator will be kept in memory pretty long and cause stability issues. For example, if a merge query touches 1 million files, all of file paths will be kept in memory until you run 1000 stages after the merge query.

We will work with the Spark community to fix this internal accumulator issue instead.

[zsxwing]

I'm closing this one. This issue will be fixed in Apache Spark 2.4.7/3.0.1/3.1.0. See https://issues.apache.org/jira/browse/SPARK-31923

[rtjarvis]

I still get this issue in AWS EMR runnning Spark 3.0.1 and Delta 0.7.0. Any ideas why?

[marmbrus]

@rtjarvis I'd suggest asking EMR support. Perhaps they are missing some patches?