Circe: Auto-derived decoder fails with NoClassDefFoundError

[jonas]

Not sure if this is a bug in circe or with how circe is used, or a problem related with spark and circe classpaths or something entirely different. Would appreciate any pointers how to debug this.

build.sbt

scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
libraryDependencies ++= Seq("core", "generic").map(module => "io.circe" %% s"circe-$module" % "0.7.0")

Main.scala

import org.apache.spark.sql._
import io.circe.generic.auto._
import io.circe.Json

case class Repo[T: Encoder](path: String) {
  def read(spark: SparkSession): Seq[T] =
    spark.read.option("header", true).option("inferSchema", true).csv(path).as[T].collect().toSeq
}

case class Record(a: Int, b: Int, c: Int)

object Main {
  val json = Json.obj("path" -> Json.fromString("data.csv"))

  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().master("local[1]").getOrCreate()

    import spark.implicits._
    val repo = json.as[Repo[Record]].right.get

    println(repo.read(spark))
  }
}

data.csv

a,b,c
1,2,3

console

> sbt clean run
[info] Set current project to circe-error (in build file:/opt/circe-error/)
[success] Total time: 1 s, completed 25-Feb-2017 11:23:00 AM
[info] Updating {file:/opt/circe-error/}circe-error...
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.
[info] Compiling 1 Scala source to /opt/circe-error/target/scala-2.11/classes...
[info] Running Main
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/02/25 11:23:28 INFO SparkContext: Running Spark version 2.1.0
[...]
17/02/25 11:23:30 INFO SharedState: Warehouse path is 'file:/opt/circe-error/spark-warehouse/'.
[error] (run-main-0) java.lang.NoClassDefFoundError: Main/repo$anon$macro$18
java.lang.NoClassDefFoundError: Main/repo$anon$macro$18
        at Main$anon$exportDecoder$macro$21$1$anon$macro$18$1.from(Main.scala:19)
        at Main$anon$exportDecoder$macro$21$1$anon$macro$18$1.from(Main.scala:19)
        at shapeless.LabelledGeneric$$anon$1.from(generic.scala:229)
        at shapeless.LabelledGeneric$$anon$1.from(generic.scala:226)
        at io.circe.generic.decoding.DerivedDecoder$$anon$1.apply(DerivedDecoder.scala:14)
        at io.circe.Json.as(Json.scala:97)
        at Main$.main(Main.scala:19)
        at Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
Caused by: java.lang.ClassNotFoundException: Main.repo$anon$macro$18
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at Main$anon$exportDecoder$macro$21$1$anon$macro$18$1.from(Main.scala:19)
        at Main$anon$exportDecoder$macro$21$1$anon$macro$18$1.from(Main.scala:19)
        at shapeless.LabelledGeneric$$anon$1.from(generic.scala:229)
        at shapeless.LabelledGeneric$$anon$1.from(generic.scala:226)
        at io.circe.generic.decoding.DerivedDecoder$$anon$1.apply(DerivedDecoder.scala:14)
        at io.circe.Json.as(Json.scala:97)
        at Main$.main(Main.scala:19)
        at Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)

[stevenchen3]

Encountered the same issue with circe-0.8.0

[stevenchen3]

Some update:

For my case, it is the dependency conflict between circe and spark. circe-generic % 0.8.0 and circe-generic-extras % 0.8.0 use shapeless % 2.3.2, while spark use shapeless % 2.0.0. And shapeless is no backward compatible in this case.

The workaround is to shade shapeless % 2.3.2 to another name during sbt assembly.

[jonas]

Current master of Spark depends on breeze version 0.13.2 which uses shapeless version 2.3.2. So there's hope that this will resolve itself.

[jonas]

I was able to reproduce this with Spark 2.2.0 which uses breeze 0.13.1 and shapeless 2.3.2.