Circe: Getting java.lang.NoSuchMethodError when running in spark 2.1.1 on EMR - shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness;

[alexmnyc]

User class threw exception: java.lang.NoSuchMethodError: shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness;
java.lang.NoSuchMethodError: shapeless.Witness$.mkWitness(Ljava/lang/Object;)Lshapeless/Witness;
at com.x.emr.StreamJob$anon$lazy$macro$16$1.inst$macro$1$lzycompute(StreamJob.scala:240)

line 240 has

implicit val myEncoder = deriveEncoder[EventSample]

[travisbrown]

This is probably a duplicate of #574. The issue is that Spark depends on an ancient version of Shapeless, and the solution is to shade one of the versions. I'm not a Spark user and can't help much here, but I know a lot of other people are doing this—it'd be great for one of them to give a complete working example either in one of these issues or in the docs.

[alexmnyc]

getting rid of

implicit val myEncoder = deriveEncoder[EventSample]

and replacing it with

implicit val encodeUser: Encoder[EventSample] = Encoder.forProduct3("event", "session_id", "timestamp")(e => (e.event, e.sessionId, e.timestamp))

worked as a workaround for the time being

[travisbrown]

@alexmnyc Right, circe-core doesn't share any dependencies with Spark (except the Scala standard library), so it won't conflict.

[pomadchin]

The workaround is to shade overlapping deps (assembly plugin example). I am not sure that it can be accounted as a solution, but shading is a pretty normal thing, especially in terms of spark.

[Thundzz]

In order to save time for those who use sbt and are facing the same issue, this can be achieved by adding the following to your build.sbt

.settings(
    assemblyShadeRules in assembly := Seq(
        ShadeRule.rename("shapeless.**" -> "shadeshapless.@1").inAll
    )
)

[ALPSMAC]

I don't believe that just shading shapeless actually fixes the problem in this case.

I believe the root issue can be traced back to ExportMacros.scala where there's a quasiquote reference to shapeless in its original package.

If you define a separate project in order to shade shapeless in circe and then depend on that project in a different build (as in the example here: http://manuzhang.github.io/2016/10/15/shading.html) you'll find that the macros will typecheck against a package that doesn't exist (because its been shaded).

In my specific use-case I'm trying to dynamically load Jars on the classpath (similar to OSGi or Modules) from within my Spark driver application.

I wonder if there is any way the macro can be made smarter... perhaps looking for a system property indicating if shapeless has been shaded and where the resultant root package for shapeless lives?

[travisbrown]

Closing this issue since I'm not interested in hacking circe-generic to try to determine whether Shapeless has been shaded and adjusting the macro expansion appropriately, especially since this should be resolved with the latest Spark releases. Please feel free to reopen if it's still occurring.