Scalapb: Scalapb plugin v0.5.34 failed to generate on scala.js 0.6.11

Created on 27 Jul 2016 · 41Comments · Source: scalapb/ScalaPB

After I upgraded scala.js to 0.6.11 (released on maven, might not be on scala.js website as the moment), I see this error:

.../ancillary/investor_events.proto]
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    com/trueaccord/scalapb/Scalapb.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistry;)V @4: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/Extension'
  Current Frame:
    bci: @4
    flags: { }
    locals: { 'com/google/protobuf/ExtensionRegistry' }
    stack: { 'com/google/protobuf/ExtensionRegistry', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0x0000000: 2ab2 0009 b600 0a2a b200 0bb6 000a 2ab2
    0x0000010: 000c b600 0ab1

Source

ngbinh

Most helpful comment

I did a manual git bisect to find out the commit that introduced this issue and it's indeed the update of the closure compiler in scala-js/scala-js@a5a3e8170072d320febe1f260886dcce7c624a5b

It looks like the closure compiler depends on protobuf 2.5.0 (I've had binary compatibility issues with 2.5.0 and 3.0.0 in the past) but I am not sure why only the recent Closure upgrade surfaced it.

thesamet on 29 Jul 2016

👍2

All 41 comments

This one is interesting. Not sure why ScalaJS is interfering with the code generation step.
Test project: https://github.com/thesamet/scalapbjs-test

thesamet on 27 Jul 2016

I can indeed reproduce the problem on Linux with OpenJDK 1.8.0_40. And the problem goes away if I downgrade to Scala.js 0.6.10.

I have no idea how Scala.js can possibly interfere here. I initially though there might be something about our upgrade of Closure (if PB itself uses it somehow), but the classpath of the build does not mention any interference with another version of Closure.

Note that I guess this warning from sbt:

[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * com.google.protobuf:protobuf-java:2.6.1 -> 3.0.0-beta-3

But that's also true with 0.6.10.

I'm afraid I do not have more insight to offer. :s

sjrd on 27 Jul 2016

@sjrd could it be that ScalaJS 0.6.11 somehow brings the project's library dependencies into the class loader used by ScalaPB sbt plugin? Google protobuf runtime is written in Java and ScalaPB (JVM) depends on it. In order to make ScalaPB work with ScalaJS, we wrote https://github.com/trueaccord/protobuf-scala-runtime to provide the minimal required functionality natively in Scala. A possible theory is that somehow the Scala stubs that are meant to be used in a JS environment get loaded into the JVM during the compile stage...

thesamet on 27 Jul 2016

could it be that ScalaJS 0.6.11 somehow brings the project's library dependencies into the class loader used by ScalaPB sbt plugin?

Not per se ... It really depends on how you create that class loader, and from which information.

sjrd on 27 Jul 2016

I did a manual git bisect to find out the commit that introduced this issue and it's indeed the update of the closure compiler in scala-js/scala-js@a5a3e8170072d320febe1f260886dcce7c624a5b

It looks like the closure compiler depends on protobuf 2.5.0 (I've had binary compatibility issues with 2.5.0 and 3.0.0 in the past) but I am not sure why only the recent Closure upgrade surfaced it.

thesamet on 29 Jul 2016

👍2

Workaround: add ScalaPB before scala-js to your project. Create project/plugins.sbt with the following lines:

addSbtPlugin("com.trueaccord.scalapb" % "sbt-scalapb" % "0.5.34")

addSbtPlugin("org.scala-js" % "sbt-scalajs" % "0.6.11")

In this order - it works. If you switch the order, the VerifyError exception is back.

thesamet on 29 Jul 2016

@thesamet thanks for the work around.

@sjrd I thought Google Closure is only used in fullOptJs mode. Also, should scala.js compiler uses a shaded version to avoid conflict with other plugins?

ngbinh on 29 Jul 2016

workaround works also for me... please notify when a full fix is available!

andreaTP on 29 Aug 2016

Thanks for the workaround!
FYI, it doesn't help with most recent versions: 0.5.40 and 0.6.12 respectively. Seems that the only option is to downgrade.

UPD: Actually, it works with these versions. I just had another plugin dependency, that was bringing in ScalaJS 0.6.12, and it was before the scalapb statement.
Moving it after ScalaJS statement fixed the issue

vpavkin on 7 Sep 2016

@sjrd any plan to update the protobuf dependency in ScalaJS?

nadavsr on 7 Sep 2016

When the Google Closure Compiler updates it, yes. I'm afraid it would be difficult/dangerous for us to change the version that GCC relies on.

sjrd on 7 Sep 2016

Maybe for the time being we'll ship the ScalaPB sbt plugin with a shadowed protobuf-java then.

nadavsr on 7 Sep 2016

Just got bitten by the same issue. 'Solved' it by replacing bundled google closure compiler with a newer one. Here is a relevant snipped from the project/plugins.sbt:

addSbtPlugin("org.scala-js" % "sbt-scalajs" % "0.6.13")
// excludeDependencies task didn't work 
libraryDependencies := libraryDependencies.value.map(
    _.exclude("org.scala-js", "closure-compiler-java-6")
     .exclude("org.scala-js", "closure-compiler-externs-java-6")
)
val deps = Seq(
    "com.google.javascript" % "closure-compiler"         % "v20161024"
  , "com.google.javascript" % "closure-compiler-main"    % "v20161024"
  , "com.google.javascript" % "closure-compiler-externs" % "v20161024"
  , "com.google.protobuf"   % "protobuf-java"            % "3.1.0"
  , "com.github.os72"       % "protoc-jar"               % "3.1.0.1"
)
libraryDependencies ++= deps
dependencyOverrides ++= deps.toSet

Note that original google closure compiler is built with java 7, so no compatibility with java 6 here.

avakhrenev on 13 Nov 2016

I saw this issue resurfaced after update to scalapb 0.5.45 and scala.js 0.6.13. Would be great if ScalaPb sbt plugin can use shadowed version of protobuf-java.

ngbinh on 21 Nov 2016

@ngbinh I released an experimental workaround - can you try and let me know if it works for you?

To test, in project/scalapb.sbt, replace

addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.1")

with

addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.1" exclude ("com.trueaccord.scalapb", "protoc-bridge_2.10"))

In the same file, replace the dependency on compilerplugin with a dependency on compilerplugin-shaded:

libraryDependencies += "com.trueaccord.scalapb" %% "compilerplugin-shaded" % "0.5.45-p2"

See
https://github.com/thesamet/scalapbjs-test/commit/4487bd1fc2254c3e5ebea1b80af1d0b5c1ca5f59

The version used is 0.5.45-p2 - if this goes well, I'll release the shaded jar for every version of ScalaPB, until we have a better solution.

thesamet on 25 Nov 2016

Thanks. Will try it out and report the result back here.

ngbinh on 26 Nov 2016

Tried it out and this is the error:

[warn 11:07:12:PM]      ::::::::::::::::::::::::::::::::::::::::::::::
[warn 11:07:12:PM]      ::          UNRESOLVED DEPENDENCIES         ::
[warn 11:07:12:PM]      ::::::::::::::::::::::::::::::::::::::::::::::
[warn 11:07:12:PM]      :: scalapb.com.google.protobuf#protobuf-java;3.0.0: not found
[warn 11:07:12:PM]      ::::::::::::::::::::::::::::::::::::::::::::::

ngbinh on 27 Nov 2016

@ngbinh Looks like protoc-bridge is adding a shaded artifact name to your library dependencies. I think this can happen only if you generate Java code.

Can you try adding this to your build:

libraryDependencies -= "scalapb.com.google.protobuf" % "protobuf-java" % "3.0.0"

thesamet on 27 Nov 2016

The above issue is fixed in 0.5.45-p3.

thesamet on 28 Nov 2016

Thanks @thesamet it seems to work but I am not sure how my project still trying to pull in "com.google.protobuf" % "protobuf-java" % "3.0.0"? It causes problem for me because it is conflicted with the newer "com.google.protobuf" % "protobuf-java" % "3.1.0" in our build.

ngbinh on 28 Nov 2016

@ngbinh sbt should evict the older version if you depend on multiple versions of the same package. Try to see what's going on using dependencyTree if you have https://github.com/jrudolph/sbt-dependency-graph installed.

thesamet on 28 Nov 2016

I am using ConflictManager.strict http://www.scala-sbt.org/0.13/docs/Library-Management.html#Conflict+Management so it didn't work. I can white-list it but would be nicer if I can solve the root problem. Using dependencyTree shows that my project pulls the older version in but I cannot find where it does that. Maybe sbt-protoc plugin pulls protobuf-java in automatically?

ngbinh on 28 Nov 2016

Yes, sbt-protoc adds it to your library dependencies.. Actually, you can choose which version of protobuf-java it will pull in, by using PB.gens.java("3.1.0") in the PB.targets key in your build.sbt

thesamet on 28 Nov 2016

Sorry for my ignorance, how do you actually do that? I tried

    PB.targets in Compile := Seq(
      scalapb.gen(
        javaConversions = false,
        grpc = false,
        singleLineToString = false
      ) -> (sourceManaged in Compile).value,
      Target(PB.gens.java("3.1.0"), sourceManaged.value) // added line
    )

But it doesn't seem to work.

ngbinh on 28 Nov 2016

Almost. It should be:

PB.targets in Compile := Seq(
  scalapb.gen(
    javaConversions = false,
    grpc = false,
    singleLineToString = false
  ) -> (sourceManaged in Compile).value,
  PB.gens.java("3.1.0") -> sourceManaged.value
)

thesamet on 28 Nov 2016

Hmm, the conflict is still there. So strange that it happens to other projects not that one that contain protobuf messages or the one that have protobuf settings

ngbinh on 28 Nov 2016

Maybe it somehow depends on 3.0.0 in a way that's unrelated to ScalaPB? Can you send a minimal example?

thesamet on 28 Nov 2016

Sure, will look into that

ngbinh on 28 Nov 2016

It looks to me like sbt-protoc applies the dependency to ALL projects. Is there a way to only enable it on certain ones?

ngbinh on 30 Nov 2016

It would only apply it to projects that have java in their PB.targets in Compile, however if you want to try to disable the plugin for some projects, add disablePlugins(sbtprotoc.ProtocPlugin) to their settings.

thesamet on 30 Nov 2016

That's strange because I don't even use Java on our protobuf project.

ngbinh on 30 Nov 2016

Interesting, if you can send over a minimal example that can really help.

thesamet on 30 Nov 2016

Adding disablePlugins(sbtprotoc.ProtocPlugin) to all the projects work. So I think there might be a bug on sbt-protoc as all of them don't have PB.targets set.

ngbinh on 30 Nov 2016

My bad - generating for Java is the default setting for sbt-protoc https://github.com/thesamet/sbt-protoc/blob/8cf503637cdd1ba1531a202571becf1e70e0d2a6/src/main/scala/sbtprotoc/ProtocPlugin.scala#L71

thesamet on 30 Nov 2016

Hmm, not sure why I get java files instead of scala when using 0.5.45-p3

ngbinh on 30 Nov 2016

Just an FYI for people upgrading to play 2.6.0 (or rather play-json): It pulls scala-js and you need to follow the instructions for scala-js to make it work

elmalto on 4 Jul 2017

❤1

Hi @thesamet this issue may still be reproduced with some special configs on plugins.sbt. In particular, the workaround introduced in https://github.com/scalapb/scalapb.github.io/commit/3cd3fcd37d153a74fba1b24a18b9780b079baff3 doesn't seem to solve all cases.

Like @elmalto I'm having issues caused by latest updates in Play 2.6 (bumped version of ScalaJS). In my case I'm using Lagom (which uses Play, which uses ScalaJS). In the plugins.sbt for my experiments I have something as simple as:

// VALID SETUP
//libraryDependencies += "com.trueaccord.scalapb" %% "compilerplugin-shaded" % "0.6.0"
//addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.11")
//addSbtPlugin("com.lightbend.lagom" % "lagom-sbt-plugin" % "1.4.0-M1")

// INVALID SETUP
addSbtPlugin("com.lightbend.lagom" % "lagom-sbt-plugin" % "1.4.0-M1")
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.11")
libraryDependencies += "com.trueaccord.scalapb" %% "compilerplugin" % "0.6.0"

I could narrow down the workaround to always adding the dependency to compilerplugin-shaded _before_ adding the sbt-protoc plugin. I don't think that's what suggested in https://github.com/scalapb/scalapb.github.io/commit/3cd3fcd37d153a74fba1b24a18b9780b079baff3

ignasi35 on 5 Jul 2017

@ignasi35 In the "VALID SETUP" you are missing the exclude ("com.trueaccord.scalapb", "protoc-bridge_2.10")) from the line with addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.11" - see https://scalapb.github.io/scala.js.html

In the "INVALID SETUP" example you're not using the compilerplugin-shaded so it's expected not to work with ScalaJS.

If there are still problems once you add the missing inclusions can you provide a git repo demonstrating it?

thesamet on 5 Jul 2017

Hi @thesamet, I should have included more information in my previous comments. I tried both alternatives you mention and in neither of them I obtained successful results.

Let me elaborate.

In the "VALID SETUP" you are missing the exclude ("com.trueaccord.scalapb", "protoc-bridge_2.10")) from the line with addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.11" - see https://scalapb.github.io/scala.js.html

I didn't observe any improvement with or without the exclusion.

In the "INVALID SETUP" example you're not using the compilerplugin-shaded so it's expected not to work with ScalaJS.

IIRC using the shaded dependency in the "INVALID SETUP" didn't fix the issue.

I'll try to review my tests and report details wrt actual errors on each case. I am aware that since I'm using Lagom the dependency resolution may be a lot more complex than what other users (that only depend on ScalaJS and ScalaPB) are observing.

ignasi35 on 6 Jul 2017

Using:

// INVALID SETUP
addSbtPlugin("com.lightbend.lagom" % "lagom-sbt-plugin" % "1.4.0-M1")
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.11")
libraryDependencies += "com.trueaccord.scalapb" %% "compilerplugin" % "0.6.0"

I get:

protoc-jar: executing: [/var/folders/xc/46s2thrs5ss3w70292zz4bf80000gn/T/protocjar3533120149096813379/bin/protoc.exe, --plugin=protoc-gen-scala=/var/folders/xc/46s2thrs5ss3w70292zz4bf80000gn/T/protocbridge645033645081919833, --scala_out=grpc:/Users/ignasi/git/projects/lightbend/grpc-sbt-experiments/helloscala-grpc-server/target/scala-2.11/src_managed/main, -I/Users/ignasi/git/projects/lightbend/grpc-sbt-experiments/helloscala-grpc-server/src/main/protobuf, -I/Users/ignasi/git/projects/lightbend/grpc-sbt-experiments/helloscala-grpc-server/target/protobuf_external, /Users/ignasi/git/projects/lightbend/grpc-sbt-experiments/helloscala-grpc-server/src/main/protobuf/helloworld.p[info] Resolving org.reactivestreams#reactive-streams;1.0.0 ...
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    com/trueaccord/scalapb/Scalapb.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @4
    flags: { }
    locals: { 'com/google/protobuf/ExtensionRegistryLite' }
    stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0x0000000: 2ab2 0009 b600 0a2a b200 0bb6 000a 2ab2
    0x0000010: 000c b600 0ab1                         

        at scalapb.ScalaPbCodeGenerator$.registerExtensions(ScalaPbCodeGenerator.scala:12)
        at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:52)
        at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:67)
        at protocbridge.frontend.PosixPluginFrontend$$anonfun$prepare$1.apply$mcV$sp(PosixPluginFrontend.scala:25)
...

Using:

addSbtPlugin("com.lightbend.lagom" % "lagom-sbt-plugin" % "1.4.0-M1")
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.11")
libraryDependencies += "com.trueaccord.scalapb" %% "compilerplugin-shaded" % "0.6.0"

I get:

protoc-jar: protoc version: 330, detected platform: mac os x/x86_64
protoc-jar: executing: [/var/folders/xc/46s2thrs5ss3w70292zz4bf80000gn/T/protocjar7941379469101087961/bin/protoc.exe, --plugin=protoc-gen-scala=/var/folders/xc/46s2thrs5ss3w70292zz4bf80000gn/T/protocbridge1430846206093352348, --scala_out=:/Users/ignasi/git/projects/lightbend/grpc-sbt-experiments/helloscala-lagom-client/target/scala-2.11/src_managed/main, -I/Users/ignasi/git/projects/lightbend/grpc-sbt-experiments/helloscala-lagom-client/src/main/protobuf, -I/Users/ignasi/git/projects/lightbend/grpc-sbt-experiments/helloscala-lagom-client/target/protobuf_external, /Users/ignasi/git/projects/lightbend/grpc-sbt-experiments/helloscala-lagom-client/src/main/protobuf/helloworld.proto]
java.lang.AbstractMethodError: scalapb.ScalaPbCodeGenerator$.registerExtensions(Lcom/google/protobuf/ExtensionRegistry;)V
        at protocbridge.frontend.PluginFrontend$.runWithBytes(PluginFrontend.scala:52)
        at protocbridge.frontend.PluginFrontend$.runWithInputStream(PluginFrontend.scala:67)
        at protocbridge.frontend.PosixPluginFrontend$$anonfun$prepare$1.apply$mcV$sp(PosixPluginFrontend.scala:25)
        at protocbridge.frontend.PosixPluginFrontend$$anonfun$prepare$1.apply(PosixPluginFrontend.scala:23)
        at protocbridge.frontend.PosixPluginFrontend$$anonfun$prepare$1.apply(PosixPluginFrontend.scala:23)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
[info] Resolving jline#jline;2.12.1 ...
[info] Done updating.

I stand corrected. The error differs.

Using:

libraryDependencies += "com.trueaccord.scalapb" %% "compilerplugin-shaded" % "0.6.0"
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.11" exclude ("com.trueaccord.scalapb", "protoc-bridge_2.10"))
addSbtPlugin("com.lightbend.lagom" % "lagom-sbt-plugin" % "1.4.0-SNAPSHOT")

The compilation succeeds.

Using:

addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.11" exclude ("com.trueaccord.scalapb", "protoc-bridge_2.10"))
libraryDependencies += "com.trueaccord.scalapb" %% "compilerplugin-shaded" % "0.6.0"
addSbtPlugin("com.lightbend.lagom" % "lagom-sbt-plugin" % "1.4.0-SNAPSHOT")

The compilation suceeds. You were right! I guess I failed to test this exact combination. thanks a lot for your help!

ignasi35 on 6 Jul 2017

Just a note for those on Play: the scalajs dependency was introduced by the sbt-twirl plugin (used to compile twirl templates). The sbt-scalajs dependency was originally added so the SbtTwirl plugin can add the scalajs version of the dependency for scalajs projects using %%%. In twirl 1.3.4 we've implemented a different trick to detect scalajs projects without requiring the scalajs plugin dependency. The upgrade to 1.3.4 will be performed automatically in Play 2.6.3.

For scalajs 1.0 we will move to sbt-crossproject, which provides a more robust way of solving this problem while minimizing dependencies.