Grpc-java: java.lang.NoSuchFieldError: NETTY_SHADED when upgrading from 1.27.2 to 1.28.0

[nathanhleung]

What version of gRPC-Java are you using?

1.29.0 -- although I tried all gRPC versions from 1.27.2 to 1.29.0 to determine the exact version of breakage and it looks like it occurs between 1.27.2 (working) and 1.28.0 (not working).

What is your environment?

• My development machine has macOS Catalina 10.15.4 with Java 11.0.6 installed
• I'm seeing the same issue in production, where we run the openjdk:11-jre-slim-buster Docker image, which contains Debian + Java 11
• We're using Kotlin 1.3.72, compiled for the JVM

Dependencies

GRPC_VERSION = "1.27.2" # Issue appears when I change this to 1.28.0

JAVA_DEPS = [
    "com.google.code.gson:gson:2.8.6",
    "com.google.protobuf:protobuf-java:3.11.4",
    "com.googlecode.protobuf-java-format:protobuf-java-format:1.4",
    "io.grpc:grpc-core:{grpcVersion}".format(grpcVersion = GRPC_VERSION),
    "io.grpc:grpc-netty-shaded:{grpcVersion}".format(grpcVersion = GRPC_VERSION),
    "io.grpc:grpc-protobuf:{grpcVersion}".format(grpcVersion = GRPC_VERSION),
    "io.grpc:grpc-services:{grpcVersion}".format(grpcVersion = GRPC_VERSION),
    "io.grpc:grpc-stub:{grpcVersion}".format(grpcVersion = GRPC_VERSION),
    "io.grpc:grpc-testing:{grpcVersion}".format(grpcVersion = GRPC_VERSION),
    "javax.annotation:javax.annotation-api:1.3.2",
    "junit:junit:4.13"
]

What did you expect to see?

First, I create a stub to access one of our gRPC services:

// All of our services require JWT authentication
val headers = Metadata().apply {
    this.put(Metadata.Key.of("Authorization", Metadata.ASCII_STRING_MARSHALLER), "Bearer $token")
}
val contentServiceStub = MetadataUtils.attachHeaders(ContentServiceGrpc.newBlockingStub(
    ManagedChannelBuilder
        .forAddress(
            System.getenv("CONTENT_SERVER_HOST"),
            System.getenv("CONTENT_SERVER_PORT").toInt()
       )
       .usePlaintext()
       .build()
), headers)

Then, I run an RPC:

val content = contentServiceBlockingStub.getContent(
    GetContentRequest
        .newBuilder()
        .addAllKeys(
            listOf(
                 "website-heading",
                 "website-subheading"
             )
         )
         .build()
).content

println(content)

I expect to see the content from the content service (in this case, website copy).

What did you see instead?

I get an error message (see "Steps to reproduce the bug")

Steps to reproduce the bug

Starting with gRPC version 1.28.0, in the terminal in which I bazel run the Kotlin executable which runs the RPC, I see this stack trace:

Apr 26, 2020 1:31:56 PM io.grpc.stub.ClientCalls$ThreadlessExecutor waitAndDrain
WARNING: Runnable threw exception
java.lang.NoSuchFieldError: NETTY_SHADED
    at io.grpc.netty.shaded.io.grpc.netty.NettyClientStream.<clinit>(NettyClientStream.java:59)
    at io.grpc.netty.shaded.io.grpc.netty.NettyClientTransport.newStream(NettyClientTransport.java:177)
    at io.grpc.internal.CallCredentialsApplyingTransportFactory$CallCredentialsApplyingTransport.newStream(CallCredentialsApplyingTransportFactory.java:117)
    at io.grpc.internal.ForwardingConnectionClientTransport.newStream(ForwardingConnectionClientTransport.java:49)
    at io.grpc.internal.InternalSubchannel$CallTracingTransport.newStream(InternalSubchannel.java:635)
    at io.grpc.internal.DelayedClientTransport$PendingStream.createRealStream(DelayedClientTransport.java:353)
    at io.grpc.internal.DelayedClientTransport$PendingStream.access$300(DelayedClientTransport.java:341)
    at io.grpc.internal.DelayedClientTransport$5.run(DelayedClientTransport.java:300)
    at io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:690)
    at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:133)
    at plus.talar.proto.ContentServiceGrpc$ContentServiceBlockingStub.getContent(ContentServiceGrpc.java:395)
    at (the file where the RPC was run)

I don't see any error logs in the shell where I'm running the grpc service.

This PR seems relevant: https://github.com/grpc/grpc-java/pull/6774

Thanks! Let me know if you need any more information.

[creamsoup]

it looks like NETTY_SHADED is added in 1.29.0. if you are using grpc-netty or grpc-netty-shaded 1.29.0, make sure grpc-api is also 1.29.0.

[nathanhleung]

Thanks! We're using grpc-netty-shaded. We've never explicitly depended on grpc-api before -- is that new?

[creamsoup]

grpc-api is introduced 1.21.0. many grpc modules are depending on the grpc-api. so, it must be resolved as transient dependency. since grpc-api is generally stable, using older version is usually okay.
hmm it is unclear how you got the 1.29.0 netty-shaded if you are using 1.28.0.

[nathanhleung]

In the meantime, we've reverted to 1.27.2 and things seem to be working OK. What do you suggest in terms of next steps?

[creamsoup]

i would check dependency tree to see why 1.29.0 netty-shaded is provided to make sure all the dependency is under control. anyways, it seems like the issue is resolved 😃

[nathanhleung]

Ok, thanks for your help!