Google-cloud-java: PubSub: grpc Uncaught exception in the SynchronizationContext. Panic! - Could not find LoadBalancer pick_first

[sceee]

1. Specify the API: PubSub
2. OS type and version: Docker image gradle:5.2-jdk11-slim (also happens with jdk8 and other images)
3. Java version: jdk11 and jdk8
4. google-cloud-java version(s): 1.62.0 and above (tested with 1.65.0, too)

Steps to reproduce

1. Use google-cloud-pubsub library with version 1.62.0 or above as dependency
2. Create a subscription in own code (see code example below)

Code example

ProjectSubscriptionName subscriptionName = ProjectSubscriptionName.of(projectId, subscriptionId);
try (SubscriptionAdminClient subscriptionAdminClient = SubscriptionAdminClient.create()) {
    subscriptionAdminClient.getSubscription(subscriptionName); // Exception is thrown in this line
}
...

Stack trace

Mar 19, 2019 6:20:52 PM io.grpc.internal.ManagedChannelImpl$1 uncaughtException
SEVERE: [io.grpc.internal.ManagedChannelImpl-1] Uncaught exception in the SynchronizationContext. Panic!
java.lang.IllegalStateException: Could not find LoadBalancer pick_first. The build probably threw away META-INF/services/io.grpc.LoadBalancerProvider
        at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.<init>(AutoConfiguredLoadBalancerFactory.java:74)
        at io.grpc.internal.AutoConfiguredLoadBalancerFactory.newLoadBalancer(AutoConfiguredLoadBalancerFactory.java:45)
        at io.grpc.internal.ManagedChannelImpl.exitIdleMode(ManagedChannelImpl.java:351)
        at io.grpc.internal.ManagedChannelImpl$ChannelTransportProvider$1ExitIdleModeForTransport.run(ManagedChannelImpl.java:447)
        at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:101)
        at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:130)
        at io.grpc.internal.ManagedChannelImpl$ChannelTransportProvider.get(ManagedChannelImpl.java:451)
        at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:241)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:392)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:689)
        at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32)
        at com.google.api.gax.grpc.GrpcHeaderInterceptor$1.start(GrpcHeaderInterceptor.java:95)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:308)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:280)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
        at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58)
        at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64)
        at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:86)
        at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
        at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
        at com.google.api.gax.tracing.TracedUnaryCallable.futureCall(TracedUnaryCallable.java:75)
        at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
        at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
        at com.google.cloud.pubsub.v1.SubscriptionAdminClient.getSubscription(SubscriptionAdminClient.java:467)
        at com.google.cloud.pubsub.v1.SubscriptionAdminClient.getSubscription(SubscriptionAdminClient.java:420)
        at my.package.MyClass.start(MyClass.java:37)
        at my.package.App$1.run(App.java:31)
        at java.base/java.lang.Thread.run(Thread.java:834)

External references such as API reference guides used

• None

Any additional information below

The error only occurs since google-cloud-pubsub 1.62.0 .
When using google-cloud-pubsub 1.61.0 or below, the error does not occur with the same user code.

[kolea2]

Thank you for the report!

@sduskis, @andreamlin, Pub/Sub version 1.62 included an upgrade in gax - any idea if there were changes that could have caused this?

[sduskis]

@kolea2, can you tell if there was a grpc upgrade? That might be one possible reason.

[kolea2]

Yep - looks like grpc went from 1.17.1 to 1.18.0: https://github.com/googleapis/google-cloud-java/pull/4496/

[ajaaym]

@sceee can you please share your docker file with full repro? I tried the steps you mentioned and couldn't reproduce.

[olavloite]

@sceee I can reproduce this when I build a 'thick' jar using the maven shade plugin and add the Google Cloud client libraries as dependencies in the jar. Are you doing something like that?

If so, the files io.grpc.LoadBalancerProvider and io.grpc.NameResolverProvider from grpc-core/META-INF/services are probably not included in you final jar and this exception will occur.
A work-around is to add these files to your own jar by adding them either manually or by including them in your build process. The files should be in the META-INF/services/ folder of your jar.

If you add the files to your own project, you can include them in your final jar by using a maven-shade-plugin resource transformer like this:

<transformer implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
  <resource>META-INF/services</resource>
  <file>io.grpc.LoadBalancerProvider</file>
</transformer>
<transformer implementation="org.apache.maven.plugins.shade.resource.IncludeResourceTransformer">
  <resource>META-INF/services</resource>
  <file>io.grpc.NameResolverProvider</file>
</transformer>

[sceee]

Thank you for your response guys. @olavloite indeed I am shading using gradle with com.github.jengelman.gradle.plugins:shadow:5.0.0

Is this an issue that will be fixed in the future in grpc or the google-java library or do I permanently need this workaround from now on?

And with the workaround, do you mean including these files in my own project and including in them in the final jar?

• https://github.com/grpc/grpc-java/blob/master/core/src/bazel-internal/resources/META-INF/services/io.grpc.LoadBalancerProvider
• https://github.com/grpc/grpc-java/blob/master/core/src/bazel-internal/resources/META-INF/services/io.grpc.NameResolverProvider

[olavloite]

@sceee Yes, you should include those files in the META-INF/services directory of your jar, which means that you need to add them to your own project (at least for now as a workaround).
I'll have to discuss with someone else on what could be a possible long-term solution for this. @sduskis Could you point me in a direction of someone who might be able to help with that?

[sceee]

@olavloite alright, thank you for your help!

[olavloite]

This problem is caused if you build a fat/über-jar that contains all your dependencies. In those cases, you also need to include all service definitions of the jars that you are packaging. The way to achieve this depends on your choice of build system.

For Maven you should use the ServicesResourceTransformer.
For Gradle you can use the ServiceFileTransformer.

These plugins will collect all the service definitions of the included jars and bundle these into the META-INF/services directory of your fat/über-jar.

[mikessoldier]

Hello there.
I have the same error and I've applied the discussed workaround but it doesn't seem to work.
Is there something else I should check?
I am running the puller from a daemon thread. Do you think that can be the reason?

[olavloite]

@mikessoldier Can you confirm that your fat jar really contains the directory structure META-INF/services at top level, and that the directory contains the files io.grpc.LoadBalancerProvider and io.grpc.NameResolverProvider?

[mikessoldier]

Maybe this will help. I did a search from the list of contents of the .jar to see if those files were added.
This is the result:

$ jar tf target/MyApp.jar | grep io.grpc.LoadBalancerProvider

META-INF/services/io.grpc.LoadBalancerProvider
io/grpc/LoadBalancerProvider.class

$ jar tf target/MyApp.jar | grep io.grpc.NameResolverProvider

io/grpc/NameResolverProvider$NameResolverFactory.class
io/grpc/NameResolverProvider$NameResolverPriorityAccessor.class
io/grpc/NameResolverProvider.class
META-INF/services/io.grpc.NameResolverProvider

[olavloite]

I would have expected that the following command:

$ jar tf target/MyApp.jar | grep io.grpc.NameResolverProvider

Would have yielded at least also the following result:

META-INF/services/io.grpc.NameResolverProvider

In other words: It looks like your build does include the io.grpc.LoadBalancerProvider in the services folder, but that the io.grpc.NameResolverProvider is not included in the services folder.

Could you copy the stack trace of the error message you are getting? I would expect your stack trace to be similar but slightly different from the one at the top of this thread.

[olavloite]

@mikessoldier I noticed that you just added META-INF/services/io.grpc.NameResolverProvider to the results of the second search, so that shouldn't be the problem then. Could you copy the stack trace that you get?

[mikessoldier]

Yes, I am sorry, I did a wrong copy of the terminal output, but the file is there.
I hoped I edited the comment on time, but, It seems I was late.
the output is this:

$ jar tf target/MyApp.jar | grep io.grpc.LoadBalancerProvider

META-INF/services/io.grpc.LoadBalancerProvider
io/grpc/LoadBalancerProvider.class

$ jar tf target/MyApp.jar | grep io.grpc.NameResolverProvider

io/grpc/NameResolverProvider$NameResolverFactory.class
io/grpc/NameResolverProvider$NameResolverPriorityAccessor.class
io/grpc/NameResolverProvider.class
META-INF/services/io.grpc.NameResolverProvider

And about the error I am getting, it should be this:

{
  "timeStamp":"2019-03-28T09:48:21.613850Z",
  "threadID":24,
  "logName":"io.grpc.internal.ManagedChannelImpl",
  "sourceMethodName":"uncaughtException",
  "logLevel":"SEVERE",
  "message":"[io.grpc.internal.ManagedChannelImpl-1] Uncaught exception in the SynchronizationContext. Panic!"
}

I cannot give you a stacktrace right now, sorry.

[mikessoldier]

I made some changes to narrow the source of the error...
after doing this:

try {
  eventProcessorExecutor.submit(() -> {
    logger.info("start MyApp !");
    Subscriber subscriber = null;

    try {
        subscriber = myService.createSubscriber(receiver);
        subscriber.startAsync();
        subscriber.awaitTerminated(5000, TimeUnit.MILLISECONDS);
        logger.info("started async subscriber !!");
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        if (subscriber != null) {
        subscriber.stopAsync().awaitTerminated();
        logger.info("stopped async subscriber !!");
        }
    }

  });
} catch(Exception e) {
  e.printStackTrace();
}

I got this:

{"timeStamp":"2019-03-28T10:45:15.842541Z","threadID":22,"logName":"com.myClassPath.MyApp","sourceMethodName":null,"logLevel":"INFO","message":"start MyApp !"}
{"timeStamp":"2019-03-28T10:45:16.349366Z","threadID":24,"logName":"io.grpc.internal.ManagedChannelImpl","sourceMethodName":"uncaughtException","logLevel":"SEVERE","message":"[io.grpc.internal.ManagedChannelImpl-1] Uncaught exception in the SynchronizationContext. Panic!"}
java.util.concurrent.TimeoutException: Timed out waiting for InnerService [RUNNING] to reach a terminal state. Current state: RUNNING
    at com.google.common.util.concurrent.AbstractService.awaitTerminated(AbstractService.java:331)
    at com.google.api.core.AbstractApiService.awaitTerminated(AbstractApiService.java:108)
    at com.myClassPath.MyApp.lambda$start$0(MyApp.java:46)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
{"timeStamp":"2019-03-28T10:45:21.335513Z","threadID":22,"logName":"com.myClassPath.MyApp","sourceMethodName":null,"logLevel":"INFO","message":"stopped async subscriber !!"}

This error seems to happen

• at subscriber.startAsync().awaitTerminated(); when subscriber.startAsync().awaitTerminated(); happens and...
• at subscriber.stopAsync().awaitTerminated(); when subscriber.startAsync();
  no exception thrown in any case.

[olavloite]

The following code:

subscriber.startAsync();
subscriber.awaitTerminated(5000, TimeUnit.MILLISECONDS);

will always end in a TimeoutException. The method subscriber.awaitTerminated(...) waits until the subscriber has been terminated (stopped), and not until it is started. I think your intention is to run this:

subscriber.startAsync();
subscriber.awaitRunning(5000, TimeUnit.MILLISECONDS);

[mikessoldier]

Using .awaitRunning() stops it right away, and throws the same error.
Just to be clear, as I said before, I changed the code to try and find the source of the problem.
But the expected behavior is this:

try {
  eventProcessorExecutor.submit(() -> {
    logger.info("start MyApp !");
    Subscriber subscriber = null;

    try {
      subscriber = myService.createSubscriber(receiver);
      subscriber.startAsync().awaitTerminated();
      logger.info("started async subscriber !!");
    } catch (Exception e) {
      e.printStackTrace();
    } finally {
      if (subscriber != null) {
        subscriber.stopAsync().awaitTerminated();
        logger.info("stopped async subscriber !!");
      }
    }
  });
} catch(Exception e) {
  e.printStackTrace();
}

In this example logger.info("started async subscriber !!"); never gets executed, and there is no exception to be printed or thrown.

[olavloite]

I think that the line subscriber.startAsync().awaitTerminated(); in the example above will wait indefinitely. That's why the logging statement will never be executed, and no exception will be thrown, so you really need to change that line into something else. I'm curious what the stacktrace or the result is if you change that line into subscriber.startAsync().awaitRunning();

[mikessoldier]

When I used subscriber.startAsync().awaitRunning();
I got the following results:

{"timeStamp":"2019-03-28T13:34:41.890145Z","threadID":22,"logName":"com.myClassPath.MyApp","sourceMethodName":null,"logLevel":"INFO","message":"start pubSubWorker verticle !"}
{"timeStamp":"2019-03-28T13:34:42.364381Z","threadID":24,"logName":"io.grpc.internal.ManagedChannelImpl","sourceMethodName":"uncaughtException","logLevel":"SEVERE","message":"[io.grpc.internal.ManagedChannelImpl-1] Uncaught exception in the SynchronizationContext. Panic!"}
{"timeStamp":"2019-03-28T13:34:42.375532Z","threadID":22,"logName":"com.myClassPath.MyApp","sourceMethodName":null,"logLevel":"INFO","message":"started async subscriber !!"}
{"timeStamp":"2019-03-28T13:34:42.377339Z","threadID":22,"logName":"com.myClassPath.MyApp","sourceMethodName":null,"logLevel":"INFO","message":"stopped async subscriber !!"}

As you said, my intention is that the subscriber will wait indefinitely, and it makes sense that the logging statement never gets executed, you are right.
It seemed to me that it should execute since the Panic! might have broken the waiting (but of course I don't really know that).

[olavloite]

It's very hard to tell what is causing the exception without the entire stacktrace. It could be the same exception as the original exception of this issue, but it could also be something else. Could you try to run the exact same code as the initial example of the issue? In other words, run this piece of code and copy-paste any stacktrace that occurs:

ProjectSubscriptionName subscriptionName = ProjectSubscriptionName.of(projectId, subscriptionId);
try (SubscriptionAdminClient subscriptionAdminClient = SubscriptionAdminClient.create()) {
    subscriptionAdminClient.getSubscription(subscriptionName); // Exception is thrown in this line
}

That should make it 100% clear whether you are actually running into the same problem as the original issue, or whether there is some other problem.

Furthermore: If I understand you correctly, what you want is to start a Subscriber that will live 'forever'. To do so, you should not use the method awaitTerminated(). If all your application is doing is starting this Subscriber and you want this Subscriber to keep on listening, you should set a ExecutorProvider that does not create daemon threads. This will prevent your application from closing when the main() method has executed.

    ExecutorProvider provider =
        InstantiatingExecutorProvider.newBuilder()
            .setExecutorThreadCount(5)
            .setThreadFactory(new ThreadFactoryBuilder().setDaemon(false).build())
            .build();
    Subscriber subscriber = 
        Subscriber.newBuilder("some-subscription", receiver)
            .setExecutorProvider(provider)
            .build();

And you should not create your own eventProcessorExecutor and submit this code to that executor. The Subscriber class will start its own threads handling all incoming messages.

[mikessoldier]

Sorry for the inconvenience. I should have tried that before.
After running the same code, I got this stacktrace:

{"timeStamp":"2019-03-28T14:48:59.148510Z","threadID":21,"logName":"io.grpc.internal.ManagedChannelImpl","sourceMethodName":"uncaughtException","logLevel":"SEVERE","message":"[io.grpc.internal.ManagedChannelImpl-1] Uncaught exception in the SynchronizationContext. Panic!"}
com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:67)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1349)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1025)
    at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:670)
    at com.google.common.util.concurrent.ForwardingListenableFuture.addListener(ForwardingListenableFuture.java:45)
    at com.google.api.core.ApiFutureToListenableFuture.addListener(ApiFutureToListenableFuture.java:52)
    at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1330)
    at com.google.api.core.ApiFutures.addCallback(ApiFutures.java:63)
    at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:67)
    at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:86)
    at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
    at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
    at com.google.api.gax.tracing.TracedUnaryCallable.futureCall(TracedUnaryCallable.java:75)
    at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
    at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
    at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.getSubscription(SubscriptionAdminClient.java:467)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.getSubscription(SubscriptionAdminClient.java:420)
    at com.myClassPath.MyApp.start(MyApp.java:46)
    Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
        ... 12 more
Caused by: io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!
    at io.grpc.Status.asRuntimeException(Status.java:530)
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:482)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:694)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more
Caused by: java.lang.IllegalStateException: Could not find LoadBalancer pick_first. The build probably threw away META-INF/services/io.grpc.LoadBalancerProvider
    at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.<init>(AutoConfiguredLoadBalancerFactory.java:74)
    at io.grpc.internal.AutoConfiguredLoadBalancerFactory.newLoadBalancer(AutoConfiguredLoadBalancerFactory.java:45)
    at io.grpc.internal.ManagedChannelImpl.exitIdleMode(ManagedChannelImpl.java:351)
    at io.grpc.internal.ManagedChannelImpl$ChannelTransportProvider$1ExitIdleModeForTransport.run(ManagedChannelImpl.java:447)
    at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:101)
    at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:130)
    at io.grpc.internal.ManagedChannelImpl$ChannelTransportProvider.get(ManagedChannelImpl.java:451)
    at io.grpc.internal.ClientCallImpl.start(ClientCallImpl.java:241)
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1.start(CensusTracingModule.java:392)
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1.start(CensusStatsModule.java:689)
    at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32)
    at com.google.api.gax.grpc.GrpcHeaderInterceptor$1.start(GrpcHeaderInterceptor.java:95)
    at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:308)
    at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:280)
    at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:189)
    at com.google.api.gax.grpc.GrpcDirectCallable.futureCall(GrpcDirectCallable.java:58)
    at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:64)
    at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:86)
    at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:63)
    at com.google.api.gax.rpc.RetryingCallable.futureCall(RetryingCallable.java:41)
    at com.google.api.gax.tracing.TracedUnaryCallable.futureCall(TracedUnaryCallable.java:75)
    at com.google.api.gax.rpc.UnaryCallable$1.futureCall(UnaryCallable.java:126)
    at com.google.api.gax.rpc.UnaryCallable.futureCall(UnaryCallable.java:87)
    at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.getSubscription(SubscriptionAdminClient.java:467)
    at com.google.cloud.pubsub.v1.SubscriptionAdminClient.getSubscription(SubscriptionAdminClient.java:420)
    at com.myClassPath.MyApp.start(MyApp.java:46)
    ... 1 more

Thank you for your patience.
I'll try using the ExecutorProvider, and let you know if there is any difference.

[olavloite]

No problem, and thanks for running the original example :+1:
That confirms that the problem is the same as the original issue. It does however puzzle me that it still happens if the META-INF/services/ directory of MyApp.jar contains these two files....
If it still doesn't work, could you please list the contents of the two files in your MyApp.jar?

META-INF/services/io.grpc.LoadBalancerProvider
META-INF/services/io.grpc.NameResolverProvider

These should contain some text referencing the classes that should be used. Something like this for LoadBalancerProvider:

io.grpc.grpclb.GrpclbLoadBalancerProvider
io.grpc.internal.PickFirstLoadBalancerProvider
io.grpc.util.SecretRoundRobinLoadBalancerProvider$Provider

[mikessoldier]

It's solved.
As you suggested, I tried to confirm the content of the files (io.grpc.LoadBalancerProvider, io.grpc.NameResolverProvider)
e.g.:

$ unzip -q -c target/myJar.jar META-INF/services/io.grpc.LoadBalancerProvider
io.grpc.grpclb.GrpclbLoadBalancerProvider

I didn't place the files correctly before, and therefore they were not being included correctly... making the fix useless.
To be clear, I have moved the files to the next location:

<project-root>/src/main/resources/META-INF/services/io.grpc.LoadBalancerProvider
<project-root>/src/main/resources/META-INF/services/io.grpc.NameResolverProvider

The content of the io.grpc.LoadBalancerProvider file is

io.grpc.internal.PickFirstLoadBalancerProvider

And the content of the io.grpc.NameResolverProvider file is

io.grpc.internal.DnsNameResolverProvider

[cakism]

So happy to have found this thread as I've had exactly the same issue and tried a plethora of increasingly weird things in what can only be described as superstition based coding, but adding the files in the

<project-root>/src/main/resources/META-INF/services/io.grpc.LoadBalancerProvider
<project-root>/src/main/resources/META-INF/services/io.grpc.NameResolverProvider

like above finally did the trick! 🙌

Running a Kotlin project with the Cloud PubSub Java Client Lib in Docker for Mac.

[wyaeld]

@olavloite

When using kotlin gradle build, I've found that a little more tuning is desirable. The normal workaround of turning on the com.github.johnrengelman.shadow plugin, which has similar function to the maven shade plugin has an issue.

If you do the default config, below:

tasks.withType<ShadowJar> {
    mergeServiceFiles ()
}

it will work, but can result in a lot of undesirable files also ending up in the output jar, I'm unsure why. I ended up with 70mb of gradle internals.

However if you do this

tasks.withType<ShadowJar> {
    mergeServiceFiles {
        setPath("META-INF/services")
        include("io.grpc.*")
    }
}

You get the solution without extra pollution.

Would it be possible to get docs updated to make it easier for people to know this? Its an awkward problem to track down the first time it happens.

[blacar]

I have found similar error using 'com.google.cloud:google-cloud-pubsub:1.83.0'.
trying to pusblish a message in a topic:

SEVERE: [Channel<42>: (pubsub.googleapis.com:443)] Uncaught exception in the SynchronizationContext. Panic!
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@2fc79023 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@50a3959b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
        at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:622)
        at io.grpc.internal.DelayedClientTransport.reprocess(DelayedClientTransport.java:297)
        at io.grpc.internal.ManagedChannelImpl.updateSubchannelPicker(ManagedChannelImpl.java:790)
        at io.grpc.internal.ManagedChannelImpl.access$5200(ManagedChannelImpl.java:102)
        at io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1UpdateBalancingState.run(ManagedChannelImpl.java:1149)
        at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:88)
        at io.grpc.internal.InternalSubchannel$TransportListener.transportReady(InternalSubchannel.java:579)
        at io.grpc.netty.shaded.io.grpc.netty.ClientTransportLifecycleManager.notifyReady(ClientTransportLifecycleManager.java:43)
        at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onSettingsRead(NettyClientHandler.java:783)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:419)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:637)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onSettingsRead(Http2InboundFrameLogger.java:93)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readSettingsFrame(DefaultHttp2FrameReader.java:542)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:263)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:390)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:254)
        at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:450)
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1476)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225)
        at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1272)
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
        at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
        at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

[olavloite]

@blacar The above issue is something else than the original issue. You seem to run into an issue where an executor that is used by your transport channel is rejecting any further tasks. If this is something reproducible, it would be better to open a new issue with a clear description on how to reproduce the issue.

[bajabob]

@olavloite & @mikessoldier & @wyaeld

First off, thank you so much for this wonderful thread. I am having the exact same issue. Using com.google.firebase:firebase-admin:6.9.0 + Shadow 5.1.0 for packaging Jars. Mike, your manual META-INF service additions remedied the problem for me. Brad, your gradle tasks for adding only the required service files did not work for me.

I am specifically following the Firestore quickstart guide for Java Admin SDK: https://firebase.google.com/docs/firestore/quickstart

[AdamSHurwitz]

@mikessoldier, I've attempted your proposed solution from March, 28, 2019 and I am still seeing the error.

I'm seeing the error above when attempting to run a _Jar_ in IntelliJ. It works on the machine it was written on, but when the project is pulled to a new machine the issue above is occurring.

Related StackOverflow post: How to fix “Could not find policy 'pick_first'” with Google TTS java client?.

It's solved.
As you suggested, I tried to confirm the content of the files (io.grpc.LoadBalancerProvider, io.grpc.NameResolverProvider)
e.g.:

$ unzip -q -c target/myJar.jar META-INF/services/io.grpc.LoadBalancerProvider
io.grpc.grpclb.GrpclbLoadBalancerProvider

I didn't place the files correctly before, and therefore they were not being included correctly... making the fix useless.
To be clear, I have moved the files to the next location:

<project-root>/src/main/resources/META-INF/services/io.grpc.LoadBalancerProvider
<project-root>/src/main/resources/META-INF/services/io.grpc.NameResolverProvider

The content of the io.grpc.LoadBalancerProvider file is

io.grpc.internal.PickFirstLoadBalancerProvider

And the content of the io.grpc.NameResolverProvider file is

io.grpc.internal.DnsNameResolverProvider

[AdamSHurwitz]

@olavloite, Where in the project should the _ServiceFileTransformer_ be placed for a Gradle based project in order to resolve the error above?

This problem is caused if you build a fat/über-jar that contains all your dependencies. In those cases, you also need to include all service definitions of the jars that you are packaging. The way to achieve this depends on your choice of build system.

For Maven you should use the ServicesResourceTransformer.
For Gradle you can use the ServiceFileTransformer.

These plugins will collect all the service definitions of the included jars and bundle these into the META-INF/services directory of your fat/über-jar.

[AdamSHurwitz]

@sceee, I've implemented the Gradle library com.github.jengelman.gradle.plugins:shadow:5.0.0, but unfortunately the error above is still persisting.

[wyaeld]

@AdamSHurwitz I know it's a long thread, try my solution here https://github.com/googleapis/google-cloud-java/issues/4700#issuecomment-513593702

[AdamSHurwitz]

Thanks @wyaeld! My issues seems to be potentially unrelated now, as I'm seeing a new error when migrating projects from an existing MacBook Pro to a newer MacBook Pro.

StackOverflow: Kotlin Jar Error: Exception in thread “Timer-0” StorageException: 401 Unauthorized

[rohith28]

I am getting the same issues after updating the PubSub emulator component. I am reading pubsub data from Dataflow job.

Current gcloud componenets versions:

Google Cloud SDK 313.0.0
beta 2020.10.02
bq 2.0.61
core 2020.10.02
gsutil 4.53
pubsub-emulator 0.1.1

Emulator working fine with these components version

Google Cloud SDK 311.0.0
alpha 2020.09.18
beta 2020.09.18
bq 2.0.60
cbt
core 2020.09.18
datalab 20190610
gsutil 4.53
kubectl 1.15.11
pubsub-emulator 2019.09.27

Error from emulator

[pubsub] SEVERE: [Channel<1>: (localhost:8085)] Uncaught exception in the SynchronizationContext. Panic!
[pubsub] java.lang.IllegalStateException: Could not find policy 'pick_first'. Make sure its implementation is either registered to LoadBalancerRegistry or included in META-INF/services/io.grpc.LoadBalancerProvider from your jar files.
[pubsub]    at io.grpc.internal.AutoConfiguredLoadBalancerFactory$AutoConfiguredLoadBalancer.<init>(AutoConfiguredLoadBalancerFactory.java:94)
[pubsub]    at io.grpc.internal.AutoConfiguredLoadBalancerFactory.newLoadBalancer(AutoConfiguredLoadBalancerFactory.java:65)
[pubsub]    at io.grpc.internal.ManagedChannelImpl.exitIdleMode(ManagedChannelImpl.java:396)
[pubsub]    at io.grpc.internal.ManagedChannelImpl$ChannelStreamProvider$1ExitIdleModeForTransport.run(ManagedChannelImpl.java:489)
[pubsub]    at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95)