Google-cloud-java: google-cloud-speech uncaught RejectedExecutionException in vendored OkHttp

[noelweichbrodt]

Intermittently, a StreamingRecognizeRequest will throw an uncaught exception:

java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@39a394 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@a520be7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 125]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2085)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:848)
    at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
    at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
    at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:654)
    at io.grpc.internal.SerializingExecutor.schedule(SerializingExecutor.java:93)
    at io.grpc.internal.SerializingExecutor.execute(SerializingExecutor.java:86)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closed(ClientCallImpl.java:588)
    at io.grpc.internal.ForwardingClientStreamListener.closed(ForwardingClientStreamListener.java:39)
    at io.grpc.internal.InternalSubchannel$CallTracingTransport$1$1.closed(InternalSubchannel.java:716)
    at io.grpc.internal.AbstractClientStream$TransportState.closeListener(AbstractClientStream.java:452)
    at io.grpc.internal.AbstractClientStream$TransportState.access$400(AbstractClientStream.java:212)
    at io.grpc.internal.AbstractClientStream$TransportState$1.run(AbstractClientStream.java:435)
    at io.grpc.internal.AbstractClientStream$TransportState.deframerClosed(AbstractClientStream.java:273)
    at io.grpc.internal.Http2ClientStreamTransportState.deframerClosed(Http2ClientStreamTransportState.java:31)
    at io.grpc.okhttp.OkHttpClientStream$TransportState.deframerClosed(OkHttpClientStream.java:282)
    at io.grpc.internal.MessageDeframer.close(MessageDeframer.java:229)
    at io.grpc.internal.MessageDeframer.closeWhenComplete(MessageDeframer.java:191)
    at io.grpc.internal.AbstractStream$TransportState.closeDeframer(AbstractStream.java:183)
    at io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:438)
    at io.grpc.okhttp.OkHttpClientTransport.startGoAway(OkHttpClientTransport.java:765)
    at io.grpc.okhttp.OkHttpClientTransport.access$1700(OkHttpClientTransport.java:102)
    at io.grpc.okhttp.OkHttpClientTransport$ClientFrameHandler.run(OkHttpClientTransport.java:980)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:764)

This appears to be a bug that is fixed in OkHttp 3.12: https://github.com/square/okhttp/pull/4365
and there is a PR to upgrade the vendored framework: https://github.com/grpc/grpc-java/pull/5248

[sduskis]

gRPC will eventually have a release that will fix this problem. We'll upgrade to that version, once it's out.

[FranRiadigos]

Hi, Is this fix available?

[sduskis]

We are at grpc 1.19.0, and the latest is 1.20.0. However, it lookes like upgrading won't help, since https://github.com/grpc/grpc-java/pull/5248 has not been merged yet.

[FranRiadigos]

I tried to downgrade it back to 1.17.1 as well as the lib to a previous version (0.80.0) but no luck.

My workaround was to apply an Executor directly to the client.

SpeechClient.create(SpeechSettings.newBuilder()
.setExecutorProvider(FixedExecutorProvider.create(Executors.newScheduledThreadPool(
         Math.max(2, Math.min(Runtime.getRuntime().availableProcessors() - 1, 4))
)));

Too much boilerplate.

But now I'm facing some inconsistencies with BidiStreaming which seems not to close properly the stream request and keeps emitting some data both if I shutdown the SpeechClient or if I close it manually with requestClientStream.closeSend().

Getting OUT_OF_RANGE randomly ...

But well, it seems to be a hard job porting this to java from GRPC.

[sduskis]

There doesn't seem to be much we can do in this repository. We'll upgrade grpc once it makes a new release with the fix.

[ejona86]

This is unrelated to okhttp.

You should not shut down executors provided to grpc before the grpc channel before it becomes terminated. If you do, bad things will happen. https://github.com/grpc/grpc-java/issues/636 discusses some of how grpc may handle that better.

If you only shut down executors after the channels are terminated, then it could be due to a race where the channel reports terminated prematurely: https://github.com/grpc/grpc-java/issues/6283 . The other possibility is you shut down executors while still performing RPCs: https://github.com/grpc/grpc-java/issues/1981

Only one of those is a clear bug in gRPC. The others are "make the best of the bad situation."

With this backtrace, it appears the executor is being shut down before the transport is terminated. That's most likely a bug outside of grpc.