Google-cloud-java: Failed to unpack response from 'any' field on v1beta2.ClusterControllerClient.createClusterAsync

[ex-hota911]

ClusterControllerClient.createClusterAsync throws the following exception when it successfully created a cluster.

java.util.concurrent.ExecutionException: com.google.api.gax.rpc.UnknownException: java.lang.IllegalStateException: Failed to unpack object from 'any' field. Expected com.google.cloud.dataproc.v1beta2.Cluster, found type.googleapis.com/google.cloud.dataproc.v1.Cluster
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:481)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83)
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:62)
        at com.google.api.gax.longrunning.OperationFutureImpl.get(OperationFutureImpl.java:127)
        at App.main(App.java:41)
Caused by: com.google.api.gax.rpc.UnknownException: java.lang.IllegalStateException: Failed to unpack object from 'any' field. Expected com.google.cloud.dataproc.v1beta2.Cluster, found type.googleapis.com/google.cloud.dataproc.v1.Cluster
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:117)
        at com.google.api.gax.grpc.ProtoOperationTransformers$ResponseTransformer.apply(ProtoOperationTransformers.java:69)
        at com.google.api.gax.grpc.ProtoOperationTransformers$ResponseTransformer.apply(ProtoOperationTransformers.java:46)
        at com.google.api.core.ApiFutures$GaxFunctionToGuavaFunction.apply(ApiFutures.java:204)
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:249)
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:239)
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:130)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:973)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:821)
        at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:663)
        at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:159)
        at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:134)
        at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:114)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:973)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:821)
        at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:663)
        at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.setResult(AbstractTransformFuture.java:255)
        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:177)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:973)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:821)
        at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:663)
        at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:159)
        at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.handle(CallbackChainRetryingFuture.java:134)
        at com.google.api.gax.retrying.CallbackChainRetryingFuture$AttemptCompletionListener.run(CallbackChainRetryingFuture.java:114)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:973)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:821)
        at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:663)
        at com.google.api.core.AbstractApiFuture$InternalSettableFuture.set(AbstractApiFuture.java:90)
        at com.google.api.core.AbstractApiFuture.set(AbstractApiFuture.java:73)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onSuccess(GrpcExceptionCallable.java:88)
        at com.google.api.core.ApiFutures$1.onSuccess(ApiFutures.java:73)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1374)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:399)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:973)
        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:821)
        at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:663)
        at io.grpc.stub.ClientCalls$GrpcFuture.set(ClientCalls.java:488)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:466)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Failed to unpack object from 'any' field. Expected com.google.cloud.dataproc.v1beta2.Cluster, found type.googleapis.com/google.cloud.dataproc.v1.Cluster
        at com.google.api.gax.grpc.ProtoOperationTransformers$AnyTransformer.apply(ProtoOperationTransformers.java:131)
        at com.google.api.gax.grpc.ProtoOperationTransformers$ResponseTransformer.apply(ProtoOperationTransformers.java:67)
        ... 62 more

[ex-hota911]

Code to reproduce the issue.
https://github.com/hota911/dataproc-issue-3585/tree/master

[yihanzhen]

Internal bug filed. Dataproc team is looking into it.

[DanSedov]

@hzyi-google

The problem here is that com.google.longrunning.Operation service itself is version-less and is not Dataproc specific. When the client calls Clusters.Create the response operation goes through v1beta2 path in the server. However, on subsequent polling, Operations.Get is called which no longer has any version information. There's nothing in the request to help Dataproc choose v1 or v1beta2 paths.

Do other Google products have multiple API versions that use Operations? Is there a common pattern for how to handle this?

[yihanzhen]

Hi @DanSedov ,
A search in this repo gives me this example in dialogflow as you mentioned: v2 and v2beta1. The method returns an LRO and the return type is defined in both v2 and v2beta1. Maybe you could check with dialogflow team?

[DanSedov]

@hota911 I have implemented a fix in Dataproc service for this issue which will become available as part of a release in next few weeks. It will be announced here: https://cloud.google.com/dataproc/docs/release-notes

[DanSedov]

@hota911 The fix has been released, please try again.

@hzyi-google I think this issue can be closed now.

[ajaaym]

This seems fixed.