Google-cloud-java: PERMISSION_DENIED for google service account

Created on 26 Mar 2018 · 18Comments · Source: googleapis/google-cloud-java

I've successfully subscribed to Pub?Sub service under my own user account. Howewer when i set GOOGLE_APPLICATION_CREDENTIALS env variable to point service account key file i got permission denied exception. This service account has Pub/Sub Admin role. Here is stack trace of exception:

SEVERE: terminated streaming with exception
com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55)
    at com.google.cloud.pubsub.v1.StreamingSubscriberConnection$1.onFailure(StreamingSubscriberConnection.java:237)
    at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675)
    at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:53)
    at com.google.cloud.pubsub.v1.StreamingSubscriberConnection$StreamingPullResponseObserver.onError(StreamingSubscriberConnection.java:173)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:419)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41)
    at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:41)
    at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:392)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:475)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:557)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:478)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:590)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to perform this action.
    at io.grpc.Status.asRuntimeException(Status.java:526)
    ... 19 more

pubsub auth question

Source

agzamovr

👍3 😕1

Most helpful comment

No, we have decided to use RabbitMQ instead.

agzamovr on 23 Apr 2018

🎉7

All 18 comments

I am also facing the same problem in production and with selective hosts. I am using the same JSON(service account credentials) on 30 different hosts and 3 of them have reported this error. This is a runtime error with a generic message. Please suggest a way to debug it.

PrashSi on 20 Apr 2018

👍2

I am also facing the same issue. Are you able to solve this?

sangamangreg on 23 Apr 2018

No, we have decided to use RabbitMQ instead.

agzamovr on 23 Apr 2018

🎉7

We have same issue, weird that some messages is received - some failed

muzuro on 10 May 2018

This sounds bad. Am I understanding correctly that this issue is intermittent? Our authentication lib has to refresh credentials every hour or so. I suspect that the bug could creep in there.

pongad on 23 May 2018

How reproducible is this on a particular host? In other words, do the hosts that exhibit this failure always fail, and the ones that succeed always succeed? Or does a single host sometimes succeed and sometimes fail?

vchudnov-g on 23 May 2018

It never succeeded for me from my mac.

-
Sangam Angre
+91 8796-804-941
https://www.facebook.com/sangam.angre https://twitter.com/sangamangre
https://www.linkedin.com/in/sangamangre/

On Thu, May 24, 2018 at 2:54 AM, Victor Chudnovsky <[email protected]

wrote:

How reproducible is this on a particular host? In other words, do the
hosts that exhibit this failure always fail, and the ones that succeed
always succeed? Or does a single host sometimes succeed and sometimes fail?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/google-cloud-java/issues/3083#issuecomment-391501231,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AS60SsmWAppNMb0dZb-uLSegMCcaNNDZks5t1dOAgaJpZM4S7H5X
.

sangamangreg on 24 May 2018

@vchudnov-g Single host sometimes succeeds and sometimes fails. We have also observed that only these 4 to 5 hosts which are ideal(haven't published a single message)for a day or more throws this exception. So maybe there is a problem in refreshing the credentials?

PrashSi on 24 May 2018

@PrashSi Am I understanding correctly that the failure is on the subscriber side and happens intermittently during streaming calls (pulling messages)?
Also are you running on any Google Cloud machines(GCE, GAE, GKE) or local machines?

yihanzhen on 31 May 2018

Also, it would be great if any of you could do the following to help debug if you have cloud-debugger available:

set a watch point in OAuth2Credentials::shouldRefresh and log anytime expiresIn < 5000L

yihanzhen on 31 May 2018

@hzyi-google Running on local machines.

PrashSi on 4 Jun 2018

I had a publisher and a few subscribers set up on my local machine and had them running for about 12 hours but could not reproduce this problem. The publisher publishes every 2 seconds. Anything different I can do to help increase the frequency of the bug?

yihanzhen on 6 Jun 2018

I have observed this issue once in the following situation:

Spring boot application
Running in Google Container Engine
Application listening to 6 subscriptions
One of the subscribers stopped with this error about 4 days and 9 hours after launching the application
Between the application starting and the error occurring 1 message was published to a topic not connected to the erroneous subscription, no other messages were published.
Dependencies from the google-cloud-bom version 0.49.0-alpha

pimotte on 4 Jul 2018

Same issue trying to use Stackdriver exporter.

io.opencensus.exporter.stats.stackdriver.StackdriverExporterWorker export WARNING: ApiException thrown when exporting TimeSeries. com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: The caller does not have permission at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:55) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:95) at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:61) at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:492) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:467) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:391) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:471) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:553) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:474) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:591) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

ejgutierrezvega on 3 Aug 2018

I was able to make it work following the instructions from:
https://github.com/census-instrumentation/opencensus-java/tree/master/exporters/stats/stackdriver#quickstart

Specifically enabling billing to my gcp account.

ejgutierrezvega on 3 Aug 2018

@hzyi-google this one looks a tad scary. Are you able to actively look at this?

JustinBeckwith on 13 Sep 2018

@JustinBeckwith I've been trying to repro this but without any luck. I've been talking with Customer Solution Engineers to see if they can get more information from customers but didn't get anything useful. Could you help escalate this?

yihanzhen on 13 Sep 2018

This issue appears to have diverged into a few scenarios: