Google-cloud-java: Storage.signUrl() fails with default credentials in java8 standard runtime

[phimar]

Invocation:

// initialization
storage = StorageOptions.getDefaultInstance().getService();

// later
storage.signUrl(info, 5L, TimeUnit.MINUTES,
                Storage.SignUrlOption.httpMethod(HttpMethod.valueOf(method)),
                Storage.SignUrlOption.withContentType()
        );

Stacktrace:

Caused by: java.lang.IllegalStateException: Signing key was not provided and could not be derived
    at com.google.common.base.Preconditions.checkState(Preconditions.java:444)
    at com.google.cloud.storage.StorageImpl.signUrl(StorageImpl.java:508)

I tracked the problem down to the com.google.auth.oauth2.GoogleCredentials.getDefaultCredentialsUnsynchronized()(https://github.com/google/google-auth-library-java/blob/51a5445b33d10f252cadfdcca82dd9e68512e483/oauth2_http/java/com/google/auth/oauth2/DefaultCredentialsProvider.java#L182) where it skips over tryGetAppEngineCredential() to return an instance of com.google.auth.oauth2.AppengineCredentials which is one of the implementations of ServiceAccountSigner required by the signUrl call with default credentials.

This may also affect other services assuming an instance of com.google.auth.oauth2.AppengineCredentials.

Is there any specific reason why to check for java7 only?

Background: We moved to the java8 runtime on GAE and upgraded our api clients to the google-cloud-java api clients version 1.10.0.

[michaelbausor]

@neozwu Do you have context around this? It looks like the relevant auth library code was last updated in these PRs:

• https://github.com/google/google-auth-library-java/commit/2ac95c18185294f74d5d38098e15a04be09c755b
• https://github.com/google/google-auth-library-java/commit/dd5fdfdc11774f9df7390efabba08edb6aaa467f

[neozwu]

@phimar For GAE java 8 standard, metadata server is used to retreive credentials. The current implementation for AppengineCredentials uses appengine APIs, which only work for java 7. It seems that ComputeEngineCredentials didn't implement ServiceAccountSigner. @lesv @ludoch do you have any context regarding this?

/cc @lesv @ludoch

[lesv]

No, but its a very important thing to fix.

[phimar]

Thank you for your comments and the linkage to the issue on google/google-auth-library-java.

[stickfigure]

I can successfully sign urls with the default storage options (on Java8 GAE Standard) using google-cloud 0.20.3-alpha. When I upgrade to 0.32.0-alpha I see this error.

I've reverted to the old version.

[ValdasK]

Same issue is present in flexible runtime too?

[lesv]

@vchudnov-g FYI
@jabubake FYI
@garrettjonesgoogle FYI

[jabubake]

@frankyn FYI

[yihanzhen]

It seems #2504 can also be solved by this.

[nbali]

should be solved by https://github.com/google/google-auth-library-java/pull/150

[saturnism]

doesn't look like it's resolved.

[jabubake]

@jadekler @garrettjonesgoogle @ludoch : has anybody diagnosed what the issue is ? It seems like ComputeEngineCredentials does implement ServiceAccountSigner now ? google/google-auth-library-java#150

[jadekler]

cc @chingor13, who dealt with the linked CL

[nbali]

nothing magical here, the version hasn't been bumped to the new version of the related library

current master:
https://github.com/GoogleCloudPlatform/google-cloud-java/blob/ab575d0eaf987cbc04f715310b4f23277bd84b9c/google-cloud-clients/pom.xml#L160

[nbali]

(the linked PR has been released in https://github.com/google/google-auth-library-java/releases/tag/v0.10.0)

[garrettjonesgoogle]

@pongad Let's bump auth to 0.10.0 for the next release

[montss]

Yes this issue present in flexible runtime too, and locally when running through the appengine maven plugin.

com.google.cloud.tools:appengine-maven-plugin

[yihanzhen]

google.auth.version has been bumped. Please update the library to the latest version (1.38.0) and see if the problem is resolved.

[NicolaSpreafico]

I had the same problem (using 1.35.0):
Caused by: java.lang.IllegalStateException: Signing key was not provided and could not be derived

Using AppEngine standard, I solved like this:

        static final AppIdentityService identityService = AppIdentityServiceFactory.getAppIdentityService();

        SignUrlOption signWith = SignUrlOption.signWith(new ServiceAccountSigner() {
            @Override
            public byte[] sign(byte[] toSign) {
                return identityService.signForApp(toSign).getSignature();
            }

            @Override
            public String getAccount() {
                return identityService.getServiceAccountName();
            }
        });

[noahdietz]

It would seem that this method documentation no longer applies to GCE, right?
https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-clients/google-cloud-storage/src/main/java/com/google/cloud/storage/Storage.java#L2045-L2047

Based on the above PRs and commentary, when running on GCE with google-cloud-storage v1.38.0, the application default credentials (ADC) should be sufficient to sign a blob, yes?

Wondering if anything else has to be done as we are still having trouble (different error message, getting a little farther in the code as far as execution) and not sure why things fail when running on GCE using ADC vs. when we provide the key file directly via GOOGLE_APPLICATION_CREDENTIALS.

thanks y'all for the support.

[chingor13]

The ComputeEngineCredentials signing method requires 2 things:

1. The Identity and Access Management (IAM) API must be enabled for the project
2. The Compute Engine service account (the default App Engine service account for GAE Standard Java 8) needs the iam.serviceAccounts.signBlob permission (available to the "Service Account Token Creator" role).

@NicolaSpreafico try updating the com.google.cloud:google-cloud-storage dependency to 1.38.0 (or com.google.auth:google-auth-library-oauth2-http to 0.10.0) and ensuring the default service account has the necessary IAM access.

@noahdietz ADC prefers the key file provided by GOOGLE_APPLICATION_CREDENTIALS over the Compute Engine credentials and uses 2 different strategies for signing URLs. If you're seeing a SigningException you can inspect the underlying cause by catching the SigningException and looking at the Throwable from getCause(). It's likely that you need to set up the IAM config above.

Created https://github.com/google/google-auth-library-java/issues/175 for documentation and will open a new issue for docs here as well.

[noahdietz]

@chingor13 fixed! we had set all of the scopes and IAM policies we thought were necessary (devstorage.full_control & bucket objectAdmin). Didn't work. Turns out we needed to use the cloud-platform scope on the instance instead. At least that was the only thing that changed and it started working.

Thanks for the help.

[JustinBeckwith]

Greetings folks! It looks like this issue has been resolved. If you're still running into problems... let us know!

[khoek]

I needed to grant iam.serviceAccounts.signBlob in order for the default service account to sign URLs while running in App Engine, but running locally with the same credentials didn't require this---is this a bug?