Google-cloud-java: FirestoreOptions not loading projectid from credentials correctly

[daltonj]

I recently migrated from an older version (0.45) where I configured firestore using the FirebaseOptions, e.g.

GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream(_firebaseCredentialsPath));
FirebaseOptions options = new FirebaseOptions.Builder()
            .setCredentials(credentials)
            .build();
 FirebaseApp.initializeApp(options);
 _database = FirestoreClient.getFirestore();

This works. I migrated to the new way of doing things with FirestoreOptions (0.56.0). I got warnings to change the configuration. So the resulting code is now:

GoogleCredentials credentials = GoogleCredentials.fromStream(new FileInputStream(_firebaseCredentialsPath));
FirestoreOptions fireStoreOptions =
            FirestoreOptions.newBuilder().setTimestampsInSnapshotsEnabled(true)
                    .setCredentials(credentials)
                    .build();
_database = fireStoreOptions.getService();

However, this code fails. It requires that the projectId field be set using .setProjectId("..."). In my case it defaulted to another project in cloud that did not have firestore enabled. Perhaps the default project in cloud? It's unclear. This seems like a bug. The project id is not available from the credentials API, even though it is there. Why is the projectid being ignored / not set from credentials correctly?

[yihanzhen]

This should have been solved by #3472 and should have been included in 0.56.0. Can you double check the version you are using?

cc/ @schmidt-sebastian

[marshallpierce]

I can confirm this, and that https://github.com/GoogleCloudPlatform/google-cloud-java/issues/3458 still is a problem. I'm using 0.56.0-beta.

For context: val credentials = ServiceAccountCredentials.fromStream(stream) loads credentials from a service account json file that is an owner in a project set aside for running tests.

• setCredentialsProvider(FixedCredentialsProvider.create(credentials)), no setProjectId:

com.google.api.gax.rpc.FailedPreconditionException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: The Cloud Firestore API is not enabled for the project <the default project for my local gcloud config, which is not the project id in the credentials>

In other words, it's looking in the wrong place.

• setCredentialsProvider with a FixedCredentialsProvider.create(credentials), with setProjectId: everything works.

• setCredentials(credentials), no setProjectId:

com.google.api.gax.rpc.FailedPreconditionException: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: The Cloud Firestore API is not enabled for the project <the default project for my local gcloud config, which is not the project id in the credentials>

• setCredentials(credentials), with setProjectId:

com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Missing or insufficient permissions.

Which, I'm guessing, is it trying to use the default credentials on the test project's Firestore, which of course won't work. In other words, https://github.com/GoogleCloudPlatform/google-cloud-java/issues/3458. (Have I mentioned how much I dislike the concept of silent default credentials?)

For comparison, with 0.51.0-beta, setCredentials(credentials), with setProjectId works fine.

[daltonj]

Thanks for confirming.

+1 for (evil) default silent credentials

[schmidt-sebastian]

I started looking into this, but I am not able to reproduce this exact behavior. In my tests:

• setCredentials works as is and uses the Project ID from the provided credentials
• setCredentialProvider doesn't work and fails with the error as shown above.

Note the issue in setCredential was addressed recently in https://github.com/GoogleCloudPlatform/google-cloud-java/pull/3413. Can you confirm that both your version of Firestore and of google-cloud-core is up to date (you should be at 1.38.0)?

[marshallpierce]

I just replicated my results again to be sure. You can try it yourself: https://bitbucket.org/marshallpierce/ktor-session-gcp-firestore/src. Fiddle with the withFirestore helper in the test class to try the different scenarios.

Here's the dependency graph for the google bits:

+--- com.google.cloud:google-cloud-firestore:0.56.0-beta
|    +--- com.google.cloud:google-cloud-core:1.38.0
|    |    +--- com.google.guava:guava:20.0
|    |    +--- joda-time:joda-time:2.9.2
|    |    +--- com.google.http-client:google-http-client:1.23.0
|    |    |    +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
|    |    |    \--- org.apache.httpcomponents:httpclient:4.0.1 -> 4.5.3
|    |    |         +--- org.apache.httpcomponents:httpcore:4.4.6
|    |    |         +--- commons-logging:commons-logging:1.2
|    |    |         \--- commons-codec:commons-codec:1.9
|    |    +--- com.google.code.findbugs:jsr305:3.0.2
|    |    +--- com.google.api:api-common:1.6.0
|    |    |    +--- com.google.code.findbugs:jsr305:3.0.0 -> 3.0.2
|    |    |    \--- com.google.guava:guava:19.0 -> 20.0
|    |    +--- com.google.api:gax:1.29.0
|    |    |    +--- com.google.code.findbugs:jsr305:3.0.2
|    |    |    +--- com.google.guava:guava:20.0
|    |    |    +--- org.threeten:threetenbp:1.3.3
|    |    |    +--- com.google.auth:google-auth-library-oauth2-http:0.9.1
|    |    |    |    +--- com.google.auth:google-auth-library-credentials:0.9.1 -> 0.10.0
|    |    |    |    +--- com.google.http-client:google-http-client:1.19.0 -> 1.23.0 (*)
|    |    |    |    +--- com.google.http-client:google-http-client-jackson2:1.19.0
|    |    |    |    |    +--- com.google.http-client:google-http-client:1.19.0 -> 1.23.0 (*)
|    |    |    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.1.3
|    |    |    |    \--- com.google.guava:guava:19.0 -> 20.0
|    |    |    \--- com.google.api:api-common:1.6.0 (*)
|    |    +--- com.google.protobuf:protobuf-java-util:3.6.0
|    |    |    +--- com.google.protobuf:protobuf-java:3.6.0
|    |    |    +--- com.google.guava:guava:19.0 -> 20.0
|    |    |    \--- com.google.code.gson:gson:2.7
|    |    +--- com.google.api.grpc:proto-google-common-protos:1.12.0
|    |    |    \--- com.google.protobuf:protobuf-java:3.5.1 -> 3.6.0
|    |    \--- com.google.api.grpc:proto-google-iam-v1:0.12.0
|    |         +--- com.google.protobuf:protobuf-java:3.5.1 -> 3.6.0
|    |         +--- com.google.api:api-common:1.5.0 -> 1.6.0 (*)
|    |         \--- com.google.api.grpc:proto-google-common-protos:1.11.0 -> 1.12.0 (*)
|    +--- com.google.cloud:google-cloud-core-grpc:1.38.0
|    |    +--- com.google.auth:google-auth-library-credentials:0.10.0
|    |    +--- com.google.cloud:google-cloud-core:1.38.0 (*)
|    |    +--- com.google.guava:guava:20.0
|    |    +--- com.google.protobuf:protobuf-java:3.6.0
|    |    +--- com.google.protobuf:protobuf-java-util:3.6.0 (*)
|    |    +--- io.grpc:grpc-protobuf:1.13.1
|    |    |    +--- io.grpc:grpc-core:1.13.1
|    |    |    |    +--- io.grpc:grpc-context:1.13.1
|    |    |    |    +--- com.google.code.gson:gson:2.7
|    |    |    |    +--- com.google.guava:guava:20.0
|    |    |    |    +--- com.google.errorprone:error_prone_annotations:2.1.2
|    |    |    |    +--- com.google.code.findbugs:jsr305:3.0.0 -> 3.0.2
|    |    |    |    +--- io.opencensus:opencensus-api:0.12.3 -> 0.15.0
|    |    |    |    |    +--- io.grpc:grpc-context:1.12.0 -> 1.13.1
|    |    |    |    |    \--- com.google.guava:guava:20.0
|    |    |    |    \--- io.opencensus:opencensus-contrib-grpc-metrics:0.12.3
|    |    |    |         \--- io.opencensus:opencensus-api:0.12.3 -> 0.15.0 (*)
|    |    |    +--- com.google.protobuf:protobuf-java:3.5.1 -> 3.6.0
|    |    |    +--- com.google.guava:guava:20.0
|    |    |    +--- com.google.api.grpc:proto-google-common-protos:1.0.0 -> 1.12.0 (*)
|    |    |    \--- io.grpc:grpc-protobuf-lite:1.13.1
|    |    |         +--- io.grpc:grpc-core:1.13.1 (*)
|    |    |         \--- com.google.guava:guava:20.0
|    |    +--- io.grpc:grpc-context:1.13.1
|    |    +--- io.grpc:grpc-netty-shaded:1.13.1
|    |    |    \--- io.grpc:grpc-core:[1.13.1] -> 1.13.1 (*)
|    |    +--- io.grpc:grpc-stub:1.13.1
|    |    |    \--- io.grpc:grpc-core:1.13.1 (*)
|    |    +--- io.grpc:grpc-auth:1.13.1
|    |    |    +--- io.grpc:grpc-core:[1.13.1] -> 1.13.1 (*)
|    |    |    \--- com.google.auth:google-auth-library-credentials:0.9.0 -> 0.10.0
|    |    \--- com.google.api:gax-grpc:1.29.0
|    |         +--- com.google.code.findbugs:jsr305:3.0.2
|    |         +--- com.google.api:gax:1.29.0 (*)
|    |         +--- io.grpc:grpc-stub:1.10.1 -> 1.13.1 (*)
|    |         +--- io.grpc:grpc-auth:1.10.1 -> 1.13.1 (*)
|    |         +--- io.grpc:grpc-protobuf:1.10.1 -> 1.13.1 (*)
|    |         +--- com.google.guava:guava:20.0
|    |         +--- org.threeten:threetenbp:1.3.3
|    |         +--- com.google.auth:google-auth-library-oauth2-http:0.9.1 (*)
|    |         +--- com.google.auth:google-auth-library-credentials:0.9.1 -> 0.10.0
|    |         +--- com.google.api.grpc:proto-google-common-protos:1.0.0 -> 1.12.0 (*)
|    |         \--- com.google.api:api-common:1.6.0 (*)
|    +--- com.google.api:gax-grpc:1.29.0 (*)
|    +--- com.google.api:api-common:1.6.0 (*)
|    +--- com.google.api.grpc:proto-google-cloud-firestore-v1beta1:0.21.0
|    |    +--- com.google.protobuf:protobuf-java:3.6.0
|    |    +--- com.google.api:api-common:1.6.0 (*)
|    |    \--- com.google.api.grpc:proto-google-common-protos:1.12.0 (*)
|    +--- com.google.auto.value:auto-value:1.4
|    +--- io.grpc:grpc-netty-shaded:1.13.1 (*)
|    +--- io.grpc:grpc-stub:1.13.1 (*)
|    +--- io.grpc:grpc-auth:1.13.1 (*)
|    +--- io.opencensus:opencensus-api:0.15.0 (*)
|    \--- io.opencensus:opencensus-contrib-grpc-util:0.15.0
|         +--- io.opencensus:opencensus-api:0.15.0 (*)
|         \--- io.grpc:grpc-core:1.12.0 -> 1.13.1 (*)

[schmidt-sebastian]

@marshallpierce We have released the PR that should address this issue yesterday as part of 0.57.0. Do you mind upgrading and reporting back?

[marshallpierce]

With 0.57.0-beta:

• setCredentialsProvider, no setProjectId: same as before (broken)
• setCredentialsProvider, with setProjectId: same as before (works)
• setCredentials, no setProjectId: same as before (broken)
• setCredentials, with setProjectId: now works same as before (broken)

One down, I guess...

Edit: in my first setCredentials with setProjectId, I hadn't commented out setCredentialsProvider, so it worked. Without setCredentialsProvider, it's still broken. In other words, 0.57-beta is behaving the same as 56.

[charlesliqlogic]

@daltonj Does your credential file has {"type": "service_account",
"project_id": "project_id,} ?

[sduskis]

If I read this issue correctly, this seems like "working as intended". The user / service account used by the application needs to have access to the project resources being accessed. In this case, it sounds like the permissions of the user being used didn't align properly with the code.

There is an effort to move away from the implicit permissions, but that's going to take some time. I don't see anything actionable on this issue, at this point, so I'm closing it. Please feel free to open a different issue to address documentation related issues around credentials.