Firebase-tools: Unable to deploy Firebase functions using service account app default credentials

[grantneale]

[REQUIRED] Environment info

firebase-tools: 6.3.1

Platform: Windows

[REQUIRED] Test case

• Create any Firebase app containing Firebase functions
• Attempt to deploy on a Google Compute instance, using a service account, via app default credentials
• Note that even if the service account is assigned the Owner role, this still fails.

[REQUIRED] Steps to reproduce

Attempt to deploy by running firebase deploy --only functions

[REQUIRED] Expected behavior

The Firebase functions are deployed successfully

[REQUIRED] Actual behavior

Deployment fails.

build@msvc-build-agent-2 MINGW64 /c/buildkite-agent/builds/msvc-1/gamurs/gamerai/cloud-functions ((bdcd8ba...))
$ firebase deploy --debug --project development --only functions
[2019-03-27T23:24:34.314Z] ----------------------------------------------------------------------
[2019-03-27T23:24:34.318Z] Command:       C:\nodejs\node.exe C:\Users\build\AppData\Roaming\npm\node_modules\firebase-tools\lib\bin\firebase.js deploy --debug --project development --only functions
[2019-03-27T23:24:34.319Z] CLI Version:   6.3.1
[2019-03-27T23:24:34.319Z] Platform:      win32
[2019-03-27T23:24:34.319Z] Node Version:  v10.15.1
[2019-03-27T23:24:34.321Z] Time:          Wed Mar 27 2019 23:24:34 GMT+0000 (Greenwich Mean Time)
[2019-03-27T23:24:34.321Z] ----------------------------------------------------------------------

[2019-03-27T23:24:34.333Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
[2019-03-27T23:24:34.333Z] > attempting to authenticate via app default credentials
[2019-03-27T23:24:35.226Z] ya29.c.ElzZBv0NxmErJ3Z4exFa26zge_0EZAT4sUfBbCBLURasfCXLwAHzbu7ma3zS9K5E4rESBAW4dpwRSpFfIf4mGHS0PVEhMi6C4jGiAibhAzJCE7qlCJQrDyFVTV0xmA
[2019-03-27T23:24:35.226Z] > retrieved access token via default credentials
[2019-03-27T23:24:35.227Z] [iam] checking project gamer-ai-development for permissions ["cloudfunctions.functions.create","cloudfunctions.functions.delete","cloudfunctions.functions.get","cloudfunctions.functions.list","cloudfunctions.functions.update","cloudfunctions.operations.get","firebase.projects.get"]
[2019-03-27T23:24:35.229Z] >>> HTTP REQUEST POST https://cloudresourcemanager.googleapis.com/v1/projects/gamer-ai-development:testIamPermissions
 permissions=[cloudfunctions.functions.create, cloudfunctions.functions.delete, cloudfunctions.functions.get, cloudfunctions.functions.list, cloudfunctions.functions.update, cloudfunctions.operations.get, firebase.projects.get]
[2019-03-27T23:24:35.347Z] <<< HTTP RESPONSE 200 content-type=application/json; charset=UTF-8, vary=X-Origin, Referer, Origin,Accept-Encoding, date=Wed, 27 Mar 2019 23:24:35 GMT, server=ESF, cache-control=private, x-xss-protection=1; mode=block, x-frame-options=SAMEORIGIN, x-content-type-options=nosniff, server-timing=gfet4t7; dur=84, accept-ranges=none, transfer-encoding=chunked

=== Deploying to 'gamer-ai-development'...

i  deploying functions
[2019-03-27T23:24:50.698Z] > [functions] package.json contents: {
  "name": "my-cloud-functions",
  "description": "Cloud Functions to be deployed to Firebase",
  "private": true,
  "devDependencies": {
    "fable-compiler": "^2.1.12",
    "fable-splitter": "^2.1.5"
  },
  "dependencies": {
    "@google-cloud/storage": "^2.4.2",
    "express": "^4.16.4",
    "firebase-admin": "~7.0.0",
    "firebase-functions": "^2.1.0"
  }
}
i  functions: ensuring necessary APIs are enabled...
[2019-03-27T23:24:50.700Z] >>> HTTP REQUEST GET https://servicemanagement.googleapis.com/v1/services/cloudfunctions.googleapis.com/projectSettings/gamer-ai-development?view=CONSUMER_VIEW

[2019-03-27T23:24:50.701Z] >>> HTTP REQUEST GET https://servicemanagement.googleapis.com/v1/services/runtimeconfig.googleapis.com/projectSettings/gamer-ai-development?view=CONSUMER_VIEW

[2019-03-27T23:24:50.736Z] <<< HTTP RESPONSE 404 vary=X-Origin, Referer, Origin,Accept-Encoding, content-type=application/json; charset=UTF-8, date=Wed, 27 Mar 2019 23:24:50 GMT, server=ESF, cache-control=private, x-xss-protection=1; mode=block, x-frame-options=SAMEORIGIN, x-content-type-options=nosniff, accept-ranges=none, transfer-encoding=chunked
[2019-03-27T23:24:50.737Z] <<< HTTP RESPONSE BODY code=404, message=Method not found., status=NOT_FOUND

Error: HTTP Error: 404, Method not found.
[2019-03-27T23:24:50.743Z] Error Context: {
  "body": {
    "error": {
      "code": 404,
      "message": "Method not found.",
      "status": "NOT_FOUND"
    }
  },
  "response": {
    "statusCode": 404,
    "body": {
      "error": {
        "code": 404,
        "message": "Method not found.",
        "status": "NOT_FOUND"
      }
    },
    "headers": {
      "vary": "X-Origin, Referer, Origin,Accept-Encoding",
      "content-type": "application/json; charset=UTF-8",
      "date": "Wed, 27 Mar 2019 23:24:50 GMT",
      "server": "ESF",
      "cache-control": "private",
      "x-xss-protection": "1; mode=block",
      "x-frame-options": "SAMEORIGIN",
      "x-content-type-options": "nosniff",
      "accept-ranges": "none",
      "transfer-encoding": "chunked"
    },
    "request": {
      "uri": {
        "protocol": "https:",
        "slashes": true,
        "auth": null,
        "host": "servicemanagement.googleapis.com",
        "port": 443,
        "hostname": "servicemanagement.googleapis.com",
        "hash": null,
        "search": "?view=CONSUMER_VIEW",
        "query": "view=CONSUMER_VIEW",
        "pathname": "/v1/services/runtimeconfig.googleapis.com/projectSettings/gamer-ai-development",
        "path": "/v1/services/runtimeconfig.googleapis.com/projectSettings/gamer-ai-development?view=CONSUMER_VIEW",
        "href": "https://servicemanagement.googleapis.com/v1/services/runtimeconfig.googleapis.com/projectSettings/gamer-ai-development?view=CONSUMER_VIEW"
      },
      "method": "GET"
    }
  }
}
[2019-03-27T23:24:50.746Z] <<< HTTP RESPONSE 404 vary=X-Origin, Referer, Origin,Accept-Encoding, content-type=application/json; charset=UTF-8, date=Wed, 27 Mar 2019 23:24:50 GMT, server=ESF, cache-control=private, x-xss-protection=1; mode=block, x-frame-options=SAMEORIGIN, x-content-type-options=nosniff, accept-ranges=none, transfer-encoding=chunked
[2019-03-27T23:24:50.746Z] <<< HTTP RESPONSE BODY code=404, message=Method not found., status=NOT_FOUND

build@msvc-build-agent-2 MINGW64 /c/buildkite-agent/builds/msvc-1/gamurs/gamerai/cloud-functions ((4b452cd...))

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[joehan]

Hey @grantneale, thanks for filing this and sorry to hear that it has been causing you issues. I just recreated the issue locally, and I'm seeing the same behavior with a fresh project and service account set to the Owner role. As you identified, the call to https://servicemanagement.googleapis.com/v1/services//projectSettings/?view=CONSUMER_VIEW is returning 404's unexpectedly when using a service account. I also tested out the API call separately from the rest of the deploy, and I see the same behavior, so this seems like a quirk with Service Manager. I don't know too much about Service Manager, so I've filed a bug with the team that works on it, to see if they can shed some more light. I'll post back here as soon as I hear back from them!

(PS - I love that avatar pic 😁)
Internal bug reference: 129704695

[crunchie84]

I am experiencing somewhat the same bug; it seems that the attempting to authenticate via app default credentials does not utilise the credentials configured by gcloud auth activate-service-account --key-file ${HOME}/gcloud-service-key.json resulting in a Bearer token with no associated OAuth scopes to query https://cloudresourcemanager.googleapis.com/v1/projects/XXXPROJECTNAMEXXX:testIamPermissions. As a GCP platinum support customer I have also created a ticket in the Enterprise Support environment to raise awareness (18944888)

[joehan]

Hey @grantneale, quick update on this. I spoke to an engineer on Service manager today, and I think I have a fix. I'm hoping to write up a PR for it in a few days - I'll keep this thread updated.
@crunchie84 Thanks for filing with support, they should be able to help you as well. In the meantime, would you mind posting the full --debug statement from whatever command is failing for you? While it sounds like your issue may be due to a different API than @grantneale 's, I'd like to check if its due to a similar reason.

[crunchie84]

@joehan I have created a gist of the logs that I am receiving - https://gist.github.com/crunchie84/1c39df75591f2ff7f9b72171cacbcd2d
in here you will find two logs:

• run without gcloud auth activate-service-account invoked to login using service account -> firebase deploy generates a bearer token (replaced in logfile with XXXBEARER_ACCES_TOKEN_RETURNED_OVER_HERE) -> 403 returned for POST https://cloudresourcemanager.googleapis.com/v1/projects/XXXGCPPROJECTIDXXXX:testIamPermissions
• run with gcloud auth activate-service-account invoked to login using service account -> firebase deploy generates a bearer token (replaced in logfile with XXXBEARER_ACCES_TOKEN_RETURNED_OVER_HERE) -> 403 returned for POST https://cloudresourcemanager.googleapis.com/v1/projects/XXXGCPPROJECTIDXXXX:testIamPermissions

This leads me to the hypothesis that the firebase deploy command is not utilising the service account configured using gcloud

[joehan]

@crunchie84 Sorry for the delay here, I had to do some research for this one. Here's what I've found:

When you run gcloud auth activate-service-account, you are setting up credentials on gcloud. Firebase CLI doesn't use gcloud when deploying, it calls Google Cloud APIs directly, so AFAIK it will not pick up these credentials configured in this way.

UPDATE
After looking through some other Github issues, I've realized that this duplicates #787 . At the moment, there is no way to deploy using a service account with Firebase. Some methods/deployments may work, but many require APIs that cannot be accessed by user created service accounts. There has been some discussion of implementing this, but I don't know of any timelines, unfortunately. Apologies for getting anyones hopes up :(

Fortunately, depending on what you wanted to do with the service account, Firebase may have support for your use case. If you are trying to use a service account for CI purposes, I recommend creating a firebase auth token to authenticate with. To do so, run
firebase login:ci
This will return an auth token. If you set the env variable FIREBASE_TOKEN to this, or provide the --token <token> flag to your firebase deploy, it will use this for authentication. This guide https://medium.com/google-cloud/how-to-set-up-cloud-build-for-firebase-cloud-functions-cffcf2812302 describes a good way to securely store and use these on your CI server under the 'The good solution' section.

[joehan]

Closing this one for now, as it duplicates another. @grantneale and @crunchie84, feel free to continue discussion here if there's anything I can help with related to this. Otherwise, if others are experiencing this too, please direct conversation to #787

[grantneale]

@joehan the duplicated issue is also closed. Does this mean that this will never be supported?

If so, I find it strange that the authentication approaches supported for deployment are not consistent across all Firebase services..

[mbleigh]

787 is still open and is going to be the canonical issue

[grantneale]

My mistake, apologies and thank you.