Firebase-tools: Unable to deploy Firebase app using service account default credentials

[miumiu0917]

[REQUIRED] Environment info

firebase-tools: 7.13.0

Platform: Ubuntu on Cloudbuild
from cloud-builders-community

[REQUIRED] Test case

• Create any Firebase app
• Attempt to deploy, using cloudbuld with a service account
• Note that even if the service account is assigned the Owner role, this still fails.
  

[REQUIRED] Steps to reproduce

• Attempt to deploy by running firebase deploy
  

[REQUIRED] Expected behavior

succesfully deployed.

[REQUIRED] Actual behavior

Starting Step #4 - "Deploy to Firebase Hosting"
Step #4 - "Deploy to Firebase Hosting": Pulling image: gcr.io/mike-242702/firebase
Step #4 - "Deploy to Firebase Hosting": Using default tag: latest
Step #4 - "Deploy to Firebase Hosting": latest: Pulling from mike-242702/firebase
Step #4 - "Deploy to Firebase Hosting": 3192219afd04: Already exists
Step #4 - "Deploy to Firebase Hosting": 17c160265e75: Already exists
Step #4 - "Deploy to Firebase Hosting": cc4fe40d0e61: Already exists
Step #4 - "Deploy to Firebase Hosting": 9d647f502a07: Already exists
Step #4 - "Deploy to Firebase Hosting": c8bd59397b26: Already exists
Step #4 - "Deploy to Firebase Hosting": f9289dfd19be: Already exists
Step #4 - "Deploy to Firebase Hosting": 9553b08b6292: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 37581b1ce15a: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 45ea08f1ad05: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Pulling fs layer
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Waiting
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Waiting
Step #4 - "Deploy to Firebase Hosting": 45ea08f1ad05: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 45ea08f1ad05: Download complete
Step #4 - "Deploy to Firebase Hosting": 37581b1ce15a: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 37581b1ce15a: Download complete
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Download complete
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Download complete
Step #4 - "Deploy to Firebase Hosting": 9553b08b6292: Verifying Checksum
Step #4 - "Deploy to Firebase Hosting": 9553b08b6292: Download complete
Step #4 - "Deploy to Firebase Hosting": 9553b08b6292: Pull complete
Step #4 - "Deploy to Firebase Hosting": 37581b1ce15a: Pull complete
Step #4 - "Deploy to Firebase Hosting": 45ea08f1ad05: Pull complete
Step #4 - "Deploy to Firebase Hosting": 134f3de54272: Pull complete
Step #4 - "Deploy to Firebase Hosting": 2d088f82ff68: Pull complete
Step #4 - "Deploy to Firebase Hosting": Digest: sha256:d7764e447534464a4ae049f7fa64f55ec4433a93e4a097003ab20e1b922b3d19
Step #4 - "Deploy to Firebase Hosting": Status: Downloaded newer image for gcr.io/mike-242702/firebase:latest
Step #4 - "Deploy to Firebase Hosting": gcr.io/mike-242702/firebase:latest
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.389Z] ----------------------------------------------------------------------
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.392Z] Command:       /usr/local/bin/node /usr/local/bin/firebase deploy --project develop --debug
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.393Z] CLI Version:   7.7.0
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.393Z] Platform:      linux
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.393Z] Node Version:  v13.8.0
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.408Z] Time:          Fri Feb 14 2020 03:27:36 GMT+0000 (Coordinated Universal Time)
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.408Z] ----------------------------------------------------------------------
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.409Z] 
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.418Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.419Z] > attempting to authenticate via app default credentials
Step #4 - "Deploy to Firebase Hosting": [2020-02-14T03:27:36.525Z] TypeError: Cannot create property 'refresh_token' on string 'Not Found
Step #4 - "Deploy to Firebase Hosting": '
Step #4 - "Deploy to Firebase Hosting":     at /usr/local/lib/node_modules/firebase-tools/node_modules/google-auth-library/lib/auth/oauth2client.js:208:28
Step #4 - "Deploy to Firebase Hosting":     at /usr/local/lib/node_modules/firebase-tools/node_modules/google-auth-library/lib/auth/computeclient.js:85:7
Step #4 - "Deploy to Firebase Hosting":     at Request._callback (/usr/local/lib/node_modules/firebase-tools/node_modules/google-auth-library/lib/transporters.js:106:7)
Step #4 - "Deploy to Firebase Hosting":     at Request.self.callback (/usr/local/lib/node_modules/firebase-tools/node_modules/request/request.js:185:22)
Step #4 - "Deploy to Firebase Hosting":     at Request.emit (events.js:321:20)
Step #4 - "Deploy to Firebase Hosting":     at Request.EventEmitter.emit (domain.js:485:12)
Step #4 - "Deploy to Firebase Hosting":     at Request.<anonymous> (/usr/local/lib/node_modules/firebase-tools/node_modules/request/request.js:1154:10)
Step #4 - "Deploy to Firebase Hosting":     at Request.emit (events.js:321:20)
Step #4 - "Deploy to Firebase Hosting":     at Request.EventEmitter.emit (domain.js:485:12)
Step #4 - "Deploy to Firebase Hosting":     at IncomingMessage.<anonymous> (/usr/local/lib/node_modules/firebase-tools/node_modules/request/request.js:1076:12)
Step #4 - "Deploy to Firebase Hosting": 
Step #4 - "Deploy to Firebase Hosting": Error: An unexpected error has occurred.
Finished Step #4 - "Deploy to Firebase Hosting"
ERROR
ERROR: build step 4 "gcr.io/mike-242702/firebase" failed: step exited with non-zero status: 2

[diogoterremoto]

Same thing happening today on a Cloud Build environment with the correct IAM roles to deploy to Firebase Hosting.

[samtstern]

@miumiu0917 and @diogoterremoto this looks very similar to this issue:
https://github.com/firebase/firebase-tools/issues/1966

It looks like a Cloud Build issue to me, you're getting a 404 from an OAuth endpoint which implies some sort of network issue inside the container. As you can see in the stack trace the 404 happens with google-auth-library rather than inside firebase-tools specific code.

Have you tried reporting this issue to Cloud Build directly?

[wvanderdeijl]

We are suffering from this same issue (unable to use firebase-tools cli on google cloud build). I am still investigating but could this be causes by the deprecation of the GCP metadata endpoints? https://cloud.google.com/compute/docs/migrating-to-v1-metadata-server#supported-google-libraries

This requires GCP auth related libraries to be updated and the version of google-auto-auth that is used by firebase-tools is very old which also means it is using a very old version of gcp-metadata and gcp-auth-library:

$ npm ls gcp-metadata
├─┬ [email protected]
│ └─┬ [email protected]
│   └── [email protected]
└─┬ [email protected]
  └─┬ [email protected]
    └── [email protected]

$ npm ls google-auth-library
├─┬ [email protected]
│ ├─┬ @google-cloud/[email protected]
│ │ └─┬ [email protected]
│ │   └── [email protected]  deduped
│ └─┬ @google-cloud/[email protected]
│   ├─┬ @google-cloud/[email protected]
│   │ └── [email protected]  deduped
│   └─┬ [email protected]
│     └── [email protected]  deduped
├─┬ [email protected]
│ ├─┬ @google-cloud/[email protected]
│ │ └── [email protected]  deduped
│ ├─┬ [email protected]
│ │ └── [email protected]
│ └─┬ [email protected]
│   └── [email protected]  deduped
└─┬ [email protected]
  ├── [email protected]
  └─┬ [email protected]
    └── [email protected]  deduped

The page about the metadata server deprecation lists a number of nodejs libraries that need updating. It seems like at least gcp-metadata and google-auth-library that firebase-tools uses are too old

[samtstern]

@wvanderdeijl I think that might be it! I will dig into it, thank you for that.

[wvanderdeijl]

Thanks for the quick response. In the meanwhile I am trying to create my own workaround as we are currently unable to perform an important production deployment.

[samtstern]

Working on this in #1973 ... do you think you could try installing firebase-tools from my branch and seeing it that is a workaround for you?

[wvanderdeijl]

this only occurs on google cloud build (due to the metadata server usage). So a roundtrip to gcb for an attempt takes some time. Do you know the easiest way to install firebase-tools from your branch in a script on gcb?

[samtstern]

You should be able to do:

npm install -g git://github.com/firebase/firebase-tools.git#ss-update-auto-auth

[wvanderdeijl]

Installing from git works fine locally, but it fails when running in the cypress/base:10.16.0 docker container we are using in google cloud build:

$ docker run -it cypress/base:10.16.0 bash
root@ecc2be77d2d8:/# npm install -g --unsafe-perm git://github.com/firebase/firebase-tools.git#ss-update-auto-auth
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@~2.1.2 (node_modules/firebase-tools/node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! path /usr/local/lib/node_modules/firebase-tools/lib/bin/firebase.js
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall chmod
npm ERR! enoent ENOENT: no such file or directory, chmod '/usr/local/lib/node_modules/firebase-tools/lib/bin/firebase.js'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

It seems like the lib directory is not created, so creating the symlink for the firebase binary fails. If I run npm install with --bin-links=false to skip that phase I can see that the lib folder is missing:

root@c1f0fe1ecce9:/# ls -al /usr/local/lib/node_modules/firebase-tools
total 56
drwxr-xr-x   5 root root   4096 Feb 14 21:02 .
drwxrwxr-x   1 root staff  4096 Feb 14 21:02 ..
-rw-r--r--   1 root root     79 Oct 26  1985 CHANGELOG.md
-rw-r--r--   1 root root   1075 Oct 26  1985 LICENSE
-rw-r--r--   1 root root   9314 Oct 26  1985 README.md
drwxr-xr-x 412 root root  12288 Feb 14 21:02 node_modules
-rw-r--r--   1 root root   5371 Feb 14 21:02 package.json
drwxr-xr-x   2 root root   4096 Feb 14 21:02 standalone
drwxr-xr-x   5 root root   4096 Feb 14 21:02 templates

I'll keep you posted

[samtstern]

@wvanderdeijl ah ok sorry about the issue there! You could also try this to install the firebase-tools branch on your machine:

git clone -b ss-update-auto-auth https://github.com/firebase/firebase-tools
cd firebase-tools
npm install
npm run build
npm link

[wvanderdeijl]

I finally got it to work by not running the npm install as root as that will not completely run the prepare script. This did the trick:

su node -c "npm install"

This means I run the npm install with the non-privileged user node. This is all unrelated to this specific firebase-tools issue, so sorry for spamming this issue.

Anyhow; the npm install completed at google cloud build and I can confirm that the deployment of firebase now runs fine using your branch and the automatic auth discovery at the google cloud build server.

Should I keep this workaround in place or do you plan to release a new version with the fix very soon?

[samtstern]

@wvanderdeijl thanks for confirming! Depends on your definition of "soon". This fix seems simple and important so we can get a new version out early next week, but we can't get it out today (we don't release on Fridays).

[wvanderdeijl]

Thanks for the update. I am already very impressed with the very quick response times. I understand that releasing a new version on a friday as a "one man job" is not the best idea in the world.
I'll use the workaround for now and will check early next week for a new release

Thanks again!

[velith]

We had the same issue with our deployments and it is resolved by using branch mentioned :+1: A bit funny this "update" that made the old library not working anymore happened on Valentine's Day... where was the love? :smile: :heart:

[samtstern]

The fix for this bug has been released in version 7.13.1

[rajivsubra1981]

Sorry for the basic q - we've been facing the same error on the deploy step on cloudbuild (last successful deploy was Feb 13th). I've updated firebase tools on my machine - but that doesnt seem to fix it on cloudbuild. How does one update the firebase tools version on cloudbuild

[crtag]

@rajivsubra1981 Had the same issue, solved by rebuilding docker image for firebase community builder from here https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/firebase, just follow the instruction in the root project readme. Once it was uploaded I also deleted the old image and everything worked fine. ,

[rajivsubra1981]

@crtag Thank you! I ran gcloud builds submit --config cloudbuild.yaml inside \cloud-builders-community\firebase
and it fixed it.

[miumiu0917]

@samtstern
Thank you!!
It worked fine for me!!