Firebase-tools: Random build timeouts using `firebase deploy`

[whaatt]

I have a large project with many cloud functions, and lately an error has been popping up on random functions in my deploy: Deployment error. Build failed: Build has timed out. Sometimes it says Build error details not available instead of the timeout.

This affects a small percentage of functions but is enough to be annoying (often requiring many redeploys).

Triage

If I check Stackdriver logs for the build failure, I see an error message INVALID_ARGUMENT in all cases.

But I never used to get this before and the timeouts are not at all deterministic (when they occur OR for which functions they occur). If this is a GCP limit I'm running into, I'm not really sure which one it is.

Moreover, some days seem to have far fewer build timeouts than other days.

Environment

The only things that have changed outside the code are package version bumps, including an upgrade from the Node 6 runtime to the Node 10 runtime, as well as a version bump to the latest firebase CLI (installed globally).

Also worth noting is that I do a large number of parallel deploys, but the errors occur even when only a handful of functions are being deployed concurrently.

Any ideas? Much appreciated!

[samtstern]

I've seen this before and I also believe it was correlated to Node
version. Since it's an intermittent issue and it's definitely happening on
the back end I don't think there's anything we can do in the CLI to help.

Would you mind writing in to Firebase support about this issue? Provide
the project ID and some timestamps of build failures. You can link this
issue and tell them to cc me if you'd like.

On Wed, Sep 11, 2019, 6:34 PM Sanjay Kannan [e-mail]
wrote:

I have a large project with many cloud functions, and lately an error has
been popping up on random functions in my deploy: Deployment error. Build
failed: Build has timed out. This affects a small percentage of functions
but is enough to be annoying (often requiring many redeploys).

If I check Stackdriver logs for the build failure, I see an error message
INVALID_ARGUMENT in all cases.

But I never used to get this before and the timeouts are not at all
deterministic (when they occur OR for which functions they occur). If this
is a GCP limit I'm running into, I'm not really sure which one it is.

The only things that have changed outside the code are package version
bumps, including an upgrade from the Node 6 runtime to the Node 10 runtime,
as well as a version bump to the latest firebase CLI (installed globally).

Any ideas? Much appreciated!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/firebase/firebase-tools/issues/1655?email_source=notifications&email_token=ACATB2VIQ3MHDRHSZ66ZCC3QJGMCLA5CNFSM4IV4LKA2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HK3R62Q,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACATB2TEJJ3YLJAOVPFHWBDQJGMCLANCNFSM4IV4LKAQ
.

[op]

We've seen an increasing amount of timeouts too. Might be our CI provider that's having issues though. (We're also on Node 10.)

Wouldn't it make sense to make an effort to retry on timeout failures though?

[mattgstevens]

Have also been experiencing this & am using Node 10 for functions.
I opened up a support ticket linking to this issue.

Thanks @samtstern for the suggestions about what the cause might be. Will see what the response is from the firebase team.

[casim]

I had the same issue consistently now with Node 8 and firebase-admin 6.12.0 for the pass few days. If I upgrade to the latest version of firebase-admin the issue doesn't happen anymore (but then I have other issues related to firebase-admin not found). I'm trying to figure out if there is a combination of admin/tools that can work for me.

[mitchelloptima]

multiple projects doing this now. failed: retries exhausted after 6 attempts

[alexanderPastornicky]

same : retries exhausted after 6 attempts

[mitchelloptima]

https://status.firebase.google.com/incident/Hosting/19015 seems they are working on it, we were just a bit early.

[whaatt]

Just for clarity, I think two separate issues are being discussed in this thread: (A) the original deploy issue for Cloud Functions and (B) the Firebase hosting issue this morning, which is being worked on (as @mitchelloptima noted).

[kledk]

Apparently the issue seems to be resolved today. My builds are passing again without timeouts.

[mattgstevens]

@whaatt agreed, this issue is about (A) cloud functions deployments (although I'm glad those experiencing hosting issues are happy again).

---

Small update from my side is that support received the ticket, I provided some additional information (timestamps of recent failed deploys, the logs from a failing firebase deploy with the --debug flag, projectId's and region) and the ticket has been raised to the engineering team.

Think its worth mentioning that the support rep said others have reported seeing "Deployment error. Build failed: Build has timed out"; if its a "big" enough / often breaking for other teams then hopefully it gets addressed sooner rather than later.

[whaatt]

Haven't gotten the original error in a while. If others are no longer experiencing the error, please feel free to close out this issue.

[mattgstevens]

back from vacation, apologies to post on this closed issue.

want to confirm that our CI has not been failing for at least 2 weeks due to this deployment issue. onward!