Firebase-functions: Deploy Error: Failed to retrieve function source code

[ahaverty]

Version info

firebase-functions: 0.5.2
firebase-tools: 3.5.0
firebase-admin: 4.1.3

Steps to reproduce

Only 15 out of 17 of my functions are deploying successfully

Were you able to successfully deploy your functions?

Not exactly,
I'm getting the following error since deploying my 17th function:

...
i  functions: updating function primeFunctions...
⚠  functions[orderStateChange]: Deploy Error: Failed to retrieve function source code
⚠  functions[slackPost]: Deploy Error: Failed to retrieve function source code
...

Expected behavior

Successful deployment of all 17 functions

Actual behavior

Two functions receive the above error. Deployment takes longer than usual to complete. Strangely it's 2 different functions each time that fail to retrieve the source code.

[ahaverty]

Update: On my third deployment attempt, 16/17 deployed successfully, but a different function failed due to the same issue.

[ahaverty]

Update 2: That deployment actually resulted in a different error

⚠  functions[merchantSettingsTask]: Deploy Error: Operation interrupted.

And here's what happens on the log for that function.
Although it seems like that functions previous deployment is still running regardless of the failure?:

[ahaverty]

Update 3: All 17 functions had a deployment success on attempt 4 (without any changes to code or the command used).
Can someone clarify if a function fails to deploy, what state is it left in (if a previous deployment was a success) are the other functions updated?
I could see inconsistent deployments causing issues if one function depended on another function. It looks like function deployment is independent at the moment?
Could I suggest some kind of flag like --rollback-on-fail to allow all or nothing deployments? 👍

[laurenzlong]

Hey @ahaverty thanks for the detailed repro instructions. This is definitely a bug, and I'll try to reproduce it on my end and figure out how to fix it. Will keep you updated on this thread. Here are the answers to your questions:

Can someone clarify if a function fails to deploy, what state is it left in (if a previous deployment was a success) are the other functions updated?

It depends on exactly how the error occurred. The best way to check is to go to https://console.cloud.google.com/functions/list. There will be check marks or x's next to the functions depending on their current state. Clicking on the name of the function, then navigating to the "source" tab will show you the current code that's deployed.

I could see inconsistent deployments causing issues if one function depended on another function. It looks like function deployment is independent at the moment?

Yes, the functions deployment is independent at the moment, since they happen in parallel.

Could I suggest some kind of flag like --rollback-on-fail to allow all or nothing deployments?

That's a great suggestion and something that we've been thinking about. Unfortunately it's a non-trivial effort for us due to our current technical constraints. We don't version control your functions yet, so it would not be feasible to do roll backs at the moment. This is still a really great feature request and something that we'd like to support in the future.

[ahaverty]

@laurenzlong Awesome, I didn't know cloud.google console was linked like that! independent deploys aren't such a big deal with the console then. Just to note, I had no issues deploying the same functions 5 minutes later to another one of my projects. Let me know if I can help with debugging👏 Thanks again👌

[laurenzlong]

That's good to hear! Did you save the firebase-debug.log file from the failed deploy? (It gets automatically created in the working directory when there's a failure) That would be really helpful.

[ahaverty]

@laurenzlong I don't have it in the working directory or in my trash. Is it removed on each new deploy? Can I grab anything from google cloud logs?

[laurenzlong]

Unfortunately it gets written over in successive deploys so it's gone now :(

[ahaverty]

Sorry @laurenzlong 😞, I never thought of it. I'll save and send if it happens again.

[laurenzlong]

Not a problem! Yes that would be great, thanks!

[laurenzlong]

Hey @ahaverty I'm unable to reproduce the bug. A new release for the runtime went to prod today, so I think that probably resolved the issue. I'm going to close this issue for now, but feel free to re-open it if you see the errors again.

[ahaverty]

Thanks @laurenzlong haven't hit it since anyway, but will do!

[ahaverty]

Hi @laurenzlong
I hit a similar issue today again on a different project

⚠ functions[merchantSettingsTask]: Deploy Error: Failure in the execution environment

But there was no firebase-debug.log in my working directory, project folder or functions folder so I'm not sure what else I can do. 🤔

[ahaverty]

Strange, the firebase-debug.log appeared after calling another firebase deploy afterwards. I'll send a private bug report with the log and reference here 👌

[ahaverty]

No I'm wrong, that debug log wasn't for the previous deploy. This may be a bug with firebase-tools? I'd have no way of verifying/testing unfortunately.
Also, the second deploy here started all functions correctly.

[laurenzlong]

Hey @ahaverty I was mistaken. CLI typically generates a firebase-debug.log for errors, but doesn't do so for functions deploy errors unless you ran the command with the --debug flag.

Next time you encounter a deploy error, can you run it again with "firebase deploy --debug"?

As for the error you saw, it seemed to be a one-off. (We're working hard to eliminate these!)

[ahaverty]

Good to know 👌 Does running firebase deploy --debug create a log for the previous deployment or the current deploy? Catching these intermittent fails would be hard unless I ran --debug everytime, right?
(Functions will massively help our team, and yourself and firebase are doing a great job, especially for a beta🤗!) Thanks for all the help

[ahaverty]

Hi @laurenzlong this issue is actually happening to me quite often, sometimes up to 4 functions fail to deploy. I finally had the foresight to run with the --debug flag and manage to catch one! I'll send the log over a support ticket

[ahaverty]

Responded to the original ticket support I opened with Gabriel, asked for him to pass it on to you. Thanks again

[Abrahamh08]

I'm having the same problem right now with one of my firebase functions. It was working earlier, but now every time I deploy (after a minor update) I get:

⚠ functions[functionName]: Deploy Error: Failed to retrieve function source code

[laurenzlong]

@Abrahamh08 can you file a support ticket? https://firebase.google.com/support/

[Abrahamh08]

@laurenzlong done

[dallen4]

Not sure if this will help, but I was having the same issue and found this on Stack Overflow. Changing the node/npm version you install firebase-tools with seemed to fixed for me.

[dodomui]

I'm having the same error now.
And it only happens to one of my project another project is just working fine.
Submitted ticket to firebase 6 hours ago, but didn't solved yet, my project get stuck here.
Change different npm version doesn't help on my case.

firebase-admin: ^8.10.0,
firebase-functions: ^3.6.1,
firebase-tools: 8.9.2

[stevehwg]

@dodomui it could be billing issues. I found this SO here which solved my problem after upgrade to Blaze pay as you go plan.

[dongido001]

I'm on "Blaze pay as you go plan" but I still get this error

[8gb]

Tried all the steps above, didnt worked for me. What work for me is:

1. Rename all functions.
2. Deploy and say yes to delete all the old functions.
3. Rename it back to the original functions name.
4. Deploy and delete all the old functions deployed in step 2.