Firebase-functions: Cloud functions first call to Firestore very slow

[xkLander]

We are experiencing extreme cold boot times, even with a small function like helloWorld, it takes like 5-6 seconds when firestore is used. Please see the following examples.
The first function is done very fast, less than a second, I get the response "works", and the second time, after querying firestore, takes 6 seconds:
Function Version 1:

export const helloWorld = functions.https.onRequest(async (req, res) => {
  console.log(process.env.FUNCTION_NAME + ' has started');
  const admin = await import('firebase-admin');
  admin.initializeApp();
  res.status(200).json('works');
)}

Function Version 2, takes 6 seconds to return the user

export const helloWorld = functions.https.onRequest(async (req, res) => {
  console.log(process.env.FUNCTION_NAME + ' has started');
  const admin = await import('firebase-admin');
  admin.initializeApp();

  const userCol = await admin.firestore().collection('users').limit(1).get();
  const userDoc = userCol.docs[0];
  const user = userDoc.data();
  res.status(200).json(user);
)}

I am using Node 8, and these dependencies:
"firebase": "^5.9.1",
"firebase-admin": "^7.1.1",
"firebase-functions": "^2.2.1",

Related issues

[thechenky]

Hi @xkLander thank you for filing this, I'm sorry to hear you're having this issue. The first call to Firestore is quite heavyweight unfortunately since we have to create a gRPC connection under the hood and that takes some time. I'm curious - on subsequent runs of the V2 function, does it return faster? Since the connection to Firestore is established, it should be faster to call Firestore a second time the function is invoked (assuming it's still on the same warm instance).

[xkLander]

The second call, in a warm state, it is much faster, like 300-400ms, so thats not an issue, the issue is the cold boot :/

[thechenky]

I want to be careful in calling this a cold boot issue - cold starts are a little different. Global scope is what affects cold starts and makes them slower. Cold start should be the same for both of your two functions, regardless of what they're doing in the function body (as that is not part of cold start). What it looks like you're experiencing is slow first calls to Firestore - this is a different issue. I'm going to reach out to some folks on relevant teams to see what we can do to improve this.

Couple clarifying questions - which region is your Firestore and which region is your function in? We suggest you put them in the same region to reduce latency.

[xkLander]

I agree, its a firestore issue. The firestore is in nam5 (us-central), and cloud functions in us-central1

[thechenky]

Hi @xkLander I've filed an internal bug (bug reference 129483100) with the Firestore team and linked to this thread here. If you have anything more that can help us debug, please don't hesitate to paste it into this thread.

As this issue is not something that can be fixed in this repo, our policy is to close this out. However, I will ping this thread with any updates from the internal bug when there are any.

Thanks again, and if you experience any other issues, feel free to open up another bug here.

[thechenky]

Hi @xkLander I followed up with the team, and this is the same issue as https://github.com/googleapis/nodejs-firestore/issues/528. To summarize, we are using gRPC to make the call to Firestore, which is very heavyweight, and so it's being lazy-loaded on the first call, which is why you're seeing such long first call times (explained in https://github.com/googleapis/nodejs-firestore/issues/528#issuecomment-457986699). There has been a push to move to a more lightweight gRPC-JS (you can opt into this for now following these instructions: https://github.com/googleapis/nodejs-firestore/issues/528#issuecomment-458355027). Thanks again for opening this issue, sorry we cannot fix this quickly, but I have reiterated the need for faster first calls to Firestore.

[madakk]

@thechenky When attempting to set the GOOGLE_CLOUD_USE_GRPC_JS flag, my calls ends up failing with the following error:

Error: Not available in this library. Use @grpc/proto-loader and loadPackageDefinition instead at Object.exports.loadObject

It appears that although firestore is seemingly using gRPC-JS, google-gax is falling back to "legacy" behavior (although it seems to rely on the same flag?). Can you advise?

[polRk]

I think, it can help
https://www.youtube.com/watch?v=v3eG9xpzNXM

[thechenky]

@polRk thanks for the suggestion, but that's actually a different problem. This thread discusses the issue of a slow first call to Firestore. I've updated the description to make that a bit clearer as the original title made it a bit misleading.

[thechenky]

@madakk I'm hearing this issue has been fixed in a patch release of Firestore. You will need to be using the latest admin and functions versions, and also will need to redeploy the function. Can you try that and see if it works for you?

[hitchingsh]

Could someone telling me the steps to set the environment variable GOOGLE_CLOUD_USE_GRPC_JS? Is it through the web control panel, command line firestore deploy, or where?

Turns out its now used automatically when using the latest version

[thechenky]

@hitchingsh I have some good news - this will be the new default in the next functions release since we will be using the latest Firestore library that defaults to the smaller gRPC library. This release is coming very soon. For the meantime though, you can set that environment variable in the cloud console by going to Cloud Functions, selecting the function, and clicking Edit. Expand the list and at the bottom you'll see "+ Add variable":

[hitchingsh]

Thanks very much for the instructions, which were really helpful. I tried that but it did not reduce my load time when it makes the first call to Firestore after uploading. Still takes three seconds for the first query / or add (thereafter its very sub second)

[MuhammadUsman786786]

Why this issue is closed. This issue should be reopened

[hvdscheer]

Reopen this issue! This is not solved!

[malikasinger1]

reopen this issue

[malikasinger1]

same problem happening with firebase functions

[docaohuynh]

Major bug!! why is this issue closed?

[mbleigh]

This issue is closed because there is nothing in the firebase-functions repo that can be done to solve it. The correct place to follow up is: https://github.com/googleapis/google-cloud-node/issues/2942

[transitive-bullshit]

If this problem is affecting you, please upvote https://github.com/googleapis/google-cloud-node/issues/2942#issuecomment-604268183

[malikasinger1]

this may help:
https://stackoverflow.com/a/60910024/4378475