Firebase-admin-node: Deadline Exceeded error!

[warlockdn]

[REQUIRED] Step 2: Describe your environment

• Operating System version: Ubuntu 16
• Firebase SDK version: v6
• Firebase Product: Firestore

[REQUIRED] Step 3: Describe the problem

I am doing a basic Add to Firestore through Node Admin SDK. I just add a object to a document and then wait for the document ID to return after that I send it to the front end to be subscribed for realtime updates. But sometimes I receive the error while adding a document and also updating as to Deadline Exceeded

Steps to reproduce:

The code that I have added to my backend.

let ref = db.collection("orders");

return new Promise((resolve, reject) => {
            ref.add(order).then((order) => {
                logger.info(order);
                resolve(order);
            }).catch((err) => {
                logger.info(err);
                reject(err);
            })
})

There you go I receive errors at catch.

Also checked my account I don't see any high usage since it is in development phase so hardly adding 10-20 docs on a normal day.

Can you please help here. Thanks

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[schmidt-sebastian]

Hi @warlockdn! Sorry to hear that you are having connection issues to Firestore. Note that the initial connection attempt can sometimes be slow, as there is a lot of overhead (Channel establishment, SSL handshakes, authentication). Once you have a connection to Cloud Firestore, the underlying channel will get re-used for further operations. You may be able to work around your issue if you start by issuing a DocumentReference.get() call. We retry those automatically if the connection establishment takes a bit longer than it should.

[warlockdn]

Hi @schmidt-sebastian sorry I don't quite understand. On application restart I already created a connection to firebase. But anyways can you guide me to a resource where I can see some demo or sample code as to why and how this can be used. ?

[schmidt-sebastian]

As said, this is not a solution, but rather a workaround:

const docRef = db.collection("orders").doc();
await docRef.get(); // This will retry on DEADLINE_EXCEEDED
// Now the channel is established.
docRef.create({...}):

[warlockdn]

Do u have a permanent solution to this ? I don't know if with this I can go into production ? If I have to wait to retry on things. Coz the point of being realtime will be of no use if on a load i receive these errors. Or may be i need to follow some other process ?

[schmidt-sebastian]

If you continue to run into these issues, then we should look at your network stack and how you connect to our data centers. If we do need to go down that route, we can let you know some environment variables that will help us debug.

[warlockdn]

Yea you can help me then lemme know what needs to be done so that I don't end up embarrassing my product haha. Thanks

[schmidt-sebastian]

Please set the following environment variables:

GRPC_TRACE=all
GRPC_VERBOSITY=DEBUG

[warlockdn]

so this is the error I finally received

I0916 13:25:14.586417074   10852 completion_queue.cc:951]    RETURN_EVENT[0x23f6660]: QUEUE_TIMEOUT
I0916 13:25:14.591104089   10852 completion_queue.cc:851]    grpc_completion_queue_next(cq=0x23f6660, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=(nil))
I0916 13:25:14.591189393   10852 completion_queue.cc:951]    RETURN_EVENT[0x23f6660]: QUEUE_TIMEOUT
I0916 13:25:14.591880859   10852 completion_queue.cc:851]    grpc_completion_queue_next(cq=0x23f6660, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=(nil))

[warlockdn]

so do I need to initialise firebase connection on each function call or does it work once on initialising the server ?

[warlockdn]

??

[schmidt-sebastian]

Sorry, I am just catching up on my emails. You only need to initialize the client connection only once, and the network channel should stay open.

[warlockdn]

So what's the issue with the time out then ? My internet connectivity is
pretty good

On Wed, Sep 19, 2018, 12:23 AM Sebastian Schmidt [e-mail]
wrote:

Sorry, I am just catching up on my emails. You only need to initialize the
client connection only once, and the network channel should stay open.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/firebase/firebase-admin-node/issues/349#issuecomment-422506974,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADR8XkgdpookRGwgcuh3JlynA0OnFEDmks5ucUEygaJpZM4WYWqy
.

[schmidt-sebastian]

Just to confirm, you are getting most of these errors on the initial document write and before you are using any other Firestore operations? I am suggesting that your initial operation should be a document read, which we will retry. The "time to first byte" from Firestore is not necessarily related to your network connection.

[CapitanRedBeard]

I get this as well. I have a collection with about 300k documents on it and simply calling .get() on that collection will result in the timeout. Its a firebase problem and not your local system.

I've seen this happen in hosted cloud functions, and the fix is to increase the timeout limit in the function details (max time is 540 seconds, default is 60 seconds). However in the node sdk I can't get figure out how to extends the default past the 60 second limit.

[BrodaNoel]

I'm dealing with the same issue.
I reported it here: https://github.com/steadyequipment/node-firestore-backup/issues/48, due to it's actually this library who is throwing the error.

[farhankhwaja]

I am having the same issue, when try to read my collection which has 100k docs, it gives me DEADLINE ERROR message. How to solve this, even a simple read is so trouble some. Is there a way to export the data from the DB without hitting this Error

[CapitanRedBeard]

My workaround is to find a size that doesn't hit the DEADLINE ERROR, like 10k, then just loop by 10k increments and read for the database until you have read all your docs.

You can do something like this:

  const FOO_READ_LIMIT = 10000
  const foosRef = firestore.collection('foo');

  try {
    let isCalculating = true;
    let lastDoc = null;

    async.whilst(() => {
      return isCalculating;
    },
    async (next) => {
      const fooQuery = foosRef.limit(FOO_READ_LIMIT)
      const pagedFooQuery = 
        lastDoc 
          ? pagedFooQuery.startAfter(lastDoc)
          : pagedFooQuery

      const fooSnapshots = await pagedFooQuery.get();
      const size = fooSnapshots && fooSnapshots.size;

      if (size < FOO_READ_LIMIT) {
        isCalculating = false;
        next()
      }

      await doStuff(fooSnapshots);

      lastDoc = fooSnapshots.docs[FOO_READ_LIMIT - 1]
      next()
    },
    (error) => {
      console.log('All data is read');
    });
  } catch(e) {
    // Something wen wrong
  }

[farhankhwaja]

@CapitanRedBeard I ended doing something similar. But, thanks for the reply.

If the Firebase team can add an export option to the Firestore DB like RTDB has, it would help some folks here

[jonrandahl]

@CapitanRedBeard & @farhankhwaja while I don't know if this will help you, still I was running into the same Deadline exceeded errors with only less than 2k of records! I have implemented this into all of my cloud functions, hopefully, this will help you too:

/**
* Set timeout and memory allocation
* In some cases, your functions may have special requirements for a long timeout value or a large * allocation of memory.
*You can set these values either in the Google Cloud Console or in the function source code (Firebase only).
*/
const runtimeOpts = {
  timeoutSeconds: 540, // 9 minutes
  memory: '2GB'
}
/**
 * When a csv file is uploaded in the Storage bucket it is parsed automatically using
 * PapaParse.
 */
exports.covertToJSON = functions
  .runWith(runtimeOpts) // <== apply timeout and memory limit here!
  .storage
  .object()
  .onFinalize( object => {
[...]

Sadly I can't remember where I found that snippet but as I said, every little bit helps!

Jon

[tallalj]

So why is this happening? what is the limit that is being exceeded here? It is on blaze plan so this should not happen. Instead firebase should let us query as much as we want..
Kindly specify the technical issue that needs to be overcome to resolve this.

[schmidt-sebastian]

We currently configure a 60 second deadline for the RunQuery call (which both Query.get() and Query.stream() use). We believe that this should fit most use cases, and would like to refrain from increasing this deadline. Having a shorter deadline limits the amount of work that the backend does for clients that are no longer interested in a Querie's result, which generally results in fewer billed reads.

While somewhat convoluted, you can configure a different deadline for your client:

 const firestore = new Firestore({
      clientConfig: {
        interfaces: {
          'google.firestore.v1.Firestore': {
              methods: {
                RunQuery: {
                  timeout_millis: YOUR_NUMBER_HERE
                }
              }
            }
          }
        }
      });

[npomfret]

I'm getting these DEADLINE_EXCEEDED error when running tests using only very small sets of data (with "firebase-admin": "^7.0.0" / node v8.14.0). I'm also using the blaze plan so I don't think it's billing related. I'm not running inside a cloud function, it's just a regular node app.

It doesn't happen consistently either. I run small tests and it feels like 50% of the time my query times-out and runs into this error. The query itself returns a single record and should run in a fraction of a second.

Is it possible a contention issue? I have a number of parts to my app that may well be issuing queries and updates at the same time. It _feels_ like something is deadlocked.

An example query that gets stuck in this way is:

        const collection = this._firestore.collection(Collections.OMS);
        const existingOpenOrders = await collection
            .where("accountId", "==", this._accountId)
            .where("exchangeId", "==", exchangeId)
            .where("type", "==", DocTypes.OPEN_ORDER)
            .get()

It returns between zero and maybe 4 records only.

I ran why-is-node-running at the point that it all locks up and got this, which doesn't mean much to me unfortunately.

[schmidt-sebastian]

Can you do two things for me:

• Do a blind write before this query to make the connection is established.
• Share you Firestore logs (via setLogFunction())?

Thanks!

[npomfret]

Hi - where can I find information about how to use setLogFunction? Sorry, have search but can't see it

[hiranya911]

https://cloud.google.com/nodejs/docs/reference/firestore/0.8.x/Firestore#.setLogFunction

When using the Admin SDK:

admin.firestore.setLogFunction((log) => {
  console.log(log);
});

[npomfret]

Thanks. I'll add this, but the error isn't happening right now. I suspect it's something to do with my app. It's very "bursty" in terms of database updates. Possibly I was testing while a burst of writes was happening and that was blocking my local node process?

[wilhuff]

OK, if this does come back, let us know.

[hesselbom]

I just got this error suddenly when running database.doc('orders/1234').set(data, { merge: true }).

Just happened one time so far.

[wilhuff]

If you can come up with something that reproduces this or if you can capture logs while this is happening we'll be able to look at it.

Note that there was a connection pooling bug addressed as part of #499. That bug would manifest in circumstances where there were many long-lived listeners running in the application.

If you start seeing this more often, try the fix we there (a new release containing the fix is imminent).

[rhyek]

I'm getting this error a bit frequently. Stack trace on Sentry does not show where in my code this is happening. I'm assuming this is being emitted inside internals of firestore running in the background. Is there a way for me to determine what query is failing?

[rhyek]

I have a few endpoints on my api that are timing out and am getting a 502 from cloudflare, while having no idea what's causing this behavior. It sounds from what I read here is the queries are just never resolving and these deadline exceptions are happening in the background?

Here is a stack trace from one of the errors Sentry caught for me:

Error: 4 DEADLINE_EXCEEDED: Deadline Exceeded
  File "/build/node_modules/grpc/src/common.js", line 91, col 15, in Object.exports.createStatusError
    let error = new Error(message);
  File "/build/node_modules/grpc/src/client.js", line 233, col 26, in ClientReadableStream._emitStatusIfDone
    var error = common.createStatusError(status);
  File "/build/node_modules/grpc/src/client.js", line 211, col 8, in ClientReadableStream._receiveStatus
    this._emitStatusIfDone();
  File "/build/node_modules/grpc/src/client_interceptors.js", line 1272, col 15, in Object.onReceiveStatus
    emitter._receiveStatus(status);
  File "/build/node_modules/grpc/src/client_interceptors.js", line 568, col 42, in InterceptingListener._callNext
    return this.delegate[method_name](args_array[0], next_listener);
  File "/build/node_modules/grpc/src/client_interceptors.js", line 618, col 8, in InterceptingListener.onReceiveStatus
    this._callNext('onReceiveStatus', [status]);
  File "/build/node_modules/grpc/src/client_interceptors.js", line 1029, col 24, in null.<anonymous>
    first_listener.onReceiveStatus(response.status);

This is with 7.4.0.

[rhyek]

I should note that my node logs show the following. These don't immediately precede the above error. Sometimes they're hours apart:

(node:17) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added. Use emitter.setMaxListeners() to increase limit

[schmidt-sebastian]

@rhyek Regarding the MaxListenersExceededWarning. This should be addressed by the latest release of grpc-js. If you re-install your modules, you should automatically get 0.4.3 of @grpc/grpc-js. In versions prior to 0.4.1, these warnings showed up especially when it took longer than normal to establish a GRPC channel.

[Sandeeprao69]

I am on blaze plan and I have about 90K documents and when i do admin.firestore().collection(collectionName).doc(id).collection('xxxx').get().then(()=> { //do something })

in my firebase function i get Error: Deadline exceeded
Please not i am running my function with 2gb memory and 540 sec of timeout.

[slinkstep]

I am on blaze plan and I have about 90K documents and when i do admin.firestore().collection(collectionName).doc(id).collection('xxxx').get().then(()=> { //do something })

in my firebase function i get Error: Deadline exceeded
Please not i am running my function with 2gb memory and 540 sec of timeout.

Having the same issue when it comes to read and iterate over large collections (+100 docs)

[schmidt-sebastian]

We are preparing a release of @google-cloud/firestore v2.3.0 that will increase the timeout for all of our operations to 60s. This applies to all operations issued by the SDK, and is independent of the timeout set by any runtime environment (such as GCF). We hope that this alleviates some of the issues you are encountering.

[slinkstep]

I started an issue with the error details and some code. I hope you can help me to find out a solution, since it's a critical feature for our project.

https://github.com/firebase/firebase-functions/issues/556

Thanks a lot. @schmidt-sebastian

[Sandeeprao69]

Hi,

I was testing my app with about 180k records and i was trying to generate csv from json. since firebase query throw this error after 60k records what i did is break my operation into multiple steps.

I wrote a recursive function which paginate and fetched only 50k in each call.

First query will be like
db.collection('XXX').doc('id').collection('xxx').orderBy("createdAt").limit(50000);

Second query will be like
db.collection('XXXXX').doc('id').collection('XXXX').orderBy("createdAt").startAfter(lastRecord).limit(50000);

Please note createdAt was a field in my collection.

My firebase function took about 1.5 minute to complete this operation.

[slinkstep]

@schmidt-sebastian Still getting the same error after @google-cloud/firestore update to v2.3.0

[schmidt-sebastian]

@Sandeeprao69: Even with 60s, you will likely not be able to read 60,000 documents. Your options are to split the query up using cursors (as shown in your comment) or to increase the deadline manually to suit your needs:

 const firestore = new Firestore({
      clientConfig: {
        interfaces: {
          'google.firestore.v1.Firestore': {
              methods: {
                RunQuery: {
                  timeout_millis: YOUR_NUMBER_HERE
                }
              }
            }
          }
        }
      });

[Sandeeprao69]

@Sandeeprao69: Even with 60s, you will likely not be able to read 60,000 documents. Your options are to split the query up using cursors (as shown in your comment) or to increase the deadline manually to suit your needs:

 const firestore = new Firestore({
      clientConfig: {
        interfaces: {
          'google.firestore.v1.Firestore': {
              methods: {
                RunQuery: {
                  timeout_millis: YOUR_NUMBER_HERE
                }
              }
            }
          }
        }
      });

I was able to fetch 60k records in less than 60s actually in firebase cloud function.
I faced deadline exceeded error after 65k.

[slinkstep]

My workaround is to find a size that doesn't hit the DEADLINE ERROR, like 10k, then just loop by 10k increments and read for the database until you have read all your docs.

You can do something like this:

  const FOO_READ_LIMIT = 10000
  const foosRef = firestore.collection('foo');

  try {
    let isCalculating = true;
    let lastDoc = null;

    async.whilst(() => {
      return isCalculating;
    },
    async (next) => {
      const fooQuery = foosRef.limit(FOO_READ_LIMIT)
      const pagedFooQuery = 
        lastDoc 
          ? pagedFooQuery.startAfter(lastDoc)
          : pagedFooQuery

      const fooSnapshots = await pagedFooQuery.get();
      const size = fooSnapshots && fooSnapshots.size;

      if (size < FOO_READ_LIMIT) {
        isCalculating = false;
        next()
      }

      await doStuff(fooSnapshots);

      lastDoc = fooSnapshots.docs[FOO_READ_LIMIT - 1]
      next()
    },
    (error) => {
      console.log('All data is read');
    });
  } catch(e) {
  // Something wen wrong
  }

That is the best aproach I have found for getting thorugh this error

[digvan]

Still happening even with pagination

[rhyek]

Stopped using firebase a while ago and this bug is one the main reasons (there were a few others). Had a really rough week in production where we had to re-architect a bunch of backend stuff as quickly as possible due to this issue and it's still not resolved.

Very interesting product, otherwise.

[schmidt-sebastian]

We are tracking this internally via bug ID 139021175.

[schmidt-sebastian]

@digvan / @rhyek: Can you send timestamps and project ID to mrschmidt(at)google.com? Thanks!

[digvan]

@schmidt-sebastian, sent you an email.

[magician11]

 const firestore = new Firestore({
      clientConfig: {
        interfaces: {
          'google.firestore.v1.Firestore': {
              methods: {
                RunQuery: {
                  timeout_millis: YOUR_NUMBER_HERE
                }
              }
            }
          }
        }
      });

I'm getting Error: 4 DEADLINE_EXCEEDED: Deadline exceeded

We're on the Blaze plan and I think we have something like 10000 records in a collection.

Seems like a possible solution is to increase the timeout value.

My code looks something like this..

    const snapshot = await admin
      .firestore()
      .collection('some-collection')
      .get();

Just not sure how to integrate the code snippet @schmidt-sebastian is suggesting into my code.

[schmidt-sebastian]

If you are using the Admin SDK, you can set is as follows:

const firestore = admin.firestore();
firestore.settings({clientConfig: ...});
await firestore.collection('...').get();

[jketcham]

FWIW, I was experiencing this error when using @google-cloud/[email protected] to write a little over 1k documents to Firestore in a cloud function, and what seemed to fix it for me was explicitly returning a promise that resolved when all my batch write operations were complete. Before I was using the async syntax and it wasn't liking that (even thought that still returns a Promise). Go figure.

[felix-lambert]

We are in 2020 and I'm still getting the same error. I'm transfering a lot of data to cloud firestore. It's working at the beginning, then i start having some DEADLINE Exceeded error on some items that are being added

[hbgxsm95]

We are in 2020 and I'm still getting the same error. I'm transfering a lot of data to cloud firestore. It's working at the beginning, then i start having some DEADLINE Exceeded error on some items that are being added

@voold Can you send timestamps and project ID to peiqinzhao(at)google.com? Thanks!

[ramzauchenna]

We are in 2020 and I'm still getting the same error. I'm transfering a lot of data to cloud firestore. It's working at the beginning, then i start having some DEADLINE Exceeded error on some items that are being added

Exactly. Still having this issue. its making me do away with real time sync and batch all my requests as i have to always restart my server to fix this. I hope i dont have this issue after doing this

[hbgxsm95]

We are in 2020 and I'm still getting the same error. I'm transfering a lot of data to cloud firestore. It's working at the beginning, then i start having some DEADLINE Exceeded error on some items that are being added

Exactly. Still having this issue. its making me do away with real time sync and batch all my requests as i have to always restart my server to fix this. I hope i dont have this issue after doing this

Hi @ramzauchenna Can you send timestamps, project ID and log error information to peiqinzhao(at)google.com? Thanks!

[ashking]

I've about 600 documents regularly queried via cloud-functions. I do get
DEADLINE_EXCEEDED errors often. The frequency of seeing this error has increased lately.

The version of packages I use in cloud-functions

"dependencies": {
    "firebase-admin": "~7.0.0",
    "firebase-functions": "^3.2.0",
}

Would upgrading the libraries make the situation any better? thanks
(cc @schmidt-sebastian @hbgxsm95)

[pkwiatkowski0]

@ashking I'm using
"firebase-admin": "~8.8.0",
"firebase-functions": "~3.3.0"
and still have this problem

[schmidt-sebastian]

I would like to clarify something that I personally wasn't aware of - increasing the request timeouts well past 60 seconds actually will not have the desired effect. 60 seconds is currently the maximum timeout supported by the backed, so the only other solution is to reduce your response size.

[ashking]

@pkwiatkowski0 , I made this error go away.

After a lot of testing and educated guesses on the code developed, here is what made the fix go away -

When the CF is triggered, we were calling a few async calls to perform some business operations. In one of the asynchronous call we had to fetch a few hundred records and that's the place where we were always getting DEADLINE_EXCEEDED errors. After having some console.time() around the documents fetch it was seen that block of code was taking about 100s of seconds to fetch a mere 200 records from the collection. Though the function execution time in total was <100ms. Strange. Strange. Strange...

We made the calls synchronous by having async/await and waited for the entire execution to complete before exiting the function. This greatly reduced the time it took to fetch 200 records from 100s of seconds to <1000ms. Post this we have not seen the DEADLINE_EXCEEDED anymore and code is stable. Hope this helps.

I guess the fire-and-forget nature of the code blocks in the CF puts the entire function on some throttle mode and degrades the performance.

[topcatcreatives]

I had the same error where I only was only reading and writing small amounts of records (not even a few hundred) per firestore get()/set(). I discovered for me the issue was I had too many operations happening in parallel as i was performing these operations nested within a for loop. Increasing timeout and memory did not help.

To resolve the error (similar to @ashking resolution) I updated my code to handle the firestore database operation callbacks using async/wait. I effectively removed any parallel firestore operations. This is working really well and no more errors, and things happen quickly which is good as I was concerned I would then hit the timeout limit if things happens one at a time. But seems faster if anything.

Hope this helps anyone stuck on a similar issue.

[yogskumar]

@pkwiatkowski0 , I made this error go away.

After a lot of testing and educated guesses on the code developed, here is what made the fix go away -

When the CF is triggered, we were calling a few async calls to perform some business operations. In one of the asynchronous call we had to fetch a few hundred records and that's the place where we were always getting DEADLINE_EXCEEDED errors. After having some console.time() around the documents fetch it was seen that block of code was taking about 100s of seconds to fetch a mere 200 records from the collection. Though the function execution time in total was <100ms. Strange. Strange. Strange...

We made the calls synchronous by having async/await and waited for the entire execution to complete before exiting the function. This greatly reduced the time it took to fetch 200 records from 100s of seconds to <1000ms. Post this we have not seen the DEADLINE_EXCEEDED anymore and code is stable. Hope this helps.

I guess the fire-and-forget nature of the code blocks in the CF puts the entire function on some throttle mode and degrades the performance.

It's not working. Even I try it multiple times

[ashking]

@Yogeshkumar9291 , Can you isolate and share a sample snippet of your cloud function?

[yogskumar]

.limit(50000);

Hi , here i mentioned the code as per your suggestion. Still facing the same error

let collectionReference = db.collection('userdatabog');
    collectionReference
        .where("date", ">", "20200315")
        .limit(500)
        .get()
        .then(snapshot => {
            dataList = snapshot.docs.map((documentSnapshot) => {
                console.log("Noti Data " + documentSnapshot.data())
                return documentSnapshot.data();
            });
            console.log("DataList  " + JSON.stringify(dataList))
        })

[ashking]

@Yogeshkumar9291 this is the exact issue I had. using .then I could not control the exit of the function before the promise completed succcessfully. I moved it to async/await structure to make sure that my functions don't exit before promises are resolved and that solved my issue. I suggest you give it a try.

let collectionReference = db.collection('userdatabog').where("date", ">", "20200315").limit(500)

const snapshot = await collectionReference.get()

const dataList = snapshot.docs.map((documentSnapshot) => {
    console.log("Noti Data " + documentSnapshot.data())
    return documentSnapshot.data();
});
console.log("DataList  " + JSON.stringify(dataList))

[weilies]

i have extended my timeout to 540sec, but yet i hit the deadline exceed issue about a min of the program run

here my CF snippet

const runtimeOpts = {
  timeoutSeconds: 540,
//   memory: '1GB'  --> deployment error if defined
}
...
export const GCPSpeech2TranscriptLRR = functions.runWith(runtimeOpts).https.onRequest((req, res) => {

    cors(req, res, async () => {

        const gcsUri = `${storageServer}/${req.body.data.fullPath}`

        const encoding = 'FLAC';
        const languageCode = 'ms-MY'

        const config = {
            encoding: encoding,
            languageCode: languageCode,
            audioChannelCount: 2, // hit 'Invalid audio channel count' if not specify
            enableAutomaticPunctuation: true, // https://cloud.google.com/speech-to-text/docs/automatic-punctuation
        };
        const audio = {
            uri: gcsUri,
        };
        const request = {
            audio: audio,
            config: config,
        };

        // Detects speech in the audio file
        const [operation] = await client.longRunningRecognize(request);
        const [response] = await operation.promise();

        // resolve src/index.ts:175:18 - error TS7006: Parameter 'result' implicitly has an 'any' type.
        const transcription = response.results
            .map((result:any) => result.alternatives[0].transcript)
            .join('\n');
        res.status(200).send({ data: { status: true, message: transcription } })
    })
})

error msg in Chrome Console

Subscriber.js:192 Error: deadline-exceeded
    at new HttpsErrorImpl (index.cjs.js:58)
    at index.cjs.js:373
    at ZoneDelegate.push../node_modules/zone.js/dist/zone.js.ZoneDelegate.invokeTask (zone.js:421)
    at Zone.push../node_modules/zone.js/dist/zone.js.Zone.runTask (zone.js:188)
    at push../node_modules/zone.js/dist/zone.js.ZoneTask.invokeTask (zone.js:503)
    at ZoneTask.invoke (zone.js:492)
    at timer (zone.js:3034)

[rubenheymans]

i have extended my timeout to 540sec, but yet i hit the deadline exceed issue about a min of the program run

here my CF snippet

const runtimeOpts = {
  timeoutSeconds: 540,
//   memory: '1GB'  --> deployment error if defined
}
...
export const GCPSpeech2TranscriptLRR = functions.runWith(runtimeOpts).https.onRequest((req, res) => {

    cors(req, res, async () => {

        const gcsUri = `${storageServer}/${req.body.data.fullPath}`

        const encoding = 'FLAC';
        const languageCode = 'ms-MY'

        const config = {
            encoding: encoding,
            languageCode: languageCode,
            audioChannelCount: 2, // hit 'Invalid audio channel count' if not specify
            enableAutomaticPunctuation: true, // https://cloud.google.com/speech-to-text/docs/automatic-punctuation
        };
        const audio = {
            uri: gcsUri,
        };
        const request = {
            audio: audio,
            config: config,
        };

        // Detects speech in the audio file
        const [operation] = await client.longRunningRecognize(request);
        const [response] = await operation.promise();

        // resolve src/index.ts:175:18 - error TS7006: Parameter 'result' implicitly has an 'any' type.
        const transcription = response.results
            .map((result:any) => result.alternatives[0].transcript)
            .join('\n');
        res.status(200).send({ data: { status: true, message: transcription } })
    })
})

error msg in Chrome Console

Subscriber.js:192 Error: deadline-exceeded
    at new HttpsErrorImpl (index.cjs.js:58)
    at index.cjs.js:373
    at ZoneDelegate.push../node_modules/zone.js/dist/zone.js.ZoneDelegate.invokeTask (zone.js:421)
    at Zone.push../node_modules/zone.js/dist/zone.js.Zone.runTask (zone.js:188)
    at push../node_modules/zone.js/dist/zone.js.ZoneTask.invokeTask (zone.js:503)
    at ZoneTask.invoke (zone.js:492)
    at timer (zone.js:3034)

Well at least I can fix your: deployment error if defined

// tslint:disable-next-line:no-duplicate-imports import { VALID_MEMORY_OPTIONS } from 'firebase-functions';
memory: VALID_MEMORY_OPTIONS[3]

[yogskumar]

@Yogeshkumar9291 this is the exact issue I had. using .then I could not control the exit of the function before the promise completed succcessfully. I moved it to async/await structure to make sure that my functions don't exit before promises are resolved and that solved my issue. I suggest you give it a try.

let collectionReference = db.collection('userdatabog').where("date", ">", "20200315").limit(500)

const snapshot = await collectionReference.get()

const dataList = snapshot.docs.map((documentSnapshot) => {
    console.log("Noti Data " + documentSnapshot.data())
    return documentSnapshot.data();
});
console.log("DataList  " + JSON.stringify(dataList))

It isn't working..

[weilies]

i have Dough who from Firebase shared that i should call onCall instead of onRequest to use callable function
https://stackoverflow.com/questions/62587762/firebase-cloud-function-set-timeout-540s-but-ended-at-60s

But yet it still doesn't resolve my issue. Strange...

[JoshBot-Debug]

Same here August of 2020 and I'm getting the DEADLINE_EXCEEDED error.

[dobromyslov]

Same error when saving a batch of 100 writes.

[micadasystems]

I have the same error when using firebase-admin in an Electron app + Vue 2.

vue.runtime.esm.js?2b0e:1888 TypeError: Cannot read property 'DEADLINE_EXCEEDED' of undefined
    at eval (gax.js?359a:355)
    at Array.map (<anonymous>)
    at constructRetry (gax.js?359a:354)
    at Object.constructSettings (gax.js?359a:475)
    at eval (util.js?cf9e:21)
    at Object../node_modules/@google-cloud/firestore/build/src/util.js (chunk-vendors.js:406)
    at __webpack_require__ (app.js:849)
    at fn (app.js:151)
    at eval (validate.js?ae2a:19)
    at Object../node_modules/@google-cloud/firestore/build/src/validate.js (chunk-vendors.js:511)
logError @ vue.runtime.esm.js?2b0e:1888
globalHandleError @ vue.runtime.esm.js?2b0e:1879
handleError @ vue.runtime.esm.js?2b0e:1839
eval @ vue.runtime.esm.js?2b0e:1856
Promise.catch (async)
invokeWithErrorHandling @ vue.runtime.esm.js?2b0e:1856
callHook @ vue.runtime.esm.js?2b0e:4219
insert @ vue.runtime.esm.js?2b0e:3139
invokeInsertHook @ vue.runtime.esm.js?2b0e:6346
patch @ vue.runtime.esm.js?2b0e:6565
Vue._update @ vue.runtime.esm.js?2b0e:3945
updateComponent @ vue.runtime.esm.js?2b0e:4066
get @ vue.runtime.esm.js?2b0e:4479
Watcher @ vue.runtime.esm.js?2b0e:4468
mountComponent @ vue.runtime.esm.js?2b0e:4073
Vue.$mount @ vue.runtime.esm.js?2b0e:8415
eval @ main.js?56d7:6
./src/main.js @ app.js:1159
__webpack_require__ @ app.js:849
fn @ app.js:151
1 @ app.js:1172
__webpack_require__ @ app.js:849
checkDeferredModules @ app.js:46
(anonymous) @ app.js:925
(anonymous) @ app.js:928

My Firestore test collection has only 1 document with 1 field.

    const db = firebase.firestore()
    const ref = db.collection('countries');

    ref.get().then((res) => {
      console.log(res)
    })  

The same code running in a standalone nodejs app runs fine, so in my case prob has to do with Electron.

[matetukacs]

You may be able to work around your issue if you start by issuing a DocumentReference.get() call. We retry those automatically if the connection establishment takes a bit longer than it should.

@schmidt-sebastian Could I issue alistCollections() command instead? Does that also get retried automatically? We are running on kubernetes, and I'm thinking I could use listCollections as a sort of ping, which I could use in the readiness check to make sure kubernetes does not send requests to the process before it completes. What do you think?

[schmidt-sebastian]

listCollections() also gets retried automatically (and read-only operation does).

[MorenoMdz]

I had the same error where I only was only reading and writing small amounts of records (not even a few hundred) per firestore get()/set(). I discovered for me the issue was I had too many operations happening in parallel as i was performing these operations nested within a for loop. Increasing timeout and memory did not help.

To resolve the error (similar to @ashking resolution) I updated my code to handle the firestore database operation callbacks using async/wait. I effectively removed any parallel firestore operations. This is working really well and no more errors, and things happen quickly which is good as I was concerned I would then hit the timeout limit if things happens one at a time. But seems faster if anything.

Hope this helps anyone stuck on a similar issue.

Would you have an example of your code? Right now I am randomly having this issue where the DB won't save the subcollection items (here the Assignments), in this case we have a 1 year dates lock so it never gets past 400 documents, yet it is triggering the error, sometimes the DB will retry and create the documents minutes later, sometimes it wont.

await Promise.all( challenge.totalUser.map(async user => await assignToUser(user)) );

[superzadeh]

Chiming in as we're also having a lot of DEADLINE_EXCEEDED. We are running similar code what was mentioned in this this thread.

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/src/call.ts:81:24) at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client.ts:570:32) at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48) at /workspace/node_modules/@grpc/grpc-js/src/call-stream.ts:249:24 at processTicksAndRejections (internal/process/task_queues.js:79:11) Caused by: Error: at Firestore.getAll (/workspace/node_modules/@google-cloud/firestore/build/src/index.js:808:23) at DocumentReference.get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:208:32) at DefaultFirestoreService.getUserData (/workspace/src/services/FirestoreService.ts:605:32) at processTicksAndRejections (internal/process/task_queues.js:97:5) at DefaultFirestoreService.updateUserTimeline (/workspace/src/services/FirestoreService.ts:667:22) at CronController.timelineUpdateForUser (/workspace/src/controllers/CronController.ts:173:7)

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/src/call.ts:81:24) at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client.ts:570:32) at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48) at /workspace/node_modules/@grpc/grpc-js/src/call-stream.ts:249:24 at processTicksAndRejections (internal/process/task_queues.js:79:11) Caused by: Error: at Firestore.getAll (/workspace/node_modules/@google-cloud/firestore/build/src/index.js:808:23) at DocumentReference.get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:208:32) at DefaultFirestoreService.getUserData (/workspace/src/services/FirestoreService.ts:605:32) at processTicksAndRejections (internal/process/task_queues.js:97:5) at DefaultCronService.reminderNotificationForUser (/workspace/src/services/CronService.ts:208:22) at CronController.reminderNotificationForUser (/workspace/src/controllers/CronController.ts:142:7)

We are getting this error when fetching a document that isn't particularly large (below 1KB):

// this.firestoreClient is a Firestore() instance
const users = await this.firestoreClient.collection(collections.USERS).doc(userId).get()

It is happening very consistently but in an unpredictable way, and is wreaking total havoc in production. We see it most often in parts of the code where we are writing/reading in parallel/very close timespan from Firestore. Is there any solution available to date, or should we just migrate to another DB that can support our load (which isn't really high, we're never doing more than ~100 read/writes per second)? Seeing that the issue was opened a couple years ago, and appears to still be happening, we're not very confident on being able to get this resolved at all with Firestore. Moving everything to another database is a significant amount of work, that we didn't plan for, so having some sort of honest transparency from the team working on this would help a lot in making sure we make the right decision.

We are also facing very similar errors with Cloud Tasks, who is using the same GRPC library; and was wondering if the issues are not related somehow, and even if you pointed out several time that the GRPC lib is only doing its job, it seems evident that there is something wrong happening on that layer (or on how both clients -- Firestore and Cloud Tasks -- are using it).

PS: increasing the DEADLINE is not an option, the code is already running slow and failing; making it even slower is not viable workaround for production environments. Our users notice and complain about performance all the time already.

[MorenoMdz]

@superzadeh you sure there isn't anything else around that fetch query that might be looping over and increasing the function execution time? The issues I had were solved by splitting a few iteration-heavy functions into smaller ones that happen after some trigger events.

[superzadeh]

Thanks for the suggestion @MorenoMdz, our call site is only making calls to Firestore with async/await "sequentially" (no Promise.all or loops). This is a endpoint called by a CRON job, and working on our users collection one by one: each user has a CRON creating a Cloud Task for them, and our service is processing them one by one. We have 5 instances running, and the queue is capped at 200 max concurrent jobs, which more less means about 40 simultaneous jobs running per service instance (running on AppEngine Flex Environment, 2 CPUs and 6GB RAM). This is by far nothing amazing/pushing the limits in terms of scale, but enough to break Firestore?

Here's the code from our one of our functions calling the getUserData that ends up crashing with DEADLINE_EXCEEDED in the stack traces I shared earlier. Is there anything here we're doing wrong?

  // FirestoreService.ts
  async updateUserTimeline(userId: string): Promise<void> {

    // DEADLINE_EXCEEDED is thrown by this call 👇 
    const userData = await this.getUserData(userId)
    const tipOfTheDay = userData.tipOfTheDay
    //...
  }

  async getUserData(userId: string): Promise<firestore.DocumentData> {
    const user = await this.firestoreClient.collection(collections.USERS).doc(userId).get()
    return user.data()
  }

This code itself is called directly form an Express controller handling the route for the CRON job:

  // CronController.ts
  public timelineUpdateForUser = async (
    req: Request,
    res: Response,
    next: NextFunction,
  ) => {
    try {
      const userId = req.params.id
      await this.firestoreService.updateUserTimeline(userId)
      res.status(200).end()
    } catch (error) {
      this.loggingService.error(error)
      next(error)
    }
  }

Results in this:

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded
   at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/src/call.ts:81:24)
   at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client.ts:570:32)
   at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
   at /workspace/node_modules/@grpc/grpc-js/src/call-stream.ts:249:24
   at processTicksAndRejections (internal/process/task_queues.js:79:11)
Caused by: Error: 
   at Firestore.getAll (/workspace/node_modules/@google-cloud/firestore/build/src/index.js:808:23)
   at DocumentReference.get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:208:32)
   at DefaultFirestoreService.getUserData (/workspace/src/services/FirestoreService.ts:605:32)
   at processTicksAndRejections (internal/process/task_queues.js:97:5)
   at DefaultFirestoreService.updateUserTimeline (/workspace/src/services/FirestoreService.ts:667:22)
   at CronController.timelineUpdateForUser (/workspace/src/controllers/CronController.ts:173:7)

[superzadeh]

Any update or workarounds we could try? We are still having this issue happening frequently (every hour at least) in production. I'm also wondering whether the issue should re-openned (or I'm happy to open a new one too).

[ramzauchenna]

The workaround i used was to convert it to a batch commit rather than one at a time limited to 450 records in each batch. Never had the issue again.

[superzadeh]

Thanks @ramzauchenna, are you referring to batched writes or something else?
The issue in our case is happening on reads, which don’t appear to have an option to be batched using the Firestore SDK.

[schmidt-sebastian]

@superzadeh Are you only reading a single document? Are you otherwise able to use Firestore?

[superzadeh]

@superzadeh Are you only reading a single document? Are you otherwise able to use Firestore?

No, we are reading multiple documents (a different one per HTTP request that comes in). The DEADLINE_EXCEEDED errors only happen during peak traffic, otherwise Firestore works as expected.

We were wondering if the soft limit of 1 write per second could be causing this issue? We are only writing to different documents in parallel to these reads, however each document is within the subcollection of a document.

Ex:

- users collection
   +-- foouser_doc_id
          +-- Timeline Subcollection
                 +-- document 1
                 +-- document 2
   +-- baruser_doc_id
          +-- Timeline Subcollection
                 +-- document 1
                 +-- document 2

We write only once in document 1 and document 2, however since they are both in a subcollection of the user document baruser_doc_id, could that count and triggers the soft limit of 1 write/second on the baruser_doc_id document?