Firebase-admin-node: onSnapshot listener dies after some time with "Error 10: ABORTED"

[StoryStar]

Describe your environment

• Operating System version: Debian 9
• Node.js version: 10.15.2
• npm version: 6.4.1
• Firebase SDK version: @google-cloud/firestore version: 0.15.4, firebase-admin: 5.13.1
• Firebase Product: database

Describe the problem

After some time alive, a onSnapshot listener dies with 'Error: 10 ABORTED' code. Similar to this issue in the .NET library: https://github.com/googleapis/google-cloud-dotnet/issues/2721

Steps to reproduce:

Start onSnapshot listener on collection with few or no documents (documents are periodically added and deleted)
Leave it for a week or so, works great,
Error: 10 ABORTED: The operation was aborted

Relevant Code:

admin.firestore().collection("search").where("response", '==', null).onSnapshot(this.handleSnapshot)

Error message

Error: 10 ABORTED: The operation was aborted. at Object.exports.createStatusError (node_modules/grpc/src/common.js:87:15) at ClientDuplexStream._emitStatusIfDone (node_modules/grpc/src/client.js:235:26) at ClientDuplexStream._receiveStatus (node_modules/grpc/src/client.js:213:8) at Object.onReceiveStatus (node_modules/grpc/src/client_interceptors.js:1290:15) at InterceptingListener._callNext (node_modules/grpc/src/client_interceptors.js:564:42) at InterceptingListener.onReceiveStatus (node_modules/grpc/src/client_interceptors.js:614:8) at node_modules/grpc/src/client_interceptors.js:1110:18 code: 10, metadata: Metadata { _internal_repr: { 'content-disposition': [Array] } }, details: 'The operation was aborted.' }

It is difficult to reliably reproduce due to the amount of time it takes before the error shows up, however has occurred 3 times in my setup over the past few weeks. In the issue I linked regarding the .NET library, they seemed to resolve it by adding the error code to the list of server statuses to retry. Is that a possible solution here as well?

[schmidt-sebastian]

Hello @StoryStar! Thanks for filing this issue. Like the C# SDK, we can update the Node SDK to retry on ABORTED errors, which should allow us to re-connect your Listen streams. We do, however, want to add a cap on the maximum number of retries before we roll this out. I will update this issue once the change lands in the @google-cloud/firestore repo.

[fabiogmartins]

Hi guys.
Any news?

I have the same situation.

Tks,

[schmidt-sebastian]

@fabiogmartins Sorry for the delay. We have a PR for this fix pending and will push it as part of the next release.

[maxim-xu]

Hi @schmidt-sebastian, would this PR also fix the retry on a normal snapshot .get()? I've been experiencing the same abort exception more often recently... Thank you!

[schmidt-sebastian]

Unfortunately, it doesn't. Our get() retries are handled by a different configuration that is shared among all Firestore clients. I'll reach out to the backend team and see if we can retry on ABORTED for all clients.