Nodejs-pubsub: Error: 13 INTERNAL: Received RST_STREAM with error code 2

[ThomWright]

After upgrading to 0.23.0 (from 0.21.1) I see my subscriber services connecting to pubsub, receiving a few messages, then some time later experiencing the following error:

Error: 13 INTERNAL: Received RST_STREAM with error code 2
    at Object.exports.createStatusError (/app/node_modules/google-gax/node_modules/grpc/src/common.js:91:15)
    at ClientDuplexStream._emitStatusIfDone (/app/node_modules/google-gax/node_modules/grpc/src/client.js:233:26)
    at ClientDuplexStream._receiveStatus (/app/node_modules/google-gax/node_modules/grpc/src/client.js:211:8)
    at Object.onReceiveStatus (/app/node_modules/google-gax/node_modules/grpc/src/client_interceptors.js:1306:15)
    at InterceptingListener._callNext (/app/node_modules/google-gax/node_modules/grpc/src/client_interceptors.js:568:42)
    at InterceptingListener.onReceiveStatus (/app/node_modules/google-gax/node_modules/grpc/src/client_interceptors.js:618:8)
    at /app/node_modules/google-gax/node_modules/grpc/src/client_interceptors.js:1123:18

Here are some example logs and timestamps from service start until the error:

"Subscriptions ready" "2019-01-22T09:39:35.890Z"
"Handling message" "2019-01-22T09:39:42.701Z"
"Handling message" "2019-01-22T09:40:21.545Z"
"Handling message" "2019-01-22T09:40:22.276Z"
"Handling message" "2019-01-22T09:40:59.133Z"
"Handling message" "2019-01-22T09:52:01.244Z"
"Handling message" "2019-01-22T09:52:10.978Z"
"Handling message" "2019-01-22T09:52:11.854Z"
"Handling message" "2019-01-22T09:52:22.544Z"
"Handling message" "2019-01-22T10:03:55.492Z"
"Handling message" "2019-01-22T10:04:03.351Z"
"Handling message" "2019-01-22T10:05:45.136Z"
"Handling message" "2019-01-22T10:06:07.888Z"
"Handling message" "2019-01-22T10:06:17.679Z"
"Error emitted from subscription" "2019-01-22T10:09:42.682Z"

It seems to take about half an hour from a service starting to receiving the error, reasonably consistently I think.

See https://github.com/googleapis/google-cloud-node/issues/2619 for a similar, previous error.

Environment details

• OS: Alpine Linux
• Node.js version: 10.15.0
• npm version: 6.6
• @google-cloud/pubsub version: 0.23.0

Steps to reproduce

1. Use version 0.23.0
2. Wait half an hour?

Sorry I don't have any better repro steps.

[callmehiphop]

@ThomWright thanks for the write up! That error is actually the default error returned from PubSub when they close a stream. The reason it occurs after 30 minutes is because thats when PubSub tends to kill off any streaming connections.

When you see this error do messages continue to come in? Or does the Subscription stop functioning altogether?

[ThomWright]

That's difficult to say, when one of our services detects an error from a subscription it starts failing health checks and gets restarted pretty soon after. Even if messages could continue to arrive, I'd be unlikely to see them.

After rolling back to 0.21.1 I no longer see these errors.

[eezing]

@callmehiphop Subscription stops functioning altogether for me. I've only seen this error on v0.23.0.

[callmehiphop]

I just made a release (0.24.0) that should resolve the issue you're seeing. If you continue to encounter this issue, please let us know! Thanks again for the report!

[ThomWright]

That seems to have fixed it, thanks.

[rthotakura-wp]

I still see the issue happening once a while with 2.5.0 and it does not look like an issue with one GCP service or client.

https://github.com/googleapis/nodejs-firestore/issues/1023