Grpc-node: Error: Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY

[ethanrubio]

Problem description

We currently use @grpc/grpc-js with Envoy in production, and we get this mystifying error Error: Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY occasionally during normal RPCs. We currently only use unary calls and not streams. It might have to do with something when we do a new deployment. Does anyone have any insight on how to debug this error?

Reproduction steps

It's difficult to reproduce because this doesn't happen in our staging environment.

Environment

• alpine linux 3.6
• Node version 10.13
• @grpc/grpc-js 0.3.2

Additional context

Error: Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY
    at Http2CallStream.call.on (/home/node/app/node_modules/@grpc/grpc-js/build/src/client.js:88:45)
    at Http2CallStream.emit (events.js:187:15)
    at Http2CallStream.EventEmitter.emit (domain.js:442:20)
    at Http2CallStream.endCall (/home/node/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:66:18)
    at Http2CallStream.<anonymous> (/home/node/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:337:18)
    at Generator.next (<anonymous>)
    at fulfilled (/home/node/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:4:58)

[murgatroid99]

That looks like a bug in @grpc/grpc-js. That error message probably means that we are not properly handling certain kinds of connection-level errors. The error message is confusing because it's coming from the underlying transport, http2, which doesn't have the unary/stream distinction.

[steve-gray]

I'm seeing a variation on this - after a GOAWAY error, we cannot re-establish new stream requests - the connection attempts to open. but then no data is seemingly received by the listening client. We literally have to process.exit and restart on this scenario.

[murgatroid99]

@steve-gray Have you tried constructing a new Client instead of restarting the process when this happens? I am not suggesting that as a permanent solution to the problem, but it might help to know what happens if you do that.

[jmacedoit]

I'm getting this as well when using firestore which appears to use gRPC. I reported it here: https://github.com/googleapis/nodejs-firestore/issues/673

[redstone99]

I'm also seeing this using a custom datastore module that uses grpc-node.

[murgatroid99]

I have published @grpc/grpc-js version 0.4.3 with a change that should fix this.

[ariofrio]

I'm still seeing this on 0.4.3:

Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY
at Http2CallStream.call.on (/app/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
at Http2CallStream.emit (events.js:187:15)
at Http2CallStream.EventEmitter.emit (domain.js:442:20)
at process.nextTick (/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:71:22)
at process._tickCallback (internal/process/next_tick.js:61:11)"}]

package.json:

"@grpc/grpc-js": "0.4.3",

We are running 10–30 servers with hundreds of Firestore onSnapshot listeners each. Every few minutes, one of the servers crashes with this error.

Any help debugging this?

[matej-svejda]

Also having the same problem with Firestore onSnapshot listeners.

[pworkpop]

We also got this error yesterday in a bunch of code 1 and code 13 errors with Google PubSub 0.30.1 with "@grpc/grpc-js": "0.4.3"
This is probably a particular situation when the server wants to actively drain clients.

[murgatroid99]

Some relevant changes were made in version 0.6. Has anyone encountered this issue with @grpc/grpc-js version 0.6.x?

[matej-svejda]

I've switched to Node 10 and havent had the problem since.

[ankurverma1994]

Hi! I am getting the following error in a Production environment.

Error: Got unhandled rejection {"stack":"Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY\n at ClientHttp2Session.request (internal/http2/core.js:1545:13)\n at ClientHttp2Session.<anonymous> (/MyProject/node_modules/@google-cloud/trace-agent/build/src/plugins/plugin-http2.js:73:32)\n at Subchannel.startCallStream (/MyProject/node_modules/@google-cloud/tasks/node_modules/@grpc/grpc-js/build/src/subchannel.js:326:42)\n at callStream.filterStack.sendMetadata.then.finalMetadata (/MyProject/node_modules/@google-cloud/tasks/node_modules/@grpc/grpc-js/build/src/channel.js:111:51)\n at <anonymous>\n at process._tickCallback (internal/process/next_tick.js:189:7)","message":"New streams cannot be created after receiving a GOAWAY"}
at process.on (index.js:22)
at emitTwo (events.js:126)
at process.emit (events.js:214)
at emitPendingUnhandledRejections (internal/process/promises.js:108)
at process._tickCallback (next_tick.js:190)

Environment details:

"@google-cloud/spanner": "4.2.0"
"@google-cloud/tasks": "1.6.0"
"@google-cloud/trace-agent": "4.2.3",
node: 8.16

We have deployed everything on GKE. Is there any way to avoid this error?
Thanks in advance.

[murgatroid99]

@ankurverma1994 What version of @grpc/grpc-js do you have installed?

[ankurverma1994]

@murgatroid99 I grep for @grpc/grpc-js in packag-lock.json file in prod environment. Haven't explicitly mentioned @grpc/grpc-js in package.json file
Here is the output

"@grpc/grpc-js": {
          "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-0.6.9.tgz",
            "@grpc/grpc-js": "0.6.9",
        "@grpc/grpc-js": {
          "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-0.6.9.tgz",
            "@grpc/grpc-js": "0.6.9",
            "@grpc/grpc-js": "^0.6.9",
        "@grpc/grpc-js": {
          "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-0.6.11.tgz",
            "@grpc/grpc-js": "0.6.9",
            "@grpc/grpc-js": {
              "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-0.6.9.tgz",
        "@grpc/grpc-js": {
          "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-0.6.9.tgz",
            "@grpc/grpc-js": "0.6.9",
    "@grpc/grpc-js": {
      "resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-0.3.6.tgz",
        "@grpc/grpc-js": "^0.3.0",

[murgatroid99]

It looks like you have 3 different versions of the library: 0.3.6, 0.6.9, and 0.6.11. Can you update whichever package depends on version ^0.3.0, to make them more consistent?

[matej-svejda]

Today I got the error again with grpc-js 0.6.9

(node:17871) UnhandledPromiseRejectionWarning: Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY
    at ClientHttp2Session.request (internal/http2/core.js:1556:13)
    at Subchannel.startCallStream (/var/www/nodejs_apps/collab_server_dev/node_modules/@grpc/grpc-js/build/src/subchannel.js:326:42)
    at callStream.filterStack.sendMetadata.then.finalMetadata (/var/www/nodejs_apps/collab_server_dev/node_modules/@grpc/grpc-js/build/src/channel.js:111:51)

Worst thing about this (in connection with firebase): onSnapshot stopped publishing DB changes without even failing...

[murgatroid99]

@grpc/grpc-js version 0.6.13 has a speculative fix for this.

[shaibt]

Hi,

Have been experiencing this error last several days - started with grpc 0.6.9 but also reoccurs with 0.6.13.
Env: Google App Engine Flex Node JS 8.
At some point (2-3 days after service start) my servers (multiple) show this in their logs:

(node:17) UnhandledPromiseRejectionWarning: Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY
at ClientHttp2Session.request (internal/http2/core.js:1424:13)
at Subchannel.startCallStream (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/subchannel.js:326:42)
 at callStream.filterStack.sendMetadata.then.finalMetadata (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/channel.js:111:51)
at <anonymous>
at process._tickDomainCallback (internal/process/next_tick.js:229:7)

And from then on, all future Firestore writes/updates fail. For each write the above error is thrown + my error catching code catches this:

{ 
Error: 4 DEADLINE_EXCEEDED: Deadline exceeded at callErrorFromStatus (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:30:26) 
at Http2CallStream.call.on (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:79:34) 
at emitOne (events.js:121:20)
at Http2CallStream.emit (events.js:211:7) 
at process.nextTick (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22) 
at _combinedTickCallback (internal/process/next_tick.js:132:7) 
at process._tickDomainCallback (internal/process/next_tick.js:219:9)

Only recourse is restarting all effected services.
After seeing @murgatroid99 comment on 0.6.13 I explicitly required that grpc-js version as a dependency but problem still occurred.

[murgatroid99]

You say that you explicitly required the newer version of grpc-js, but can you verify that Firestore is actually using that version? If Firestore has a pinned dependency on grpc-js 0.6.9 then declaring your own separate dependency won't change that.

[shaibt]

Looking at a package-lock.json file on one of the target servers shows:

• @google-cloud/firestore (2.6.0) requires "google-gax": "^1.7.5”
• The actual google-gax version installed is 1.11.1 and it explicitly requires: "@grpc/grpc-js": “0.6.9"

Looking into the the deployed node_modules folder:

• the installed google-gax version is indeed 1.11.1
• It installed a private @grpc-js dependency version 0.6.9

So you’re right - Firestore is still using 0.6.9 although I explicitly required a newer version.

On 16 Dec 2019, at 19:51, Michael Lumish [e-mail] wrote:

You say that you explicitly required the newer version of grpc-js, but can you verify that Firestore is actually using that version? If Firestore has a pinned dependency on grpc-js 0.6.9 then declaring your own separate dependency won't change that.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/grpc/grpc-node/issues/734?email_source=notifications&email_token=AGGCXODQJBWXJEMB7GZF4NDQY65ZNA5CNFSM4GXICT22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG7Q5BA#issuecomment-566169220, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGGCXOH2ZZV2VPULY3OADY3QY65ZNANCNFSM4GXICT2Q.

[murgatroid99]

With no unambiguous reports of this bug on newer versions of the library, I believe this bug has been fixed. If you experience a bug like this, please open a new issue.

[khaderkhan]

restarting the pod on which pubsub was scheduled worked for me