Nodejs-storage: "Maximum call stack size exceeded" on GKE

[lauramorillo]

With versions 2.0.0 and 2.0.3 I'm getting this error when I try to access a bucket from Cloud Storage from a container running in my GKE cluster:

"RangeError: Maximum call stack size exceeded",
" at replaceProjectIdToken (/code/node_modules/@google-cloud/projectify/src/index.ts:27:1)",
" at replaceProjectIdToken (/code/node_modules/@google-cloud/projectify/src/index.ts:36:22)",
" at /code/node_modules/@google-cloud/projectify/src/index.ts:29:42",
" at Array.map ()",
" at replaceProjectIdToken (/code/node_modules/@google-cloud/projectify/src/index.ts:29:33)",
" at replaceProjectIdToken (/code/node_modules/@google-cloud/projectify/src/index.ts:36:22)",
" at replaceProjectIdToken (/code/node_modules/@google-cloud/projectify/src/index.ts:36:22)",
" at /code/node_modules/@google-cloud/projectify/src/index.ts:29:42",

I downgraded to version 1.7.0 and it works now.
The nodejs-logging project had a similar issue created and resolved a few months ago https://github.com/googleapis/nodejs-logging/pull/87 (in case it can help)

Thank you!

[adriaanmeuris]

We're having this issue as well since upgrading to >= 2.0.0. Reverting to 1.7.0 solved this for us as well.

Capture of the exception we see:

[chislin]

Is there any workaround besides of downgrading?

[jkwlui]

We believe to have fixed a similar issue (#489). Please try it out - https://github.com/googleapis/nodejs-storage/issues/489#issuecomment-434465217

[igorivaniuk]

Have same error
@google-cloud/storage - 2.3.0
@google-cloud/common - 0.26.1
@google-cloud/projectify - 0.3.1

After set resumable: true it start work

Problem only on prod server, i can`t reproduse on local machine

It fails when value is

{ 'Content-Type': 'image/jpeg', body: [DestroyableTransform] }

Start stack

at replaceProjectIdToken (/usr/src/app/node_modules/@google-cloud/projectify/build/src/index.js:30:23)
    at replaceProjectIdToken (/usr/src/app/node_modules/@google-cloud/projectify/build/src/index.js:37:30)
    at replaceProjectIdToken (/usr/src/app/node_modules/@google-cloud/projectify/build/src/index.js:37:30)
    at value.map.v (/usr/src/app/node_modules/@google-cloud/projectify/build/src/index.js:30:32)
    at Array.map (<anonymous>)
    at replaceProjectIdToken (/usr/src/app/node_modules/@google-cloud/projectify/build/src/index.js:30:23)
    at replaceProjectIdToken (/usr/src/app/node_modules/@google-cloud/projectify/build/src/index.js:37:30)
    at replaceProjectIdToken (/usr/src/app/node_modules/@google-cloud/projectify/build/src/index.js:37:30)
    at Object.replaceProjectIdToken (/usr/src/app/node_modules/@google-cloud/projectify/build/src/index.js:37:30)
    at reqOpts.multipart.reqOpts.multipart.map.part (/usr/src/app/node_modules/@google-cloud/common/build/src/util.js:444:37)
    at Array.map (<anonymous>)
    at Util.decorateRequest (/usr/src/app/node_modules/@google-cloud/common/build/src/util.js:443:51)
    at onAuthenticated (/usr/src/app/node_modules/@google-cloud/common/build/src/util.js:312:34)
    at authClient.authorizeRequest.then.res (/usr/src/app/node_modules/@google-cloud/common/build/src/util.js:350:21)
    at process._tickCallback (internal/process/next_tick.js:68:7)

[jkwlui]

Looks like a related issue with replacing project id token where it shouldn't be looking - in a Stream object.

[jkwlui]

Hi @igorivaniuk, can you attach a reproduction case, even if it isn't reproducible locally?

[igorivaniuk]

Hi @kinwa91 I investigate deeply and found cause.

Only on prod we run Sentry which use domains for track context, that's why it works on local machine and on tests.

Upload start after user request and DestroyableTransform has property domain which contains members of IncomingMessage and ServerResponse.

IncomingMessage to has property domain which contains IncomingMessage and ServerResponse.

In this place we go to Infinity loop.

Some debug logs

{ 'Content-Type': 'image/jpeg', body: [DestroyableTransform] }
>>>>replaceProjectIdToken value: { 'Content-Type': 'image/jpeg', body: [DestroyableTransform] }
loop key: Content-Type
>>>>replaceProjectIdToken value: 'image/jpeg'
loop key: body
>>>>replaceProjectIdToken value: DestroyableTransform {
  _readableState: [ReadableState],
  readable: true,
  domain: [Domain],
  _events: [Object],
  _eventsCount: 6,
  _maxListeners: undefined,
  _writableState: [WritableState],
  writable: true,
  allowHalfOpen: true,
  _transformState: [Object],
  _destroyed: false,
  _transform: [Function: noop] }

loop key: domain
>>>>replaceProjectIdToken value: Domain {
  domain: null,
  _events: [Object],
  _eventsCount: 3,
  _maxListeners: undefined,
  members: [Array],
  sentryContext: [Object] }

loop key: members
>>>>replaceProjectIdToken value: [ [IncomingMessage], [ServerResponse] ]
>>>>replaceProjectIdToken value: IncomingMessage {
  _readableState: [ReadableState],
  readable: false,
  domain: [Domain],
  _events: {},
  _eventsCount: 0,
  _maxListeners: undefined,
  socket: [Socket],
  connection: [Socket],
  httpVersionMajor: 1,
  httpVersionMinor: 1,
  httpVersion: '1.1',
  aborted: false,
  upgrade: false,
.... }

loop key: domain
>>>>replaceProjectIdToken value: Domain {
  domain: null,
  _events: [Object],
  _eventsCount: 3,
  _maxListeners: undefined,
  members: [Array],
  sentryContext: [Object] }

loop key: members
>>>>replaceProjectIdToken value: [ [IncomingMessage], [ServerResponse] ]
>>>>replaceProjectIdToken value: IncomingMessage {
  _readableState: [ReadableState],
  readable: false,
  domain: [Domain],
  _events: {},
  _eventsCount: 0,
  _maxListeners: undefined,
  socket: [Socket],
  connection: [Socket],
  httpVersionMajor: 1,
  httpVersionMinor: 1,
  httpVersion: '1.1',
  aborted: false,
  upgrade: false,
.... }

[igorivaniuk]

I try your fix !(value instanceof Stream) and it works.

https://github.com/googleapis/nodejs-projectify/pull/53

[stephenplusplus]

That's great news 👍 I think we're ready to close this issue, then. The update to @google-cloud/projectify was released. If there's any work still remaining, please let me know!

[adriaanmeuris]

happy to report this is resolved for us as well in2.3.1 👍