Google-cloud-node: Pub/Sub Deadline Exceeded

[dennismartensson]

Environment details

• OS: Docker in GKE
• Node.js version: 6.9
• google-cloud-node version: "google-cloud": "^0.52.0",

Steps to reproduce

We are publishing messages on a number of topics, We hade memory leak problems if calling publish every time we got a message with would put it at around 1000 times per second. So we implemented a batching where we publish 225 messages in every batch. Is do solve the memory problem but we have started to get Error: Deadline Exceeded with the later versions.

{
code: 4, 
metadata: {
_internal_repr: {}
}, 
note: Exception occurred in retry method that was not classified as transient
}

/usr/src/app/node_modules/grpc/src/node/src/client.js in null.<anonymous> at line 434:17
              code: grpc.status.INTERNAL,
              details: 'Failed to parse server response'
            };
          }
        }
      }
      if (status.code !== grpc.status.OK) {
        error = new Error(status.details);
        error.code = status.code;
        error.metadata = status.metadata;
        args.callback(error);
      } else {
        args.callback(null, deserialized);
      }
      emitter.emit('status', status);

Cant figure out way its started to be thrown more often only change is the version.

Let me know if we can provide anymore information to get this resolved.

Thanks!

[stephenplusplus]

@jmuk @lukesneeringer this happened since we switched to GAX. Is there a better default timeout or similar value GAX should set?

EDIT: Ignore this! I was mixing Pub/Sub with Logging. We haven't switched to GAX yet.

[dennismartensson]

Is this a timeout you need to set our can we do some thing to mitigate away from it? Is it in some way created by our strategy to bundle up messages? or is that a good strategy?

[stephenplusplus]

I'm sorry, I misspoke above. We currently don't use the google-gax transport layer, but we intend to soon. That would mean that each API call would allow you to customize the timeout and other low-level settings.

For now, the only way to do this would require a change to our code, allowing a timeout option. We actually just had this come up for ack() in #2042. Would you be open to send a PR with basically the same code, just in the publish() method? No worries if not, we will get to it eventually.

[dennismartensson]

There is now a PR for this in #2232

[stephenplusplus]

This has been released in @google-cloud/[email protected] and [email protected].

[alejomendoza]

I still get this error, I'm publishing a lot of messages at once (more than 1000 per second), not sure if it throws the error because of this. I'm using pubsub V0.13.0 Thanks!

[g-ericso]

I am also seeing the same error without much traffic.