Nodejs-pubsub: Ack fails, with "Failed to acknowledge" or Failed to "modifyAckDeadline"

[gberth]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

• Search the issues already opened: https://github.com/googleapis/nodejs-pubsub/issues
• Search the issues on our "catch-all" repository: https://github.com/googleapis/google-cloud-node
• Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js

If you are still having issues, please be sure to include as much information as possible:

Environment details

• OS: Ubuntu 18.04
• Node.js version: 11.15
• npm version: 6.11.2
• @google-cloud/pubsub version: 32.0

Steps to reproduce

1. Use version 0.31 or 0.32

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

80 % of messages fails to ack. Going back to version 0.22.2 the Ack errors disappears.

[callmehiphop]

@gberth I'm sorry you're having an issue! Would you be able to provide the entire error you're seeing? Usually there's something more specific about the failure, like DEADLINE_EXCEEDED.

[gberth]

{ BatchError: Failed to "modifyAckDeadline" for 1 message(s). Reason: 4 DEADLINE_EXCEEDED: Deadline Exceeded

Sometimes "Bandwitdh exhausted". 80 % only that ack fails, the rest goes ok (at least as it seems). Switching to 0.22.2 - no more ack errors, and the subscription behaves almost normally.

In the current running application there is done an immediate Ack after receive (not crusial if a message is lost)

[callmehiphop]

@gberth thanks! You might try increasing the request timeout for acks/modacks. You can do it like so

const sub = topic.subscription('my-sub', {
  batching: {
    callOptions: {
      timeout: 60000
    }
  }
});

It might also be worth noting that other users have reported similar issues when they create a lot of Subscription objects. The gRPC channel can only handle so many concurrent streams, so if you do have a lot of Subscription objects, I might suggest trying to either cache and re-use them where possible OR lower the number of streams created per Subscription object.

const sub = topic.subscription('my-sub', {
  streamingOptions: {
    maxStreams: 1
  }
});

[gberth]

Hi and Thanks for follow up.

There is (was) only one stream for this subscription. Only two subscriptions on the actual topic. The topic receives about 20-50 rather large messages a second. Don't think timeout will help, and the reason I think it is some sort of bug is that version 0.22.2 works fine, while 0.31 and 0.32 does not.

Noteworthy also is that the Node pubsub adapter (v 0.31-0.32) regularly hangs (seemingly indefinitely) and when reinitiated without terminating the running process (with new Pubsub) in a hang situation there is a Node warning about memory leak due to emitters.

Best regards
Gunnar

[callmehiphop]

@gberth interesting, would you be able to provide us with a small reproduction case that triggers the hanging?

[gberth]

I'm sorry but can't find any regularity. But during the time from 2019-09-12 12:14:06 to 2019-09-13 13:10:30 I had 72 observations of hang. That is, I know there are 20-50 messages coming every second, I monitored the count every minute, and if nothing had changed for a minute, I closed the subscription, created a new Pubsub object and started listening. (and got the warning from Node about possible memory leak). I also tried to remove the listener and create a new listners, but that did not have any effect. The problem all the time was that the Ack did not work. Had retention time on 10 minutes and google obviously had a lot to clean up continuously.

All these problems are gone when running 0.22.2. (and my feeling is that it is closely related to the ack problems - but have been mistaken before ;-) )

G

[gberth]

Fyi: Deployed app to google cloud, and ack errors are gone. Errors above occurs when running a local workstation on version 1.0

[callmehiphop]

@gberth interesting, are you by chance using a proxy on your local machine?

[gberth]

No, native Node with project reference. As errors disappeared after deploying to the cloud, I think you can close this, and keep it as a reference that something is different (probably some network thing) between 0.22 and 1.0.

Regards
Gunnar

[glesage]

@gberth thanks! You might try increasing the request timeout for acks/modacks. You can do it like so

const sub = topic.subscription('my-sub', {
  batching: {
    callOptions: {
      timeout: 60000
    }
  }
});

Hi there, is there some information anywhere about which setting this batching.callOptions.timeout affects? What is the default value and what is it's effect?

eg in this file I see lots of settings but not sure which it applies to https://github.com/googleapis/nodejs-pubsub/blob/master/src/v1/subscriber_client_config.json