Amazon-ssm-agent: Avoid thundering herd on calls to "UpdateInstanceInformation"

[emil2k]

When one starts up many instances (25+) at once which all have an SSM agent start at boot, causes a thundering herd problem on the UpdateInstanceInformation endpoint resulting in ThrottlingExceptions being thrown repeatedly. The problem gets worse and worse with the more instance one starts at the same time, when starting more than a 100 instances at once most instances are stuck in hibernation for 15 or more minutes simply because they cannot execute this call.

The problem is further exasperated by the lack of documentation from AWS about the limits on this particular endpoint, UpdateInstanceInformation, or how to raise them. Contacting AWS support so far has also not been helpful as they are insisting I need to upgrade to a paid technical support package because the issue is more complex than the regular support can handle. I prefer not to upgrade as I'm not in need of technical support.

From the SSM agents standpoint, the problem could be somewhat mitigated by changing the hibernation behavior to randomize the waits, i.e. instead of a fixed hibernation of 5 minutes try hibernating for a duration between 3-5 minutes chosen randomly. I could provide the patch if the maintainers feel this is worthwhile.

[AmyZhangAmz]

Hi @emil2k

Thank you for posting here. This “ThrottlingException” is caused by instance sending “UpdateInstanceInformation” API request too frequently. Based on SSM Agent logic, the default frequency of sending “UpdateInstanceInformation” is five minutes. API request limits are uniformly enforced to ensure quality of service for every AWS customer.