Sceptre: AWS CFN - Exceeded request limit

[laur1s]

We have a usecase where our CI/CD pipeline updates around 50 cloudformation stacks during the deployment. Because of that we face the request limit exceeded error. Could this be fixed or are there any workarounds?

$ sceptre --var-file config/UAT/MA/mac --var "imageversion=${IMAGE_TAG}" launch  TEST/ecs-service -y
[2019-07-08 09:50:11] - TEST/ecs-service - Launching Stack
[2019-07-08 09:50:25] - Request limit exceeded, pausing...
[2019-07-08 09:50:26] - TEST/ecs-service - Stack is in the UPDATE_COMPLETE state
[2019-07-08 09:50:26] - TEST/ecs-service - Updating Stack
[2019-07-08 09:50:31] - Request limit exceeded, pausing...
[2019-07-08 09:50:42] - Request limit exceeded, pausing...
[2019-07-08 09:50:53] - Request limit exceeded, pausing...
[2019-07-08 09:51:03] - Request limit exceeded, pausing...
[2019-07-08 09:51:10] - Request limit exceeded, pausing...
[2019-07-08 09:51:22] - Request limit exceeded, pausing...
[2019-07-08 09:51:34] - Request limit exceeded, pausing...
[2019-07-08 09:51:53] - Request limit exceeded, pausing...
[2019-07-08 09:52:05] - Request limit exceeded, pausing...
[2019-07-08 09:52:26] - Request limit exceeded, pausing...
[2019-07-08 09:52:44] - Request limit exceeded, pausing...
[2019-07-08 09:53:05] - Request limit exceeded, pausing...
[2019-07-08 09:53:29] - Request limit exceeded, pausing...
[2019-07-08 09:53:49] - Request limit exceeded, pausing...
[2019-07-08 09:54:09] - Request limit exceeded, pausing...
[2019-07-08 09:54:35] - Request limit exceeded, pausing...
[2019-07-08 09:54:59] - Request limit exceeded, pausing...
[2019-07-08 09:55:22] - Request limit exceeded, pausing...
[2019-07-08 09:55:50] - Request limit exceeded, pausing...
[2019-07-08 09:56:17] - Request limit exceeded, pausing...
[2019-07-08 09:56:48] - Request limit exceeded, pausing...
[2019-07-08 09:57:16] - Request limit exceeded, pausing...
[2019-07-08 09:57:49] - Request limit exceeded, pausing...
[2019-07-08 09:58:22] - Request limit exceeded, pausing...
[2019-07-08 09:58:56] - Request limit exceeded, pausing...
[2019-07-08 09:59:33] - Request limit exceeded, pausing...
[2019-07-08 10:00:02] - Request limit exceeded, pausing...
[2019-07-08 10:00:42] - Request limit exceeded, pausing...
[2019-07-08 10:01:15] - Request limit exceeded, pausing...
"Exceeded request limit 30 times. Aborting."

[ngfgrant]

This is a AWS thing. Its very annoying we get it too with our integration tests occasionally. We reduced the number of concurrent builds to try and help. I don't think there is much we can do from our side.

[ngfgrant]

https://github.com/Sceptre/sceptre/commit/62e7a5de2484c1741d003c311729309ade25f82d#diff-1d37e48f9ceff6d8030570cd36286a61

[chhibber]

We are launching 30+ stacks at once, and we leverage "Decorrelated Jitter" in almost all our boto calls and it has made a big impact, to the point where we don't see failed deployments anymore.

Here is the updated retry function we are testing out now:

connection_manager.py

    @functools.wraps(func)
    def decorated(*args, **kwargs):
        max_retries = 30
        attempts = 1
        mdelay = 1
        delay_cap = 45
        while attempts < max_retries:
            try:
                return func(*args, **kwargs)
            except ClientError as e:
                if e.response["Error"]["Code"] == "Throttling":
                    logger.error("Request limit exceeded, pausing {}...".format(mdelay))
                    time.sleep(mdelay)

                    # Using Decorrelated Jitter Algorithm
                    # We are picking number between a cap of 45 seconds and the last
                    # delay multiplied by 2.5, rounding to two decimal places.
                    mdelay = min(delay_cap, round((random.uniform(1, mdelay * 2.5)), 2))

                    attempts += 1
                else:
                    raise
        raise RetryLimitExceededError(
            "Exceeded request limit {0} times. Aborting.".format(
                max_retries
            )
        )

Great article on it and the impact jitter can have:
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/

[leonrodenburg]

Not sure if that is the case here, but I have also seen this when you have stacks that use a lot of !stack_output_external resolvers. I fixed it then by creating a custom resolver that only calls the CFN API once to fetch all stack outputs and caches them on disk. Every subsequent call to that resolver will check if a file is already on disk and uses the cached value if it is there.

[chhibber]

PR #803 - This should work for that as well. The resolve leverages the following method connection_manager.call to make the API call and that is wrapped with the retry functionality.

[leonrodenburg]

@chhibber I can imagine yes, but with the !stack_output_external it is actually calling the same endpoint every time. So if you have 5 of those in your template, it will basically make 5 of the same calls and only filter out the value it needs. Bit of waste in my opinion, also with smart retries.

[chhibber]

@leonrodenburg Understand now and generally agree, though I don't know enough of the overall workflow and if there is some need for realtime calls vs cache.

[leonrodenburg]

@chhibber No worries, your solution will work regardless. I just thought I'd drop in a note on how I had solved the issue with resolvers making lots of API calls on my side before. Might be inspiration for others if they face a similar issue.