Aws-sam-cli: Error while using sam deploy

[gruebel]

Description

We have a SAM stack, with ~40 AWS::Serverless::Application resources in it. When we run following command in CodeBuild, we get after some time a ClientError, which result in a failed CodePipeline state, but the stack update ends successfully in the target account.

$ sam deploy --debug --template-file template.yaml --stack-name sam \
    --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND

Deploying with following values 
=============================== 
Stack name                 : sam 
Region                     : None 
Confirm changeset          : False 
Deployment s3 bucket       : None 
Capabilities               : ["CAPABILITY_IAM", "CAPABILITY_AUTO_EXPAND"] 
Parameter overrides        : {} 

...

Changeset created successfully. arn:aws:cloudformation:eu-central-1:XXX:changeSet/samcli-deploy1574865126/34eb88cf-9262-4ff1-83a2-7e343bb0b4dd 

2019-11-27 14:33:08 - Waiting for stack create/update to complete 

...

Traceback (most recent call last): 
  File "/usr/local/bin/sam", line 8, in <module> 
    sys.exit(cli()) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__ 
    return self.main(*args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main 
    rv = self.invoke(ctx) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke 
    return _process_result(sub_ctx.command.invoke(sub_ctx)) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke 
    return ctx.invoke(self.callback, **ctx.params) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke 
    return callback(*args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func 
    return ctx.invoke(f, obj, *args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke 
    return callback(*args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 93, in wrapped 
    raise exception  # pylint: disable=raising-bad-type 
  File "/usr/local/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 62, in wrapped 
    return_value = func(*args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/deploy/command.py", line 187, in cli 
    ctx.profile, 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/deploy/command.py", line 295, in do_cli 
    deploy_context.run() 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/deploy/deploy_context.py", line 129, in run 
    self.confirm_changeset, 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/deploy/deploy_context.py", line 170, in deploy 
    self.deployer.wait_for_execute(stack_name, changeset_type) 
  File "/usr/local/lib/python3.7/site-packages/samcli/lib/deploy/deployer.py", line 379, in wait_for_execute 
    self.describe_stack_events(stack_name, self.get_last_event_time(stack_name)) 
  File "/usr/local/lib/python3.7/site-packages/samcli/commands/_utils/table_print.py", line 68, in wrap 
    result = func(*args, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/samcli/lib/deploy/deployer.py", line 339, in describe_stack_events 
    for event_items in response_iterator: 
  File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in __iter__ 
    response = self._make_request(current_kwargs) 
  File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request 
    return self._method(**current_kwargs) 
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call 
    return self._make_api_call(operation_name, kwargs) 
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call 
    raise error_class(parsed_response, operation_name) 
botocore.exceptions.ClientError: An error occurred (Throttling) when calling the DescribeStackEvents operation (reached max retries: 4): Rate exceeded

It works fine with the old 0.32.0 version.

Steps to reproduce

Probably deploy a large ChangeSet.

Observed result

See description.

Expected result

No error :)

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

• CodeBuild
  ComputeType: BUILD_GENERAL1_LARGE
  Image: aws/codebuild/standard:2.0
  Type: LINUX_CONTAINER

• SAM 0.34.0

• Python 3.7

[sriram-mv]

Thanks for the report! We do have retries and client back pressure getting applied in case we face throttling. So maybe the way forward is not fail with a non-zero exit code, because the changeset is already being executed and we are streaming events back at that point.

[sriram-mv]

This has been merged and released.

[gruebel]

@TheSriram Thanks, looks good so far!

[agnewp]

it seems like these line of code end up calling Describe stack events in a tight loop
https://github.com/awslabs/aws-sam-cli/blob/develop/samcli/lib/deploy/deployer.py#L337-L339

we are observing this via cloud trail with just a few somewhat uncomplicated stacks

[piersf]

In which version of AWS SAM was this fixed? We're being hit hard by this whenever we release multiple microservices at the same time.

[gruebel]

@piersf in version 0.35.0

[piersf]

@gruebel I'm using SAM CLI, version 0.41.0 and it seems that it still is an issue. I just got the below error today while it was waiting for the stack update to complete:

Describing stack events for STACK failed: An error occurred (Throttling) when calling the DescribeStackEvents operation (reached max retries: 4): Rate exceeded
 -------------------------------------------------------------------------------------------------
 Error: Failed to create/update the stack: STACK, Waiter StackUpdateComplete failed: Waiter encountered a terminal failure state

Any idea why i am still seeing this?

Thank you!

[OperationalFallacy]

We're seeing this problem as well - rate exceeded and in cloudwatch insights its all requests from ci/cd

You should add an option to disable progress altogether, or at least make it much less frequent