Packer: amazon-ebs: Error modify AMI attributes: RequestLimitExceeded: Request limit exceeded.

[nitrocode]

Overview of the Issue

I'm trying to build an AMI in account A in region us-west-2 and copy it to us-east-1, then share that AMI with account B. It seems to fail when modifying users.

Reproduction Steps

1. packer build

Packer version

1.6.2

Simplified Packer Buildfile

    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "us-west-2",
      "instance_type": "t3.medium",
      "ami_virtualization_type": "hvm",
      "ssh_username": "ec2-user",
      "ssh_interface": "private_ip",
      "iam_instance_profile": "packer",
      "ami_name": "amzn2-ebs-{{isotime | clean_resource_name}}",
      "ami_regions": ["us-east-1"],
      "ami_users": [
        "snip",
        "snip"
      ],
      "security_group_ids": ["sg-snip"],
      "subnet_id": "subnet-snip",
      "source_ami_filter": {
        "filters": {
          "virtualization-type": "hvm",
          "name": "amzn2-ami-ecs-hvm-2.0.????????-x86_64-ebs",
          "root-device-type": "ebs"
        },
        "owners": ["amazon"],
        "most_recent": true
      },
    }

Setting "max_retries": 10, worked

Operating system and Environment details

Amazon2 x64

Log Fragments and crash.log files

2020-10-08T19:52:57Z: ==> amazon-ebs: Creating AMI snip from instance snip
--
  | 2020-10-08T19:52:57Z:     amazon-ebs: AMI: ami-snip
  | 2020-10-08T19:52:57Z: ==> amazon-ebs: Waiting for AMI to become ready...
  | 2020-10-08T19:55:13Z: ==> amazon-ebs: Copying/Encrypting AMI (ami-snip) to other regions...
  | 2020-10-08T19:55:13Z:     amazon-ebs: Copying to: us-east-1
  | 2020-10-08T19:55:13Z:     amazon-ebs: Waiting for all copies to complete...
  | 2020-10-08T20:00:24Z: ==> amazon-ebs: Modifying attributes on AMI (ami-snip)...
  | 2020-10-08T20:00:24Z:     amazon-ebs: Modifying: users
  | 2020-10-08T20:00:29Z: ==> amazon-ebs: Error modify AMI attributes: RequestLimitExceeded: Request limit exceeded.
  | ==> amazon-ebs:     status code: 503, request id: snip

[SwampDragons]

Looks like you're being throttled by Amazon. Most of our API calls have retries to handle that but this one must not. While you wait for Packer to fix that on our end, I'm glad you have a workaround.

[nitrocode]

I'm a little bit confused on these areas.

1. What is the default value of max_retries if the value is not set? It seems to have made a difference above by setting it to an arbitrary value of 10. 🤷‍♂️
2. What is the difference between using max_retries and setting environment value AWS_MAX_ATTEMPTS. I believe the latter is for the aws waiter retry logic... ?
3. Does the environment variable AWS_MAX_ATTEMPTS matter if a retry logic is not surrounding the aws API call ?
4. There are 12 previous issues containing RequestLimitExceeded. Should all aws api calls be surrounded by a similar retry logic to prevent this issue in the future?

[SwampDragons]

1) the default value is that Packer doesn't set it, so Amazon uses the default retries for each service.
2) You're correct. AWS_MAX_ATTEMPTS is for the waiters, waiting for long-running tasks like image copies to complete. It makes an api call every few seconds and waits for the answers it gets to be different. max_retries is for single API calls for short-running tasks, where if the API call fails due to throttling or something else the AWS sdk will automatically repeat the call.
3) Yes, it matters, because it has to do with waiters and not one-off api calls See #2.
4) probably? Though after looking over this I'm wondering why we do all those custom Packer retries at all when the config has a max_retries option. Maybe we just auto-default max_retries to 10 or 20. If the only likely failure is throttling rather than something to do with eventual consistency, we probably should let the aws sdk handle it.

[nitrocode]

I like the idea of defaulting the max_retries to a higher number. Seems like that might fix other RequestLimitExceeded conditions that haven't been tripped over yet.

[nitrocode]

@swampdragons where is the max_retries default set in code? Can we raise the default and add the extra retry logic for good measure?

[SwampDragons]

I think it's here: https://github.com/hashicorp/packer/blob/master/builder/amazon/common/access_config.go#L219

[SwampDragons]

Yeah I think doing the combo is a good idea.