Packer: Using temporary_iam_instance_profile_policy_document requires a "pause"

[rahul0705]

Overview of the Issue

When running a build with the option temporary_iam_instance_profile_policy_document in the JSON file, we get a failure to find the iamInstanceProfile. If we run in debug mode and wait ~5 secs after the policy/role is created before starting the server, it all works.

I've experienced this back when I handled policies in Ansible, we had to do a pause 5 after creating profiles so that the AWS API would update with the new profiles.

Reproduction Steps

Use the temporary_iam_instance_profile_policy_document in the JSON file

Packer version

Packer v1.4.5

Simplified Packer Buildfile

{                                                                                                                                                                                             
   "variables": {                                                                                                                                                                             
      "aws_access_key": "",                                                                                                                                                                   
      "aws_secret_key": "",                                                                                                                                                                   
      "zone_id": ""                                  
   },                                                                                                                                                                                                                                                                                                                                                                               
   "builders": [{ 
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "us-east-1",
      "source_ami_filter": {
         "filters": {
            "virtualization-type": "hvm",
            "name": "RHEL-7.5*",
            "root-device-type": "ebs",
            "architecture": "x86_64"
         },
         "owners": ["309956199498"],
         "most_recent": true
      },
      "temporary_iam_instance_profile_policy_document": {
         "Version": "2012-10-17",
         "Statement": [
         {
            "Effect": "Allow",
            "Action": [
               "route53:ListHostedZones",
               "route53:GetChange"
            ],
            "Resource": "*"
         },
         {
            "Effect" : "Allow",
            "Action" : [
               "route53:ChangeResourceRecordSets"
               ],
            "Resource": "arn:aws:route53:::hostedzone/{{user `zone_id`}}"
         }]
      },
      "ami_block_device_mappings": [{
         "device_name": "/dev/sda1",
         "delete_on_termination": "true",
         "volume_size": 20,
         "volume_type": "standard"
      }],
         "instance_type": "t3.micro",
         "ssh_username": "ec2-user",
         "ssh_interface": "private_dns",
         "ami_name": "packer-example {{timestamp}}"
   }]
}

Operating system and Environment details

This is a Red Hat server running on AWS EC2 with a power user role attached to the box connected to the RHUI repositories.

Linux hostname.sanitized 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Log Fragments and crash.log files

==> amazon-ebs: Creating temporary keypair: packer_5dc9c53d-6df8-a1bc-b368-9a5f687096ec
==> amazon-ebs: Creating temporary instance profile for this instance: packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e
2019/11/11 20:31:58 packer: 2019/11/11 20:31:58 Using specified security groups: [sanitized]
2019/11/11 20:31:58 packer: 2019/11/11 20:31:58 [DEBUG] Waiting for temporary instance profile: packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e
2019/11/11 20:31:58 packer: 2019/11/11 20:31:58 [DEBUG] Found instance profile packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e
==> amazon-ebs: Creating temporary role for this instance: packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e
2019/11/11 20:31:58 packer: 2019/11/11 20:31:58 [DEBUG] Waiting for temporary role: packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e
2019/11/11 20:31:58 packer: 2019/11/11 20:31:58 [DEBUG] Found temporary role packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e
==> amazon-ebs: Attaching policy to the temporary role: packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Adding tags to source instance
    amazon-ebs: Adding tag: "Name": "Packer Builder"
==> amazon-ebs:         status code: 400, request id: sanitized
==> amazon-ebs: Error launching source instance: InvalidParameterValue: Value (packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name
==> amazon-ebs:         status code: 400, request id: sanitized
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Detaching temporary role from instance profile...
==> amazon-ebs: Removing policy from temporary role...
==> amazon-ebs: Deleting temporary role...
==> amazon-ebs: Deleting temporary instance profile...
==> amazon-ebs: Deleting temporary keypair...
2019/11/11 20:32:05 [INFO] (telemetry) ending amazon-ebs
        status code: 400, request id: sanitized
2019/11/11 20:32:05 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2019/11/11 20:32:05 machine readable: amazon-ebs,error []string{"Error launching source instance: InvalidParameterValue: Value (packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name\n\tstatus code: 400, request id: sanitized"}
        status code: 400, request id: sanitized
==> Builds finished but no artifacts were created.
2019/11/11 20:32:05 [INFO] (telemetry) Finalizing.
Build 'amazon-ebs' errored: Error launching source instance: InvalidParameterValue: Value (packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name
        status code: 400, request id: sanitized

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Error launching source instance: InvalidParameterValue: Value (packer-5dc9c53e-bc95-7602-6a6c-29fc9071890e) for parameter iamInstanceProfile.name is invalid. Invalid IAM Instance Profile name
        status code: 400, request id: sanitized

==> Builds finished but no artifacts were created.

[SwampDragons]

Thanks, this makes sense to me. We should do a retry and catch that error.

[GregPriceDev]

Thanks so much for adding the new config key: temporary_iam_instance_profile_policy_document. In my case I need to do a docker pull of an image from AWS ECR. This saves me from having to create/secure/maintain a permanent IAM role.

I can confirm what @rahul0705 is seeing, that a wait is needed between the temporary policy being created and the temporary EC2 instance being able to use it. For me that time has been between 3 and 10 seconds. +1 for retry function =>

[sylviamoss]

Hello @rahul0705 and @DevOpsEtc!
I think I have found a fix for this. I have updated the retry mechanism, as suggested, and now it should retry whenever the IAM instance profile is not found when launching the instance. Here are some binaries https://circleci.com/gh/hashicorp/packer/23008#artifacts/containers/0.
Could you please tell me if the fix works for you ? :D

Thank you!

[GregPriceDev]

@sylviamoss

Your fix works great for the Linux version. I will keep using this binary until the fix is merged and released. Thank you for looking into this so quickly, and for all your contributions to Packer!

[WeekendsBull]

@SwampDragons do we have any ETA on getting this fix released? Will be it included in 1.4.6? I hope that Resource can take more than 'string'. You guys are the best!!!

[SwampDragons]

It'll be part of the 1.5.0 release, which we've scheduled for Dec 17th, but we're furiously working on some Big Features and that deadline maaaay slip by a couple of days so we can be really confident and test everything thoroughly. It'll definitely be out before HashiCorp does its holiday shutdown.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.