Packer: Race condition on spot request

[citosid]

Overview of the Issue

We use gitlab pipelines to build our system using AWS. Packer will create the main image and then deploy this at the last step.

However, sometimes (most of the times actually), it fails with this error:

==> amazon-ebs: Error finding source instance.
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Error terminating instance, may still be around: UnauthorizedOperation: You are not authorized to perform this operation.

At firsts, it looks like an error with permissions, but when we look up the error in cloudtrail, it shows this:

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "<-redacted->",
        "arn": "arn:aws:iam::<-redacted->:user/vault-approle-<-redacted->",
        "accountId": "<-redacted->",
        "accessKeyId": "<-redacted->",
        "userName": "vault-approle-<-redacted->"
    },
    "eventTime": "2019-09-26T18:05:56Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "DescribeInstances",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "<-redacted->",
    "userAgent": "aws-sdk-go/1.22.2 (go1.12.8; linux; amd64)",
    "errorCode": "Client.InvalidInstanceID.NotFound",
    "errorMessage": "The instance ID 'i-<-redacted->' does not exist",
    "requestParameters": {
        "instancesSet": {
            "items": [
                {
                    "instanceId": "i-<-redacted->"
                }
            ]
        },
        "filterSet": {}
    },
    "responseElements": null,
    "requestID": "<-redacted->",
    "eventID": "<-redacted->",
    "eventType": "AwsApiCall",
    "recipientAccountId": "<-redacted->"
}

It is important to notice that if we look up that instance id in the console, it shows there. This lead us to think that it might be a race condition caused by the fact that the AWS EC2 API is eventually consistent.

Packer version

1.4.3

Simplified Packer Buildfile

{
  "variables": {
    "ami_name": "",
    "aws_region": "",
    "project_name": "",
    "security_group_id": "",
    "pkg_artifact": "",
    "subnet_id": "",
    "vpc_id": ""
  },
  "builders": [{
    "type": "amazon-ebs",
    "ami_block_device_mappings": [{
        "device_name": "/dev/sda1",
        "volume_size": 20,
        "volume_type": "gp2",
        "delete_on_termination": true
      },
      {
        "device_name": "/dev/sdf",
        "volume_size": 50,
        "volume_type": "gp2",
        "delete_on_termination": true
      }
    ],
    "ami_description": "Project: {{user `project_name`}}",
    "ami_name": "{{user `ami_name`}}",
    "force_deregister": true,
    "force_delete_snapshot": true,
    "instance_type": "t2.xlarge",
    "region": "{{user `aws_region`}}",
    "run_tags": {
      "Name": "{{user `project_name`}}-packer-builder"
    },
    "security_group_id": "{{user `security_group_id`}}",
    "shutdown_behavior": "terminate",
    "source_ami_filter": {
      "filters": {
        "virtualization-type": "hvm",
        "name": "ubuntu/images/*ubuntu-bionic-18.04-amd64-server-*",
        "root-device-type": "ebs"
      },
      "owners": [
        "<-redacted->"
      ],
      "most_recent": true
    },
    "ami_users": [
      "<-redacted->"
    ],
    "spot_price": "auto",
    "ssh_interface": "public_ip",
    "ssh_username": "ubuntu",
    "subnet_id": "{{user `subnet_id`}}",
    "tags": {
      "Name": "{{user `project_name`}}"
    },
    "vpc_id": "{{user `vpc_id`}}"
  }],
  "provisioners": []
}

Operating system and Environment details

Ubuntu 18.04

[SwampDragons]

Thanks for reporting! I've created PR #8165 to address this. You can find builds of the patch here: https://circleci.com/gh/hashicorp/packer/14324#artifacts/containers/0

[citosid]

Oh wow! That was fast! Thank you so much @SwampDragons !!!

[colinpade]

WOW I found @SwampDragons in the wild!

just saying hi and keep being awesome!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.