Packer: amazon-import: If the import job fails Packer wait until timeout instead of failing immediately

[rickard-von-essen]

There are three issues:

1) If the import job fails Packer still waits for success until the timeout is reached instead of detecting the failure and exit immediately.

2) Interrupt is not handled by the post-processor.

3) We should hardcode the Format to OVA

[rickard-von-essen]

Repro code:

{
  "variables": {
    "aws_bucket": "{{env `AWS_BUCKET`}}",
    "iso_dir": "{{env `HOME`}}/Downloads/"
  },
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E bash '{{.Path}}'",
      "script": "setup.sh"
    }
  ],
  "builders": [
    {
      "type": "virtualbox-iso",
      "output_directory": "builds",
      "format": "ova",
      "boot_command": [
        "<esc><wait>",
        "<esc><wait>",
        "<enter><wait>",
        "/install/vmlinuz<wait>",
        " auto<wait>",
        " console-setup/ask_detect=false<wait>",
        " console-setup/layoutcode=us<wait>",
        " console-setup/modelcode=pc105<wait>",
        " debconf/frontend=noninteractive<wait>",
        " debian-installer=en_US<wait>",
        " fb=false<wait>",
        " initrd=/install/initrd.gz<wait>",
        " kbd-chooser/method=us<wait>",
        " keyboard-configuration/layout=USA<wait>",
        " keyboard-configuration/variant=USA<wait>",
        " locale=en_US<wait>",
        " netcfg/get_domain=vm<wait>",
        " netcfg/get_hostname=vagrant<wait>",
        " grub-installer/bootdev=/dev/sda<wait>",
        " noapic<wait>",
        " preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/preseed.cfg<wait>",
        " -- <wait>",
        "<enter><wait>"
      ],
      "boot_wait": "10s",
      "disk_size": 81920,
      "guest_os_type": "Ubuntu_64",
      "headless": false,
      "http_directory": "http",
      "iso_urls": [
        "{{ user `iso_dir` }}/ubuntu-18.04.1-server-amd64.iso",
        "http://cdimage.ubuntu.com/ubuntu/releases/bionic/release/ubuntu-18.04.1-server-amd64.iso"
      ],
      "iso_checksum_type": "sha256",
      "iso_checksum": "a5b0ea5918f850124f3d72ef4b85bda82f0fcd02ec721be19c1a6952791c8ee8",
      "ssh_username": "vagrant",
      "ssh_password": "vagrant",
      "ssh_wait_timeout": "15m",
      "shutdown_command": "echo 'vagrant' | sudo -S shutdown -P now",
      "guest_additions_path": "VBoxGuestAdditions_{{.Version}}.iso",
      "virtualbox_version_file": ".vbox_version",
      "vm_name": "packer-ubuntu-18.04-amd64",
      "vboxmanage": [
        [
          "modifyvm",
          "{{.Name}}",
          "--memory",
          "1024"
        ],
        [
          "modifyvm",
          "{{.Name}}",
          "--cpus",
          "1"
        ]
      ]
    }
  ],
  "post-processors": [
    {
      "type": "amazon-import",
      "keep_input_artifact": true,
      "s3_bucket_name": "{{user `aws_bucket`}}",
      "license_type": "BYOL",
      "tags": {
        "Description": "packer amazon import "
      }
    }
  ]
}

preseed.cfg

choose-mirror-bin mirror/http/proxy string
d-i base-installer/kernel/override-image string linux-server
d-i clock-setup/utc boolean true
d-i clock-setup/utc-auto boolean true
d-i finish-install/reboot_in_progress note
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i partman-auto/disk string /dev/sda
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select atomic
d-i partman-auto/method string lvm
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-lvm/device_remove_lvm boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman/confirm_write_new_label boolean true
d-i pkgsel/include string openssh-server cryptsetup build-essential libssl-dev libreadline-dev zlib1g-dev linux-source dkms nfs-common
d-i pkgsel/install-language-support boolean false
d-i pkgsel/update-policy select none
d-i pkgsel/upgrade select full-upgrade
d-i time/zone string UTC
tasksel tasksel/first multiselect standard, ubuntu-server

d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/layoutcode string us
d-i keyboard-configuration/modelcode string pc105
d-i debian-installer/locale string en_US

# Create vagrant user account.
d-i passwd/user-fullname string vagrant
d-i passwd/username string vagrant
d-i passwd/user-password password vagrant
d-i passwd/user-password-again password vagrant
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false
d-i passwd/user-default-groups vagrant sudo
d-i passwd/user-uid string 1000

#!/bin/bash -eux

# Add vagrant user to sudoers.
echo "vagrant        ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers
sed -i "s/^.*requiretty/#Defaults requiretty/" /etc/sudoers

# Disable daily apt unattended updates.
echo 'APT::Periodic::Enable "0";' >> /etc/apt/apt.conf.d/10periodic

# AWS stuff
KERNEL_VERSION=4.15.0-1021-aws
apt-get update
apt-get install -y linux-image-${KERNEL_VERSION} linux-image-${KERNEL_VERSION} cloud-init

# Clean up
rm -f /home/vagrant/*.sh
sync

[rickard-von-essen]

This is show by packer still waiting for import to complete (``) but:

$ AWS_PROFILE=admin aws --region eu-west-1 ec2 describe-import-image-tasks
{
    "ImportImageTasks": [
        {
            "ImportTaskId": "import-ami-03202c13763ba4333",
            "LicenseType": "BYOL",
            "SnapshotDetails": [
                {
                    "DiskImageSize": 955217920.0,
                    "Format": "VMDK",
                    "Status": "completed",
                    "UserBucket": {
                        "S3Bucket": "rickard-von-essen-tmp",
                        "S3Key": "packer-import-1537366089.ova"
                    }
                }
            ],
            "Status": "deleted",
            "StatusMessage": "ClientError: Unsupported kernel version 4.15.0-1021-aws"
        }
   ]
}

[rickard-von-essen]

A working example is using chef/bento with this patch:

diff --git a/centos/centos-7.5-x86_64.json b/centos/centos-7.5-x86_64.json
index af5df48..5472646 100644
--- a/centos/centos-7.5-x86_64.json
+++ b/centos/centos-7.5-x86_64.json
@@ -18,6 +18,7 @@
       "shutdown_command": "echo 'vagrant' | sudo -S /sbin/halt -h -p",
       "ssh_password": "vagrant",
       "ssh_port": 22,
+      "format": "ova",
       "ssh_username": "vagrant",
       "ssh_wait_timeout": "10000s",
       "type": "virtualbox-iso",
@@ -152,8 +153,13 @@
   ],
   "post-processors": [
     {
-      "output": "../builds/{{user `box_basename`}}.{{.Provider}}.box",
-      "type": "vagrant"
+      "type": "amazon-import",
+      "keep_input_artifact": true,
+      "s3_bucket_name": "{{user `aws_bucket`}}",
+      "license_type": "BYOL",
+      "tags": {
+        "Description": "packer amazon import "
+      }
     }
   ],
   "provisioners": [
@@ -174,6 +180,7 @@
         "../_common/virtualbox.sh",
         "../_common/vmware.sh",
         "../_common/parallels.sh",
+        "scripts/aws.sh",
         "scripts/cleanup.sh",
         "../_common/minimize.sh"
       ],
@@ -181,6 +188,7 @@
     }
   ],
   "variables": {
+    "aws_bucket": "rickard-von-essen-tmp",
     "box_basename": "centos-7.5",
     "build_timestamp": "{{isotime \"20060102150405\"}}",
     "cpus": "1",
diff --git a/centos/scripts/aws.sh b/centos/scripts/aws.sh
new file mode 100644
index 0000000..8bcac6a
--- /dev/null
+++ b/centos/scripts/aws.sh
@@ -0,0 +1,5 @@
+#!/bin/bash -
+
+set -o nounset
+
+yum install -y cloud-init awscli

Then this can be built with:

$ AWS_PROFILE=<profile> AWS_REGION=<region> packer build -only virtualbox-iso -var aws_bucker=<bucket name>  centos-7.5-x86_64.json

[lmayorga1980]

Should Packer also check for specific ClientErrors?

"StatusMessage": "ClientError: Unsupported kernel version 4.15.0-1021-aws"

[rickard-von-essen]

Yes if the import fails we should display the StatusMessage.

[SwampDragons]

I think we can probably add a new WaiterAcceptor to https://github.com/hashicorp/packer/blob/master/builder/amazon/common/state.go#L208-L220
with State: request.FailureWaiterState,

something like

https://github.com/hashicorp/packer/blob/master/vendor/github.com/aws/aws-sdk-go/service/ec2/waiters.go#L386-L390

Though we'll have to figure out what the "failed" state is called for these imports.

[SwampDragons]

oh nevermind, looks like the status we're looking for is probably "deleted" according to Rickard's query above.

[SwampDragons]

I'll push a patch later today for you to test Rickard :)

[SwampDragons]

I think https://github.com/hashicorp/packer/pull/6747 should fix this, if someone wants to test.

[SwampDragons]

@rickard-von-essen have you had a chance to test whether the linked PR fixes this?

[rickard-von-essen]

Running a test now.

[rickard-von-essen]

Works great! 👍

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.