Packer: Amazon-import: Packer takes a while to perfom the entire process

[ghost]

Hi all,
I'm using packer to create AWS ami file with the main propose to use it then with Terraform.
My starting point is a centOS 7 machine in OVA format using VirtualBox-OVF builder, provisioned with packer ansible provisioner and exported again to OVA format using the amazon-import post-processors plugin to upload it to an S3 bucket and then to import to the EC2 service. The packer output-virtualbox-ovf size is around 750 MB.
The entire process lasts around 1 hour, where most part of the time is spent in the OVA to AMI conversion inside AWS infrastructure. Is this a normal estimate time of processing to packer to get the whole image? Any idea on how to improve this times?

The template looks like that:

{
    "provisioners": [{
                "type": "ansible",
                "playbook_file": "/home/sysadmin/ansible/apache_playbook/provision.yml",
        "groups": "name_body",
        "sftp_command": "/usr/libexec/openssh/sftp-server -e"
        }],
    "builders": [{
        "type": "virtualbox-ovf",
        "source_path": "centos-packer.ova",
        "headless": "true",
        "ssh_username": "centos",
        "ssh_password": "centos",
        "shutdown_command": "echo 'packer' | sudo -S shutdown -P now",
        "format": "ova"
    }],
    "post-processors": [{
        "type": "amazon-import",
        "access_key": "",
        "secret_key": "",
        "region": "us-east-1",
        "s3_bucket_name": "mybucketname",
        "s3_key_name": "",
        "skip_clean": "false",
        "tags": {
             "Description": "body {{timestamp}}"
        }
    }] 
}

[rickard-von-essen]

If you run packer with PACKER_LOG=1 packer build ... and gist the log it will hopefully be easy to see which action is taking up the time.

I think there is a reason for the message _"may take a while"_. When I tested this it took a long time even to import considerable smaller ova's, so my hunch is that this is just AWS being slow.

[DanHam]

Hi,
I follow a pretty much identical process to produce my AMIs. The times you mention in your post seem fairly typical.

For me the entire process takes around 40 - 50 mins and like you I find that most of this time is spent in the 'upload' and 'import' phases of the amazon-import post-processor. My OVA's vary in size from around 330 - 380 MB. Very roughly the entire process is as follows:

1. Build OVA - 5 mins.
2. Upload OVA to Amazon S3 bucket - 10 mins.
3. Import and convert OVA to AMI - 30 mins.

Clearly, with the majority of time spent in the 'import and convert phase' there is not much you can do to reduce the overall time taken. As Rickard has said, the time taken for the import is in the hands of the AWS import service and out of your control. If you are really keen to reduce times you could try and reduce the time spent in the upload phase by purchasing 'S3 Transfer Acceleration'. I'm guessing that you can't or wouldn't want to reduce the size of your image any.

As you are no doubt aware, throughout steps 2 and 3 there is not much to see in terms of Packer output. For me, the long waits with no indication of forward progress can be a little worrying. To keep myself entertained (or make it look like I'm busy!) I check in on the progress of the upload and import process 'behind the scenes' using the AWS CLI.

OVA upload:
aws s3api list-multipart-uploads --bucket my-import-bucket

The output from the aws s3api command is a little boring - there's not much to see there, but at least you know _something's_ happening.

Import process:
aws ec2 describe-import-image-tasks

The output from the aws ec2 command is much more interesting and includes 'Status Message' and percentage 'Progress' indicators. Note that if you have multiple import tasks running you can filter the output by 'import task ID' by appending --import-task-ids import-ami-xxxxxx to the command above. This gives a fairly good picture of what the AWS VM Import service is actually doing and a clearer idea of why it takes so long!

Dan

[rickard-von-essen]

I'll close this as it's noting Packer can do. If you find any step of the process that is taking longer time when Packer performs it compared to doing the same manually with aws cli please provide more details and reopen.

In general I would recommend to only use amazon-import as last resort and only as a first step to get an AMI into AWS that isn't otherwise available. When it has been imported it's much better to use the amazon builders to iterate over it. And if speed is a high concern take a look at the amazon-chroot builder which is really fast.

[ghost]

Thank you so much, I adopted another way to do it with aws-ebs builder first and then vagran post-processor so I recuded the overall time in just around 10 minutes.