Packer: Select latest source AMI

[JamieCressey]

Is there a way to use the latest available AMI (filtered by owner/AMI name/region etc) to be used as the source image? E.g. "Use the latest Ubuntu 14.04 AMI with an instance type of hvm:ebs-ssd provided by Canonical from the EU-West region"

MSOpenTech currently have this functionality for when building Azure images, and as I look to automate our cloud OS images it would be ideal if the source AMI could be dynamically chosen rather than hard coded into the template/require an env var.

The AWS SDK supports such filtering, although my GO knowledge is a little too limited to provide a suitable PR, sorry!

[cbednarski]

Thanks for the suggestion! To my knowledge packer does not currently support this. I'm not sure whether we can implement this against the Amazon API, but this might be a fun one to look into.

[lalyos]

eeeeasy. well sort of ;)

latestUbuntu=$( aws ec2 describe-images  \
        --region eu-west-1 \
        --filter Name=owner-id,Values=099720109477  \
        --query 'Images[? starts_with(ImageLocation, `099720109477/ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server`)] | sort_by(@, & ImageLocation) | [-1] | ImageId'  --out text
)
packer build -var "source-ami=${latestUbuntu}"  packer.json

tldr

I needed some similar thing. So i played quite a bit with aws cli. First I went down with the pattern:

aws ec2 somethingSpecific \
   | jq ".some.sub.elemnts"

Than I realized that the --query parameter is quite strong, and you can avoid jq completely. You can filter, sort, count ... So first you have to know the ownerId of the publisher. You can get it by checking the OwnerId of an image, by aws ec2 describe-images --image-ids <some-ubuntu-image>

Now you can get the full list of all ubuntu images in a region:

aws ec2 describe-images  \                                                                                                                                   
        --region eu-west-1 \
        --filter Name=owner-id,Values=099720109477

Now comes the trick: unfortunately there is no metadata about images, such as os_version/architrecture/kernel_version/disk_type, but good aws citizens such as Canonical, put all this info into the ImageLocation field. So with the mad sience of JMesPath, you can:

• filter by 099720109477/ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server prefix:
• thensort_by() ImageLocation, which ends with the date, so its sorted by date
• get the last element [-1]
• keep only the ImageId field
• use the text output

Put togethers as:

aws ec2 describe-images  \
  --region eu-west-1 \
  --filter Name=owner-id,Values=099720109477  \
  --query 'Images[? starts_with(ImageLocation, `099720109477/ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server`)] | sort_by(@, & ImageLocation) | [-1] | ImageId'  --out text

[lorengordon]

I'd be interested in this as well. I was envisioning a new parameter for the amazon-ebs builder, maybe source_ami_name, that is used with the describe-images api as a pattern to filter against AMI names. i.e. Specifying source_ami_name: "amzn-ami-hvm-*.x86_64-gp2" in the template would find all AMI names that match that pattern, sort them, and use the first. Since a lot of AMIs bake in a version of some kind in the AMI name, this works fairly well.

I did something like this with a Lambda function, to be used as a custom CloudFormation resource, based on an example provided by Amazon. Not sure if it helps, but you're welcome to take a look, of course.

This type of approach might also alleviate issues like #3132...

[csoutherland]

After looking at the naming mechanisms Canonical uses for the Ubuntu AMIs, I ultimately switched to parsing the CSVs they produce on their cloud images site as there is more information about the differences between similarly named AMIs. The obvious choice here would be to get the AMI ID into the CSV by passing the information in via an environment variable grabbed in the user variables section, but I would prefer to keep all the needed information to build an AMI within the template unless there is a specific reason not to (like with the access keys).

Since there is already a mechanism within the templates to gather an outside source of information into the user variables at runtime (in this case from the environment), would it make sense to add support for other user variable plugins? I've noticed that the code is not really set up to do that now; was this an intentional omission? Has work started on such a feature? Would it be worth submitting such a patch? Or am I the only one who thinks it would make sense to put AMI search functionality into a plugin which extends the variable system?

[CpuID]

@csoutherland +1 took the same approach here (parsing the CSVs on cloud-images.ubuntu.com), has been quite reliable (minimal format changes upstream).

[tomgoren]

@lalyos's solution works fine for me. Thanks for putting that together.

[ChrisLundquist]

If we were to integrate native packer support, I would propose a syntax like the following

Adding dynamic_source_ami as an option that could set the value of source_ami for the rest of the code.
Having both set would cause an error.

"dynamic_source_ami": {
    "filters": {
      "virtualization-type": "paravirtual",
      "name": "ubuntu/images/ebs/ubuntu-trusty-14.04-amd64-server-*"
    },
    "owners": ["099720109477"] # Canonical
    "most_recent": true
}

As this would be fairly in line with terraform: https://www.terraform.io/docs/providers/aws/r/instance.html

Likewise, we could borrow some of the code from terraform

builtin/providers/aws/data_source_aws_ami.go

215     params := &ec2.DescribeImagesInput{}
216     if executableUsersOk {
217         params.ExecutableUsers = expandStringList(executableUsers.([]interface{}))
218     }
219     if filtersOk {
220         params.Filters = buildAmiFilters(filters.(*schema.Set))
221     }
222     if ownersOk {
223         params.Owners = expandStringList(owners.([]interface{}))
224     }
225 
226     resp, err := conn.DescribeImages(params)
227     if err != nil {
228         return err

[ChrisLundquist]

I made a proof of concept implementation using the proposed format
https://github.com/mitchellh/packer/compare/master...ChrisLundquist:dynamic-source-ami?expand=1

A test packer file looks like this:

{
"builders":[{
  "type": "amazon-ebs",
  "access_key": "YOUR KEY",
  "secret_key": "YOUR SECRET",
  "region": "us-west-2",
  "source_ami": "ami-foobar PLACE HOLDER FIX ME",
  "dynamic_source_ami": {
    "filters": {
      "virtualization-type": "hvm",
      "name": "*ubuntu-xenial-16.04-amd64-server-*",
      "root-device-type": "ebs"
    },
    "owners": ["099720109477"],
    "most_recent": true
  },
  "instance_type": "t2.micro",
  "ssh_username": "ubuntu",
  "ami_name": "packer-quick-start {{timestamp}}",
  "subnet_id": "YOUR SUBNET",
  "vpc_id": "YOUR VPC"
}]
}

Naturally, we'd want to make source_ami and dynamic_source_ami exclusive options.

[lorengordon]

@ChrisLundquist nice! Rather than a new param, why not just use source_ami, and test if it is a map?

[ChrisLundquist]

@lorengordon I thought about it. Static typing in the config made me reconsider the approach though.

You'd need to change the config type to interface{} then try type assertions of string and DynamicAmiFilter in config.Prepare and hope one of them works.

It is doable, I wasn't sure how well it would come out though with corner cases.
Likewise, I wasn't sure what would be preferable from a packer file API perspective, different keys, or one key that has two different behaviors. I was imaging how I'd write the docs, and which would be simpler to explain. The Docker builder has the precedent of "I need exactly one of these keys", so I went that way.

Either way works for me, I'm sure other folks have stronger opinions on both the implementation side and the user API side.

[lorengordon]

@ChrisLundquist, ahh, static typing. Apparently, I'm far too used to python. Alright. Just one other suggestion then, and I'll shut up. And I realize I'm getting a bit fussy here, so, really, I'll slink off if no one else cares. But, the chosen param name feels a little awkward. Perhaps source_ami_lookup, instead? Besides seeming (to me) to better describe what the param does, it has the benefit of nicely sorting alphabetically with the source_ami param. :grin:

[ChrisLundquist]

@lorengordon I went with source_ami_filter. Likewise, I tried to get Owners into the generic filter section, but types made it hard, so it is special cased as it is in Terraform.

[lorengordon]

@ChrisLundquist, sounds good! Thanks for your work on this!

[sumit-srivastavaa]

I did same for Amazon linux AMI
Here irrespective of account, it is picking ami from AWS marketplace, which is suitable in a different use case, but if you have to run packer for creating your ami, you dont have to register an ami in your account, instead pick it from market place and play with it.

Function to strip quotes

function stripquotes (){
input=$1
output=${input//\"/}
output=${output//" "/-}
echo "$output"
}

Function ends here

base_ami=$(aws ec2 describe-images --region eu-west-1 --query 'Images[? starts_with(ImageLocation, amazon/amzn-ami-hvm-2018.03.0.20180622-x86_64-gp2)]' | jq .[].ImageId)
base_ami1=$(stripquotes $base_ami)

packer build -var "source_ami=${base_ami1}" build.json

maybe this stripquotes part could help:

[rickard-von-essen]

This is a really old question. You should use sourcr_ami_filter.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.