Packer: ami_regions attempts to set destination AMI as unencrypted?

[nullify005]

Hi There,

As of this morning I'm having issues copying encrypted AMI's from us-west-2 to ap-southeast-2 using the ami_regions param.

Specifically it appears that packer is now trying to make unencrypted AMI's in the destination region instead of encrypted ones. It's possible that the AWS API has changed & doesn't allow something now which it did before (like a default changing), but I still have an issue.

So here is the packer error:

15:28:05 2018/09/28 05:28:05 machine readable: amazon-chroot,error []string{"1 error(s) occurred:\n\n* Error Copying AMI (ami-010d7627b94c82dfe) to region (ap-southeast-2): InvalidRequest: Snapshot snap-0d3ab678393ea04ff is encrypted. Creating unencrypted copy from an encrypted snapshot is not supported.\n\tstatus code: 400, request id: c4865d61-5d4b-4987-8cfc-dd08f206df8a"}

• the source AMI which is being used is encrypted using the default KMS key
• the intent is for it to remain encrypted when being copied amongst a few regions
• host platform is Ubuntu Xenial x86_64 in EC2 us-west-2 region
• packer 1.2.3

I have tried:

• toggling the encrypted_boot flag from true to false
• specifying kms_key_id to ""
• specifying region_kms_key_ids to "" for each copy region
• actually adding in the default ebs key id in for each region into the above
• adding ami_block_device_mappings and specifying encrypted true with key ids

Nothing appears to change the behaviour

I can copy the AMI's in the EC2 console without issues, so it's not something in AWS which appears to be broken at the moment.

A contrived config I'm using is:

    "builders": [
        {
            "type": "amazon-chroot",
            "access_key": "{{ user `aws_access_key` }}",
            "secret_key": "{{ user `aws_secret_key` }}",
            "ami_name": "{{ user `app_name` }} {{ user `branch_name` }} {{ user `git_commit` }} {{ isotime \"2006-01-02\" }} {{ user `build_number` }}",
            "source_ami": "{{ user `aws_source_ami` }}",
            "root_volume_size": 20,
            "encrypt_boot": "{{ user `encrypt_boot` }}",
            "kms_key_id": "",
            "pre_mount_commands": [
                ".packer/growfs.sh {{ .Device }}"
            ],
            "ami_regions": [ "ap-southeast-2", "us-west-2" ],
            "region_kms_key_ids": {
                "ap-southeast-2": "",
                "us-west-2": ""
            }
        }
    ],
    "provisioners": [
        {
            "type": "shell",
            "inline": [
                "echo '#!/bin/sh' > /usr/sbin/policy-rc.d",
                "echo 'exit 101' >> /usr/sbin/policy-rc.d",
                "chmod a+x /usr/sbin/policy-rc.d"
            ]
        },
        {
            "type": "shell",
            "inline": [
                "set -o xtrace",
                "rm -f /usr/sbin/policy-rc.d",
                "sudo /etc/init.d/xinetd stop",
                "sudo /etc/init.d/filebeat stop",
                "sudo /etc/init.d/rsyslog stop"
            ]
        }
    ]
}

Help?

[nullify005]

ok, so after much fiddling I think I found that:

• encrypt_boot must be set to true
• you do now need to set the kms key ids even if you are using the defaults ...?

[Photon0gen]

Also started experiencing this yesterday when copying AMIs to other regions with packer 1.2.1 and 1.3

[netflash]

Same story here :-(

[mzeier]

Also happening with the amazon-ebs builder.

[johnrothermel]

I am having the same issue and @nullify005's solution of specifying the region-specific kms_key_id is allowing the copy to work. Be nice to know what made the behavior change. In the mean time I'm looping through the regions and querying the key-aliases and passing the TargetKeyId from the alias/aws/ebs alias.

[mpetruzelli]

Same issue here with amazon-ebs builder.

Not using snapshot_users

Contrary to documentation:

If you want a region to be encrypted with that region's default key ID, you can use an empty string "" instead of a key id in this map. (e.g. "us-east-1": "") 

If the packer.json file, is as below, us-west-2 copy fails :

{
      "type": "amazon-ebs",
      "region": "us-east-1",
      "ami_regions": [
        "us-east-1",
        "us-west-2"
      ],
      "encrypt_boot": true,
      "region_kms_key_ids": {
        "us-east-1": "",
        "us-west-2": ""
      }

The default Keys must be set explicitly to produce a working packer.json

{
      "type": "amazon-ebs",
      "region": "us-east-1",
      "ami_regions": [
        "us-east-1",
        "us-west-2"
      ],
      "encrypt_boot": true,
      "region_kms_key_ids": {
        "us-east-1": "this-is-set-to-default-east-key-id",
        "us-west-2": "this-is-set-to-default-west-key-id"
      }

[yruss972]

happening for me on 1.2.4 and 1.3.2

[GoneWacko]

I opened #6789 to report this issue, but then I saw this ticket.
Ticket #6789 includes a slightly neater workaround for the issue in case you want to use the default KMS keys, as well as a pointer to the exact location in the source code where this bug originates.

The neater workaround (in my opinion) is still to explicitly specify the KMS key for each region in region_kms_key_ids, but using the KMS key alias for the default key. Because the alias is the same in each region (aws/ebs), your template is less bound to a particular AWS account:

"region_kms_key_ids": {
  "us-east-1": "alias/aws/ebs",
  "us-west-2": "alias/aws/ebs"
}

The alias/ prefix is required according to the CopyImage API docs: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CopyImage.html

[azr]

Hey here, I made PR #6787 that should allow to not set encrypt_boot field which makes copy step keep whatever setting was on the copied image; more explanation in the PR.

Please tell me if you think that this is a valid fix for your case.

packer_darwin_amd64.zip

packer_linux_amd64.zip

thanks 🙂

[nullify005]

So @GoneWacko has got the workaround sorted & it solves my original issue.

If AWS requires the keys to be specified now, then perhaps this is best resolved by:

• a documentation change in packer which states that the keys are necessary
• packer validating that keys are necessary when when encryption of the source or destination is specified

[matthiasr]

or packer could explicitly implement what the documentation states (instead of relying on it being implicit)? i.e. when the key for a region is not set, use alias/aws/ebs?

[GoneWacko]

@nullify005 AWS does not require that the key is specified. The CopyImage API will use the default key for the destination region if the Encrypted flag is true but the kms-key-id was not specified.

The issue is that the semantics for Packer (according to the Packer documentation) is to assume the default key should be used if the key ID is an empty string. E.g. the following specifies that you want to have an image copied over to eu-central-1 and us-east-1, using the default key for each region:

"ami_regions:" [ "eu-central-1", "us-east-1" ],
"region_kms_key_ids": {
  "eu-central-1": "",
  "us-east-1": ""
}

The mere fact that the source AMI is encrypted, or the mere fact that the destination region is even mentioned in region_kms_key_ids should cause packer to set Encrypted to true in the CopyImage call. If the value in region_kms_key_ids for the destination region is non-empty, then it should also specify that value as the KmsKeyId parameter for the CopyImage API call. That's how I understand the Packer documentation, at any rate.

Unfortunately right now what Packer actually _does_ is different from the documentation, because it will simply not set the Encrypted flag for the CopyImage call if the KMS Key ID for the destination region is an empty string.

So the bug is that Packer doesn't do what its own documentation says. And in my opinion the way the documentation says it should work is better than the way Packer actually works, because it maps to the way the actual Amazon API works the best: If you don't specify the key ID, just use the default.

@matthiasr: I think it's better to just leave the key id out of CopyImage call, because that would presumably break in the unlikely event that AWS renames (or lets users rename) the default key alias.

[matthiasr]

makes sense, thank you for clarifying!

On Tue, Oct 9, 2018 at 8:46 AM Stijn Gijsen [e-mail]
wrote:

@nullify005 https://github.com/nullify005 AWS does not require that the
key is specified. The CopyImage
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CopyImage.html
API will use the default key for the destination region if the Encrypted
flag is true but the kms-key-id was not specified.

The issue is that the semantics for Packer (according to the Packer
documentation) is to assume the default key should be used if the key ID is
an empty string. E.g. the following specifies that you want to have an
image copied over to eu-central-1 and us-east-1, using the default key
for each region:

"ami_regions:" [ "eu-central-1", "us-east-1" ],
"region_kms_key_ids": {
"eu-central-1": "",
"us-east-1": ""
}

The mere fact that the source AMI is encrypted, or the mere fact that the
destination region is even mentioned in region_kms_key_ids should cause
packer to set Encrypted to true in the CopyImage call. If the value in
region_kms_key_ids for the destination region is non-empty, then it
should also specify that value as the KmsKeyId parameter for the
CopyImage API call. That's how I understand the Packer documentation, at
any rate.

Unfortunately right now what Packer actually does is different from the
documentation, because it will simply not set the Encrypted flag for the
CopyImage call if the KMS Key ID for the destination region is an empty
string.

So the bug is that Packer doesn't do what its own documentation says. And
in my opinion the way the documentation says it should work is better than
the way Packer actually works, because it maps to the way the actual Amazon
API works the best: If you don't specify the key ID, just use the default.

@matthiasr https://github.com/matthiasr: I think it's better to just
leave the key id out of CopyImage call, because that would presumably break
in the unlikely event that AWS renames (or lets users rename) the default
key alias.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/hashicorp/packer/issues/6778#issuecomment-428111560,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAICBhihDxQjaRiEbK1Q79FKgmPd8bC-ks5ujGJvgaJpZM4W94of
.

[WeekendsBull]

@GoneWacko, if you're using the same default KMS key in different regions, the below might work.
"region_kms_key_ids": {
"us-east-1": "alias/aws/ebs",
"us-west-2": "alias/aws/ebs"
}
However, we are using the custom generated KMS key for each region which I defined in variables section, do I still need to add alias in the different? I am not sure if it works for me.

My packer code was working fine a month ago when we built and released the new AMI. However, this started not working a moth after?? I'm still on packer v.1.2.3, that it fails with the error message, "Error waiting for AMI Copy: unexpected state 'failed', wanted target 'available'"
I have configured the below environment keys are still getting the error message.
export AWS_POLL_DELAY_SECONDS=50
export AWS_MAX_ATTEMPTS=500

Any thoughts?

@azr, I have used the packer.zip you have attached here, which shows v1.3.2 but I got the below error,

[BETA: ] 2 error(s) occurred:
[BETA: ]
[BETA: ] * Unknown region: us-west-1
[BETA: ] * Unknown region: us-west-1
I can post the whole log if you want to.

[SwampDragons]

@ WeekendsBull I just fixed that error in another patch. I rebased @azr's patch on master and hopefully it fixes this bug for you:

packer.zip

packer.zip

[WeekendsBull]

@SwampDragons Thanks for the quick response and I was able to download and tried the fix from https://github.com/hashicorp/packer/files/2503422/packer.zip (first one). I don't have the error message about the unknown regions I submitted previously.

[BETA: aws] ==> aws-internal: 1 error(s) occurred:
[BETA: aws] ==> aws-internal:
[BETA: aws] ==> aws-internal: * Error waiting for AMI (ami-0c9555ab7a3029481) in region (us-west-1): ResourceNotReady: failed waiting for successful resource state
[BETA: aws] ==> aws-internal: Deregistering the AMI because cancellation, error or it was temporary (encrypt_boot was set)...
[BETA: aws] ==> aws-internal: Terminating the source AWS instance...
[BETA: aws] ==> aws-internal: Cleaning up any extra volumes...
[BETA: aws] ==> aws-internal: No volumes to clean up, skipping
[BETA: aws] ==> aws-internal: Deleting temporary security group...
[BETA: aws] ==> aws-internal: Deleting temporary keypair...
[BETA: aws] Build 'aws-internal' errored: 1 error(s) occurred:
[BETA: aws]
[BETA: aws] * Error waiting for AMI (ami-0c9555ab7a3029481) in region (us-west-1): ResourceNotReady: failed waiting for successful resource state
[BETA: aws]
[BETA: aws] ==> Some builds didn't complete successfully and had errors:
[BETA: aws] --> aws-internal: 1 error(s) occurred:
[BETA: aws]
[BETA: aws] * Error waiting for AMI (ami-0c9555ab7a3029481) in region (us-west-1): ResourceNotReady: failed waiting for successful resource state

----> I have set up encrypt_boot: true

The 2nd one I couldn't do even packer --version due to the error message that binary can't be executed.

Would you like to have a full log?

[WeekendsBull]

@SwampDragons we haven't changed anything that I don't understand that our last build on Sep 21st worked fine. There is no packer or ansible changes on our master. Is this happening due to the change on AWS side?

[WeekendsBull]

@SwampDragons I have created the packer box on my macbook and didn't have any issue at all with version 1.2.3 and version 1.3.2 you have provided. So, I did run "pip list" to compare what's got installed between my local and jenkins-slave. I have installed awscli, botocore, cryptography and setuptools to match the version on jenkins-slave build machine but still not success to make things working. Any suggestion?

[SwampDragons]

Full debug logs would definitely help.

Maybe try bumping the timeouts?

https://www.packer.io/docs/builders/amazon.html#exceeded-wait-attempts-while-waiting-for-tasks-to-complete

The other possibility I can think of is that maybe your jenkins machine's credentials don't have permissions to copy into that region?

[WeekendsBull]

@SwampDragons yes, copying was related to IAM permission described @ https://www.packer.io/docs/builders/amazon.html

I have added additional IMA as below and started to copying fine.

"ec2:CopySnapshot",
"ec2:CreateSnapshot",
"ec2:DescribeImportImageTasks",
"ec2:DescribeImportSnapshotTasks",
"ec2:ImportImage",
“ec2:ImportSnapshot",
"kms:GenerateDataKeyWithoutPlaintext",

• Error waiting for AMI (ami-0b26069be46a5c4f2) in region (us-west-1): ResourceNotReady: failed waiting for successful resource state is the current issue. Let me try to increase the timeout.
  --> this also got verified and had to add kms iam permissions and added ec2:Describe* to make this working. I'm still in packer v1.2.3. Thank you, @SwampDragons again :)

kms:GenerateDataKeyWithoutPlaintext was required in the IAM permission and we set boot_encrypt: true so that KMS encrypts the snapshot attached to AMI.

[alanivey]

The latest test build works great. I do, however, have to delete the unencrypted snapshot after the encrypted copies are created. I'm using AWS CodeBuild, and have the following post_build commands:

PACKER_INSTANCE_ID="$( awk '/Creating AMI [A-z0-9]+ from instance/ {print $NF}' build.out )"

PACKER_ORIG_SNAPSHOT_ID="$( aws ec2 describe-snapshots --owner-ids 'self' --filters '[{"Name":"description","Values":["Created by CreateImage('"${PACKER_INSTANCE_ID:?}"')*"]}]' --query Snapshots[].SnapshotId --output text )"

aws ec2 delete-snapshot --snapshot-id "${PACKER_ORIG_SNAPSHOT_ID:?}"

Packer json: https://gist.github.com/alanthing/316bf6d5bc4b1b2b65a21cd972d92cfa#file-centos7-json

Command run via:

CLIENT=ExampleCorp
ADDITIONAL_REGIONS=us-east-2

timeout --foreground --kill-after=15m 3h packer build -var "client=$CLIENT" -var "user_ami_regions=$ADDITIONAL_REGIONS" centos7.json 2>&1 | tee build.out

[azr]

Hey @alanthing, thanks for telling, I will take a look at this when I get time. Should be soon 🙂 !

[rickard-von-essen]

Closed in #6787

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.