Packer: AWS Error: "UnknownParameter: The parameter KmsKeyId is not recognized" when attempting to encrypt AMI

[alexgottschalkmedal]

FOR BUGS:

Describe the problem and include the following information:

• Packer version 1.2.2
• Ubuntu 16.04 Xenial server amd64
• Log output and build template are in
  https://gist.github.com/alexgottschalkmedal/a9f2d983eb505cdbf804f0e7b0cacee0

Packer gets through the entire build and provisioning process, and successfully stops the build instance. When trying to create a new AMI, the error in the subject occurs - UnknownParameter: The parameter KmsKeyId is not recognized.

When I omit the kms_key_id parameter from the block device mapping, I get an AMI encrypted with the account master KMS key. This does not meet our requirements - specifically, copying encrypted AMIs between accounts requires use of a KMS sub-key in the originating account.

[SwampDragons]

Thanks for opening this. I just got a chance to dig in, and I think this is actually a bug in the docs, rather than the amazon builder. I think what happened is some docs got cleaned up after we merged #5774, and even though conversation on that PR explicitly mentioned here that this wasn't a valid key for the block device mapping, we still managed to mess up the docs.

I'm sorry for the confusion; I'll clean up the docs.

[alexgottschalkmedal]

Does this mean that it's not possible at all to create an AMI with volumes encrypted by a specific KMS key, if I'm using the amazon-ebs builder?

[SwampDragons]

Sorry for the delayed response -- I'm going to do some testing and get back to you on this one. I feel certain it's possible and that our docs got out of whack but I need to verify.

[joshisumit]

Hey Guys, I am facing the same error "UnknownParameter: The parameter KmsKeyId is not recognized". I am specifying my custom KMS key in it.

ami_block_device_mappings": [ { "encrypted": true, "kms_key_id": "arn:aws:kms:us-east-1:001001010:key/fgddgsg-xhaha-ggh-kjuqoiq" } ]
I want to encrypt the EBS with my Key not the default KMS key.
Any Solution/Workaround ?

[rickard-von-essen]

@joshisumit Ensure you have the latest versions of packer 1.2.3 or 1.2.4 and if it still doesn't work open a new issue.

[joshisumit]

Thanks @rickard-von-essen for quick response.
I have tried it with version 1.2.4 also, but still facing the same issue. I will open a new issue.

I have a quick question here: I want to encrypt EBS root volume (/dev/xvda1 which is mounted on / ). Currently I am only using ami_block_device_mappings with following config:

{
"ami_block_device_mappings": [
      {
          "device_name": "/dev/sda1",
          "volume_size": 30,
          "volume_type": "gp2",
          "encrypted": true,
          "delete_on_termination": true,
          "kms_key_id": "arn:aws:kms:us-east-1:32000991:key/sdfdsf-sdsdsds-dsd-se9ssfsd"
      }
    ]
}

So, do I need to enable "encrypt_boot" also ? (It is for encrypting root volume ? )

[rickard-von-essen]

Re-reading the original PR this seems to be an doc issue. See my comment here: https://github.com/hashicorp/packer/pull/5774#issuecomment-356204445
You can't use kms_key_id in ami_block_device_mappings only in launch_block_device_mappings.

[joshisumit]

Thank you @rickard-von-essen.

My main goal is: To copy the encrypted AMI in different region and other AWS accounts also. config for that is:

{
"region": "us-east-1",
"ami_block_device_mappings": [
      {
          "device_name": "/dev/sda1",
          "volume_size": 30,
          "volume_type": "gp2",
          "encrypted": true,
          "delete_on_termination": true
      }
    ],
 "kms_key_id": "arn:aws:kms:us-east-1:32000991:key/sdfdsf-sdsdsds-dsd-se9ssfsd",
 "snapshot_users": ["900099099", "995100000000"],
 "ami_users": ["900099099", "900099099"]
}

So according to that above config will
1) Create Encrypted AMI in 'us-east-1' region with key provided in "kms_key_id" and in other region it will use default KMS key to encrypt the AMI
2) It copies the Encrypted snapshot & AMI in other AWS Account

Right?

[rickard-von-essen]

I just recently posted some examples for using kms_key_id, region_kms_key_ids, and encrypting only certain devices on the mailing list. Please check that and post there if you need help.

[rickard-von-essen]

Let's reopen this. See my comment here: https://github.com/hashicorp/packer/pull/5774#issuecomment-356204445
You can't use kms_key_id in ami_block_device_mappings only in launch_block_device_mappings. This needs to be documented and verified.

[starlton-eb]

I can confirm that you cannot use kms_key_id in ami_block_device_mappings only in launch_block_device_mappings, but the rub is that even if you do use kms_key_id in the launch_block_device_mappings, it only gets used when creating temporary snapshots, which are then copied to new snapshots using either the boot kms key or default if encrypted.

This drove me nuts as the final snapshots tied to the AMI would not be encrypted to the kms key I had specified in the launch_block_device_mappings, so I thought it was just being ignored until I watched in AWS console while packer was running and saw it create snapshots encrypted with the key I had specified and then before finishing up, would copy that snapshot to a new snapshot with either the boot kms or system default kms and throw away the one I wanted. This behavior was the same regardless of whether I defined corresponding ami_block_device_mappings or not.

So I thought maybe I was wrong and you cannot create an AMI like this so I attempted to do so manually through AWS console and was successful in doing so. Of course I cannot copy that AMI to another AMI in a different region as AWS AMI copy only allows you to override one kms key but that's okay because if I could get this to work in packer, I would just rebake for each region.

IMHO, it would be awesome if ami_block_device_mappings honored the kms_key_id, and use that when copying the final volume snapshot that would be used by the AMI, even knowing that you will loose the kms granularity across the volumes if you attempt to copy that AMI to another region.

Honestly, I don't have years of experience with packer and thus this may be possible and I am just not seeing it -- see my post on stackoverflow.

[SwampDragons]

We use the CopyImage request. According to the AWS docs:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIEncryption.html

CopyImage accepts only one key at a time and encrypts all of an image's snapshots (whether root or data) to that key. However, it is possible to manually build an AMI with snapshots encrypted to multiple keys.

Sounds like we may need to have a more complex custom solution than CopyImage provides. I'll do some more research.

[rickard-von-essen]

We should remove the whole copy-and-encrypt machinery since now AWS supports starting a EC2 with encrypted boot volume from an unencrypted AMI.

https://aws.amazon.com/blogs/security/create-encrypted-amazon-ebs-volumes-custom-encryption-keys-launch-amazon-ec2-instance-2/

[starlton-eb]

@rickard-von-essen the problem with that is if you are spinning up say 50 new instances with say 6 volumes each it would add a tremendous amount of time. With the use of another wonderful Hashicorp tool called terraform, it is not uncommon for one of us to spin up and tear down at least this many instances in a day when developing changes to our automation.

For our larger environments (prod/stage) we found ourselves having to create a script-in-the-middle that's kind of a hybrid to this, were we provision an EC2 instance using a new KMS key and then generating an AMI from it. The reason is because with the max grants being 2,500 and we have at least 6 volumes, sometimes 1 or 2 more, it limits us to ~400 EC2 instances max. Thus, we have to manage multiple AMIs and add another manual step in our flow plus complicate our terraform AMI filter. If we were able to encrypt each volume within the AMI using a different KMS key, or biggest system sits at around 500 EC2 instances so we would be covered and have plenty of room to grow.

With that said, I feel very spoiled with Hashicorp's tools and this is a minor nit that would be nice to have, but please don't remove what it supports now for encryption.

[rickard-von-essen]

@starlaton-eb sorry I was a bit brief in my comment. I didn't sugest that we remove the ability to create an encrypted AMI. Rather that we simplify how that is achieved.

Currently to create an encrypted AMI Packer launches a EC2 from the source_ami, provisions it, creates an unencrypted AMI from it, and then copies it to an encrypted AMI.

We could remove the copies to an encrypted AMI if we instead launched the EC2 with an encrypted EBS volume.

[chakradharavarma]

I am trying to create an AMI that its root volume to be encrypted with a custom kms key id,

The image is failing to copy the AMI,

amazon-ebs: Creating encrypted copy in build region: us-east-1
amazon-ebs: Waiting for all copies to complete...
==> amazon-ebs: 1 error(s) occurred:
==> amazon-ebs:
==> amazon-ebs: * Error waiting for AMI (ami-046d5f616127cf0e1) in region (us-east-1): ResourceNotReady: failed waiting for successful resource state

Here is the config file, Please suggest what is wrong in here. I am using latest 1.41 version of packer.

"encrypt_boot": true,
"kms_key_id": "{{user kms_arn}}",
"ami_regions": "us-east-1"

Thanks in advance

[SwampDragons]

Hi, I've been doing a stale issue cleanup and I think the final part of this issue, launching an encrypted volume rather than copying later, is already implemented as explained in #8105; All you need to do is _not_ set the encrypt_boot or kms_key_id options in the builder, instead only setting them in the launch_block_device_mappings. Packer will preserve the encryption in the boot region.

I'm pushing some docs to make this more clear in #9766 and then I think this can be closed. 🎉

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.