Packer: Encrypted Snapshot not getting tagged

[rheak]

Packer_Version: 1.2.4
Host Platform: osx
Description:
When using "encrypt_boot": true, the encrypted snapshot does not get tagged.

[rickard-von-essen]

Please supply the information requested in the issue template:

• Packer version (packer version)
• Host platform (uname -a etc.)
• Debug log output from PACKER_LOG=1 packer build template.json.
  Please paste this in a gist.
• The _simplest example template and scripts_ needed to reproduce the bug.
  Include these in your gist.

[FrancescoFerro]

@rickard-von-essen

Packer 1.2.4 (1.1.0 same thing)
Linux xxxxx 3.10.0-862.3.2.el7.x86_64
[coming]

• It is not only the Tags that don't get added but also the attributes like "Permissions"
• It is kind of random, sometimes Tags are not there, sometimes Tags are there but Permissions aren't and sometimes both are not there. And if you get lucky both are there
• This was not happening couple weeks / months ago and started happening mid last week (~8th of June)
• Same issue with Packer 1.1.0 and 1.2.4

[mwhooker]

Curious to hear if #6332 solves this. Is anyone able to test? I can provide a binary if needed, just need to know os and architecture

[nictrix]

@mwhooker - I thought this would be as easy as telling you yes or no, but I don't think it is..see below:

My setup:

• Source AMI has a single encrypted volume (root) - customer managed key
• I specify "encrypt_boot": true" and "kms_key_id": "<key_id>" and add another volume that is encrypted by the default aws/ebs key (the kms_key_id is different then the source AMIs key)
• The user account I'm using is required to work with snapshots that have tags on them, can create them without tags, but any other action the snapshot needs to have a specific tag.

The build works until the very end where Packer is cleaning up the 1st AMI's snapshots and fails because tags are missing. It appears the 1st AMI and the 2 encrypted snapshots associated with it do not have tags on them at all.

I did another build and added the necessary tags while the 2nd AMI with the KMS key I provided in the template was still being created and Packer finished successfully. That 2nd AMI had the proper tags on it, including the snapshots associated with it.

In summary, this is what I see during the build before the 1st AMI is deregistered:

1st AMI - no tags
  root encrypted snapshot with source AMI's KMS key - no tags
  additional encrypted snapshot with aws/ebs default KMS Key - no tags
2nd AMI - has all the tags
  root encryped snapshot with my specified kms key - has all the tags
  additional encrypted snapshot with my specified kms key - has all the tags

Two Packer versions I was trying:
1.3.0-dev provided from: https://github.com/hashicorp/packer/pull/6332#issuecomment-403948998
1.2.5

The output from a failed build:

==> amazon-ebs: Stopping the source instance...
    amazon-ebs: Stopping instance, attempt 1
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating the AMI: <redacted>
    amazon-ebs: AMI: ami-<redacted>
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Creating Encrypted AMI Copy
==> amazon-ebs: Copying AMI: us-east-1(ami-<redacted>)
==> amazon-ebs: Encrypting with KMS Key ID: <redacted>
==> amazon-ebs: Waiting for AMI copy to become ready...
==> amazon-ebs: Deregistering unencrypted AMI
==> amazon-ebs: Deleting unencrypted snapshots
    amazon-ebs: Deleting Snapshot ID: snap-<redacted>
==> amazon-ebs: Error deleting snapshot, may still be around: UnauthorizedOperation: You are not authorized to perform this operation.
==> amazon-ebs:     status code: 403

[nictrix]

@mwhooker @rickard-von-essen Any update on this issue, thoughts on what I posted? I also tried using the latest packer release 1.3.1 and still failed with the same issues.

@rheak @FrancescoFerro the scenario I posted, is that what was happening for you also?

[SwampDragons]

This was not happening couple weeks / months ago and started happening mid last week (~8th of June

Did you upgrade to a newer version of Packer at that time?

[nictrix]

Any updates or thoughts on this issue? Any additional information I can provide? @SwampDragons @mwhooker @rickard-von-essen

We've moved to the latest Packer 1.3.3 - with the same results. To boil the problem down, Packer doesn't tag the AMI or snapshots before it tries to delete them. This happens consistently on each run.

[SwampDragons]

I've been swamped with other work and I haven't had a chance to work on this, sorry.

[nictrix]

No worries @SwampDragons I've tried to find time to try a fix myself, but I haven't had the chance either. Thanks for the response and update; I appreciate your acknowledgment.

[SwampDragons]

The fact that sometimes this happens and sometimes it doesn't makes me think it's a race condition. I wonder if this is a problem with Amazon's "eventual consistency" -- We create the tag but Amazon sometimes hasn't had a chance to fully recognize that it's been created, and we don't wait long enough to see if the tag gets generated.

I'll look and see if there are any waiters for waiting for a tag to be created; barring that, I wonder if we could work around this with a retry mechanism for snapshot deletion.

[SwampDragons]

hm actually, I'm gonna try moving the delete snapshot step to the cleanup. Can you let me know if this works for you?

osx build:
packer.zip

[nictrix]

@SwampDragons I will try this in a bit and report back. (also this has consistently failed for me, never had it succeed)

[nictrix]

@SwampDragons This worked!

This is what I saw in the logs:

==> amazon-ebs: Creating unencrypted AMI lDo0zme from instance <redacted>
    amazon-ebs: AMI: <redacted>
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Creating Encrypted AMI Copy
==> amazon-ebs: Copying AMI: us-east-1(<redacted>)
==> amazon-ebs: Encrypting with KMS Key ID: <redacted>
==> amazon-ebs: Waiting for AMI copy to become ready...
==> amazon-ebs: Deregistering unencrypted AMI
==> amazon-ebs: Deleting unencrypted snapshots

[SwampDragons]

Ah, I was thinking of @FrancescoFerro's comment.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.