Copilot-cli: Can't create env in a different account (not authorized to perform: kms:UntagResource)

[hnrc]

I tried creating a new environment in a different AWS account but ran into an issue with KMSKey.

Exception=[class software.amazon.awssdk.services.kms.model.KmsException] ErrorCode=[AccessDeniedException], ErrorMessage=[User: arn:aws:sts::123:assumed-role/foo-executionrole/abc-123 is not authorized to perform: kms:UntagResource on resource: arn:aws:kms:eu-central-1:123:key/def-456]

I can see that the key policy on arn:aws:kms:eu-central-1:123:key/def-456 does not allow kms:UntagResource.

{
    "Version": "2012-10-17",
    "Id": "StackSet-sample-java-infrastructure-db3e2f97-ea6a-4a49-9e62-3533e652841a",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123:root"
            },
            "Action": [
                "kms:Create*",
                "kms:Describe*",
                "kms:Enable*",
                "kms:List*",
                "kms:Put*",
                "kms:Update*",
                "kms:Revoke*",
                "kms:Disable*",
                "kms:Get*",
                "kms:Delete*",
                "kms:ScheduleKeyDeletion",
                "kms:CancelKeyDeletion",
                "kms:Tag*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123:root"
            },
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": "*"
        }
    ]
}

CLI version: 0.2.0

[kohidave]

Oh that's definitely odd - we don't try to untag KMS keys. This is happening when you run copilot env init in the linking account part of environment creation?

Did you find this error in the CloudFormation stack?

Thanks for your help, hope we can get this resolved quickly!

[hnrc]

This is happening when you run copilot env init in the linking account part of environment creation?

Here's the exact command I was running:

$ AWS_DEFAULT_REGION=eu-central-1 copilot env init --profile another-account
What is your environment's name? dev
✔ Shared DNS permissions for this application to account 010101.

✔ Created the infrastructure for the dev environment.
- Virtual private cloud on 2 availability zones to hold your services     [Complete]
- Virtual private cloud on 2 availability zones to hold your services     [Complete]
  - Internet gateway to connect the network to the internet               [Complete]
  - Public subnets for internet facing services                           [Complete]
  - Private subnets for services that can't be reached from the internet  [Complete]
  - Routing tables for services to talk with each other                   [Complete]
- ECS Cluster to hold your services                                       [Complete]
- Application load balancer to distribute traffic                         [Complete]
✘ Failed to link account 010101 and region eu-central-1 to application sample-java.

✘ deploy env dev to application sample-java: adding dev environment resources to application: operation 4 for stack set sample-java-infrastructure failed

Did you find this error in the CloudFormation stack?

Aye!

[kohidave]

Super helpful - thank you! A few more questions that maybe you can help me out with. Are you trying to spin up this environment in eu-central-1 (is that why you're using AWS_DEFAULT_REGION?)

Second, which region is the stack sample-java-infrastructure-roles in?

[hnrc]

Are you trying to spin up this environment in eu-central-1 (is that why you're using AWS_DEFAULT_REGION?)

Correct.

which region is the stack sample-java-infrastructure-roles in?

It’s also in eu-central-1.

[kohidave]

Ok even more questions for you! (thank you so much for your time! I realize this is super inconvenient for you).

Did you happen to use the additional --resource-tags feature when you set up your app the first time? If you can't remember, you can look at the tags that are part of your CloudFormation stack.

I was able to reproduce it. Let me dig a little deeper and figure out how to help you out!

[kohidave]

So after investigating this, I can confirm that adding the kms:UntagResource to the key policy works, but I'm still not sure why CloudFormation thinks it needs to untag resources. In fact, even after adding UntagResource to the KMS key policy, no tags were removed.

Super interesting, I'll reach out to the KMS/CloudFormation team to dive deeper into what's going on (but checking CloudTrail it looks like there are a few empty tag/untag requests... hmm!). In the mean time, there isn't a great remediation story I can help you with. You can add the policy to your KMS key, but that value will be overwritten when the StackSet is updated. I'll fork and build some binaries with a fix patched in that you can use - but I'll have to do that tomorrow.

[hnrc]

Did you happen to use the additional --resource-tags feature when you set up your app the first time?

I did, yep :)

I'll fork and build some binaries with a fix patched in that you can use - but I'll have to do that tomorrow.

No rush. This is just a toy app I'm using to figure out a good workflow. I can just recreate it without the resource tags.

Thank you so much for your support and the work you're doing here. Everything looks very very promising!

[hnrc]

I tried setting everything up from scratch without --resource-tags (and even in another set of AWS accounts) but I weirdly still ran into the same issue with kms:UntagResource.

$ AWS_PROFILE=non-default-profile copilot init --app hello-copilot-001
(...)
All right, you're all set for local development.
Deploy: No

$ AWS_PROFILE=non-default-profile copilot env init --name dev --profile non-default-profile --app hello-copilot-001
(...)
✔ Created environment dev in region eu-central-1 under application hello-copilot-001.

$ AWS_PROFILE=non-default-profile copilot env init --name prod --profile another-profile --app hello-copilot-001
(...)
✘ deploy env prod to application hello-copilot-001: adding prod environment resources to application: operation 3 for stack set hello-copilot-001-infrastructure failed

[kohidave]

Thank you for your patience again. So we dug into this - and it looks like this was a recent change introduced to the KMS CloudFormation resource. We spoke with them and they're in the process of fixing it 🎊 Thank you so much in helping us find this!

In the mean time, we've merged our own fix into Copilot.

[hnrc]

Beautiful, thanks @kohidave 🎉