Copilot-cli: Using ResourceTag with iam:PassRole conditions prevents env interaction

[miketheman]

(originally reported as part of a Discussion thread)

After deploying to an environment with copilot svc deploy --name app I was unable to remove the env that was created, getting this IAM response:

Are you sure you want to delete environment load from application app1? Yes
✘ Failed to delete environment load from application app1.
✘ delete environment load stack: delete stack app1-load: AccessDenied: User: arn:aws:sts::[REDACTED]:assumed-role/app1-load-EnvManagerRole/1617489726744667000 is not authorized to perform: iam:PassRole on resource: arn:aws:iam::[REDACTED]:role/app1-load-CFNExecutionRole
status code: 403, request id: 0a03c9fd-6fb8-436b-92ec-60144cf2a58e

After some further debugging, I found:

The iam:PassRole permission in the EnvManagerRole policy specifies:

{
    "Condition": {
        "StringEquals": {
            "iam:ResourceTag/copilot-application": "[MY APP NAME]",
            "iam:ResourceTag/copilot-environment": "[MY ENV NAME]"
        }
    },
    "Action": [
        "iam:GetRole",
        "iam:PassRole"
    ],
    "Resource": "*",
    "Effect": "Allow",
    "Sid": "GetAndPassCopilotRoles"
},

When looking at the CFNExecutionRole, there are no Tags associated with the role - so the condition could not be satisfied.

I then tried to manually add both conditions to the Tags on CFNExecutionRole, but got the same error response.

Reading further, the AWS IAM Docs also specifically call out to NOT use ResourceTags for PassRole:

Do not use the ResourceTag condition key in a policy with the iam:PassRole action.
You cannot use the tag on an IAM role to control access to who can pass that role.

I ended up manually removing the condition from the policy in the IAM console to be able to complete the env removal.

So it seems like there's a problem with how we're intending on restricting access to the PassRole condition?

[iamhopaul123]

Hello @miketheman. Thank you for documenting this unexpected behavior!

From the error i see

User: arn:aws:sts::[REDACTED]:assumed-role/app1-load-EnvManagerRole/1617489726744667000 is not authorized to perform: iam:PassRole on resource: arn:aws:iam::[REDACTED]:role/app1-load-CFNExecutionRole

It seems like the envManagerRole cannot perform iam:PassRole to CFNExecutionRole. So I guess for whatever reason the app1-load-CFNExecutionRole isn't tagged correctly. I think that's the reason why it failed. Basically according to my test you have to add the tag in this order:

copilot-environment prod
copilot-application couch-demo

If you swap those two tags upside down then it's not gonna work. I think it is a very strange behavior on IAM side and that's probably part of the reason why they don't recommend to use ResourceTag on iam:PassRole. Please keep tracking this issue. We should probably follow the IAM recommendation to ditch the condition. Again, thank you so much for reporting this issue.

[iamhopaul123]

Hello @miketheman. Also do you know the reason why those tags for the IAM roles did not exist? Because even if we update the permission, without those tags Copilot still won't work for the other commands, since there's no limitation for ResourceTag with the other actions.

[miketheman]

Hey @iamhopaul123 !

do you know the reason why those tags for the IAM roles did not exist?

I tried again today, and with a basic copilot init, the autogenerated test environment doesn't have the Resource Tags either.

The Cloudformation Stack shows the correct Tags that are supposed to be inherited by any resource created by the CFN template. Other resources, like VPC, show these two tags, as well as aws:cloudformation:stack-name, aws:cloudformation:stack-id, and aws:cloudformation:logical-id - very in line with CFN expected behavior.

Looks like this may be related? https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/328

Does this not reproduce for you?

[miketheman]

And to the Tag order part - I logged into the account's IAM privileged user account (yay safety!) and manually applied the two tags to the CFNExecutionRole in the order you specified.
I was then able to successfully delete the stack with copilot app delete (I'm guessing that copilot env delete ... would have worked as well) - so at least that is a possible.

[miketheman]

I continued to test this, and it (should have been obvious, but wasn't 🤦 ) prevents any interactions that would use PassRole, like copilot svc deploy.... too. I've updated the description to reflect that finding.
Similarly, adding the Resources Tags manually (in the correct order) unblocks the execution.

[iamhopaul123]

Does this not reproduce for you?

Unfortunately I am not able to reproduce this one and it seems like you are the first user reporting this issue (IAM roles produced by CFN do not inherit tags from CFN). Does it happen every time when you create an environment? Also thank you so much for your efforts for investigating this!

[miketheman]

@iamhopaul123 Did you happen to dig into the linked issue from the cloudformation link above?

My understanding is that my user creates a role, and that role creates new IAM resources - and based on the CFN bug link, a role creating IAM resources via CFN won't tag-on-create.

It's pretty consistent - I create the app, then I create an env, and then we see the issue. If there's some other debugging I can provide, let me know, and I'll do what I can to produce!

[iamhopaul123]

Oh I see what you meant. Did you create the environment using temporary credentials of an IAM role?

[miketheman]

No, the credentials are (I think) long-lived IAM user access key/secret - I don't think the linked cloudformation issue specifies that it applies to temp credentials, does it?