Aws-cdk: [CLI] cdk deploy doesn't work after force upgrading to the new bootstrap version

[jonny-rimek]

I upgraded to cdk 1.61 and got asked to upgrade my bootstrap version from 3 to 4, which at first failed as described here https://github.com/aws/aws-cdk/issues/10016, only I didn't wait for the fix and force upgraded.

If I run cdk diff I get the following output (with out the changes I made, only the ones relating to the bootstrap):
````
Stack wm
IAM Statement Changes
┌───┬──────────────────────────────────────────┬────────┬──────────────────────────────────────────┬────────────────────────────────────────────┬───────────┐
│ │ Resource │ Effect │ Action │ Principal │ Condition │
├───┼──────────────────────────────────────────┼────────┼──────────────────────────────────────────┼────────────────────────────────────────────┼───────────┤
│ - │ {"Fn::ImportValue":"CdkBootstrap-hnb659f │ Allow │ kms:Decrypt │ AWS:${Custom::CDKBucketDeployment8693BB649 │ │
│ │ ds-FileAssetKeyArn"} │ │ kms:DescribeKey │ 68944B69AAFB0CC9EB8756C/ServiceRole} │ │
└───┴──────────────────────────────────────────┴────────┴──────────────────────────────────────────┴────────────────────────────────────────────┴───────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Parameters
[+] Parameter BootstrapVersion BootstrapVersion: {"Type":"AWS::SSM::Parameter::Value","Default":"/cdk-bootstrap/hnb659fds/version","Description":"Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store."}

Resources
[~] AWS::IAM::Policy Custom::CDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C/ServiceRole/DefaultPolicy CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRoleDefaultPolicy88902FDF
└─ [~] PolicyDocument
└─ [~] .Statement:
└─ @@ -35,16 +35,6 @@
[ ] },
[ ] {
[ ] "Action": [
[-] "kms:Decrypt",
[-] "kms:DescribeKey"
[-] ],
[-] "Effect": "Allow",
[-] "Resource": {
[-] "Fn::ImportValue": "CdkBootstrap-hnb659fds-FileAssetKeyArn"
[-] }
[-] },
[-] {
[-] "Action": [
[ ] "s3:GetObject",
[ ] "s3:GetBucket",
[ ] "s3:List*",
[~] AWS::Lambda::Function LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8a LogRetentionaae0aa3c5b4d4f87b02d85b201efdd8aFD4BFC8A
└─ [~] Metadata
├─ [-] Removed: .aws:asset:path
└─ [-] Removed: .aws:asset:property

Other Changes
[+] Unknown Rules: {"CheckBootstrapVersion":{"Assertions":[{"Assert":{"Fn::Not":[{"Fn::Contains":[["1","2","3"],{"Ref":"BootstrapVersion"}]}]},"AssertDescription":"CDK bootstrap stack version 4 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."}]}}
````

when I run cdk deploy I get the following error message:
````

❌ wm failed: Error [ValidationError]: AccessDenied. User doesn't have permission to call ssm:GetParameters
at Request.extractError (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
at Request.callListeners (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/usr/lib/node_modules/aws-cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'ValidationError',
time: 2020-08-28T15:20:02.851Z,
requestId: '629a101e-a63a-43c7-8f45-dc70bbeba284',
statusCode: 400,
retryable: false,
retryDelay: 743.7164270441746
}
AccessDenied. User doesn't have permission to call ssm:GetParameters
````

Environment

• **CLI Version : 1.16.1
• **Framework Version: 1.16.1
• **Node.js Version: v12.18.3
• **OS : ubuntu 20.4
• **Language (Version): ts

This is :bug: Bug Report

[shivlaks]

@jonny-rimek how did you force the upgrade?

npx cdk bootstrap \
  --profile account1-profile \
  --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess \
  aws://ACCOUNT1/<region>

with a --force added to it? it'd be good to see what your CloudFormation execution role looks like because excluding it could result in access denied errors.

[jonny-rimek]

I simply ran cdk bootstrap --force My default profile I use in aws cli and that cdk uses has the __AdministratorAccess__ managed policy

[shivlaks]

you might want to try running bootstrap again (without the force and with cloudformation-execution-policies created)

Your CdkToolkit (or whatever you named your bootstrap stack) should have a resource called CloudFormationExecutionRole - can you check the AdministratorAccess policy is attached to it?

I'm guessing it's empty. We should probably add validation here to prevent users from getting into this scenario.

You can run bootstrap again to get that updated

cdk bootstrap --context @aws-cdk/core:newStyleStackSynthesis=1 --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://accountid/region

Your cdk deploy should work after that. let me know how it goes!

[jonny-rimek]

My CloudFormationExecutionRole did indeed not have any policy attached to it running cdk bootstrap --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess fixed it.

thanks for your help