Aws-cdk: [cli] bootstrap upgrade to version 4 fails after upgrade from 1.60.0 to 1.61.0

[jacques-]

Bootstrapped using new style CDK bootstrap (BootstrapVersion: 3), then upgraded to CDK 1.61. Got errors like:

❌  InitiativeConfigStack-development failed: Error: InitiativeConfigStack-development: publishing assets requires bootstrap stack version '4', found '3'. Please run 'cdk bootstrap' with a newer CLI version.

Did as instructed in the error and #8678 , but got the following:

env CDK_NEW_BOOTSTRAP=1 npx cdk bootstrap --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess aws://123456789012/eu-west-1
(node:69834) ExperimentalWarning: The fs.promises API is experimental
CDK_NEW_BOOTSTRAP set, using new-style bootstrapping
 ⏳  Bootstrapping environment aws://123456789012/eu-west-1...
 ❌  Environment aws://123456789012/eu-west-1 failed bootstrapping: Error: Not downgrading existing bootstrap stack from version '3' to version '0'. Use --force to force.
Not downgrading existing bootstrap stack from version '3' to version '0'. Use --force to force.

Am I missing something obvious here?

• CLI Version : 1.61.0
• Framework Version: 1.61.0
• Node.js Version: v10.16.3
• OS : mac
• Language (Version):

This is :bug: Bug Report

[tvanhens]

Dug into this a little bit and it looks like the parameter it is looking for is incorrect:

https://github.com/aws/aws-cdk/blob/960ef1237e8379090e78dca554401213c81d2be7/packages/aws-cdk/lib/api/bootstrap/deploy-bootstrap.ts#L64

I believe this should reference BOOTSTRAP_VERSION_RESOURCE instead:

https://github.com/aws/aws-cdk/blob/960ef1237e8379090e78dca554401213c81d2be7/packages/aws-cdk/lib/api/bootstrap/bootstrap-props.ts#L12

That is assuming it is supposed to be referencing this resource from the bootstrap-template:

https://github.com/aws/aws-cdk/blob/960ef1237e8379090e78dca554401213c81d2be7/packages/aws-cdk/lib/api/bootstrap/bootstrap-template.yaml#L358-L364

[polothy]

Reproduced by doing the following:

1. Bootstrapping a new region with CDK 1.60.0
2. Upgraded to 1.61.0
3. Re-run the same bootstrap command

Got the same error as reported in the ticket. I re-reran the bootstrap command with the --force flag and that appeared to upgrade it (saw the new SSM parameter and KMS alias).

[polothy]

It would be good to hear from a project owner if --force is an OK workaround and really the only problem is that the upgrade version checking logic just isn't working correctly, so the bootstrap upgrade is working as intended.

[shivlaks]

looking into it:

minimal repro steps:

Deploy bootstrap version 3

npm install -g [email protected]
cdk bootstrap --context @aws-cdk/core:newStyleStackSynthesis=1

Upgrade & try to deploy bootstrap version 4

npm install -g [email protected]
cdk bootstrap --context @aws-cdk/core:newStyleStackSynthesis=1

will result in:

Environment aws://123456789012/us-east-1 failed bootstrapping: Error: Not downgrading existing bootstrap stack from version '3' to version '0'. Use --force to force.
    at Object.deployBootstrapStack (/usr/local/lib/node_modules/aws-cdk/lib/api/bootstrap/deploy-bootstrap.ts:29:11)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at /usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:360:24
    at async Promise.all (index 0)
    at CdkToolkit.bootstrap (/usr/local/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:357:5)
    at main (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:244:16)
    at initCommandLine (/usr/local/lib/node_modules/aws-cdk/bin/cdk.ts:188:9)
Not downgrading existing bootstrap stack from version '3' to version '0'. Use --force to force.

[shivlaks]

patch to fix bootstrapping issue was released in 1.61.1

[polothy]

Tested and works just fine! Thanks for fix and also releasing so quickly, very much appreciated! 💯

[asterikx]

@shivlaks this is still broken for me. Upgrading from 1.60.0 straight to 1.61.1 fails with the same error message

$ cdk deploy PipelineStack --region eu-central-1
PipelineStack: deploying...

 ❌  PipelineStack failed: Error: PipelineStack: publishing assets requires bootstrap stack version '4', found '3'. Please run 'cdk bootstrap' with a newer CLI version.

CLI and framework version is 1.61.1.

EDIT: could fix it by manually re-running cdk bootstrap in all affected environments as prompted by the error message, but I suspect this shouldn't be required since I was already using the "new" bootstrap style?

[shivlaks]

@asterikx - That's an expected error message and remediation steps for users of the new bootstrap style when upgrading as we've bumped bootstrap version (for new bootstrap users) with some updates to the template.

The issue that required patching was the inability to upgrade the bootstrap stack successfully.

[dhirajkhodade]

• I am using latest version of cdk i.e. 1.64.1
• My cdk.json has this line in context "@aws-cdk/core:newStyleStackSynthesis": true
• I have bootstrapped my target environment properly with admin access policy
• cdk synth command also succeeded

but when I deploy my stack I am getting below validation error

message: 'Parameters [/cdk-bootstrap/hnb659fds/version] referenced by template have types not supported by CloudFormation.',
code: 'ValidationError',
time: 2020-09-27T14:29:39.972Z,
requestId: '80a35b87-36dc-4bb1-b689-3157d9f171a9',
statusCode: 400,
retryable: false,
retryDelay: 278.94358746834325

I am doubtful about below section in my cdk synth command output this section was not there before when I was using aws cdk version 1.53.0

Parameters:
BootstrapVersion:
Type: AWS::SSM::Parameter::Value
Default: /cdk-bootstrap/hnb659fds/version
Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store.
Rules:
CheckBootstrapVersion:
Assertions:
- Assert:
Fn::Not:
- Fn::Contains:
- - "1"
- "2"
- "3"
- Ref: BootstrapVersion
AssertDescription: CDK bootstrap stack version 4 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI