Aws-cdk: "Stack [stack-id] does not exist" while creating a new stack

[Dzhuneyt]

I run a cdk deploy '*' command to deploy multiple stacks that depend on each other. All of the previous stacks have deployed successfully, however, when I attempt to add a new CDK stack, occasionally (actually quite rarely) I encounter the following error:

 ❌  Attendor-dzhuneyt-redis failed: Error [ValidationError]: Stack [Attendor-dzhuneyt-redis] does not exist
    at Request.extractError (/Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/request.js:685:12)
    at Request.callListeners (/Users/dzhuneytahmed/PhpstormProjects/attendor/cdk/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  message: 'Stack [Attendor-dzhuneyt-redis] does not exist',
  code: 'ValidationError',
  time: 2020-04-14T08:49:45.963Z,
  requestId: 'a6cbcb26-7a34-44d1-abc8-d7a3b4e4073e',
  statusCode: 400,
  retryable: false,
  retryDelay: 234.4330453810892
}
Stack [Attendor-dzhuneyt-redis] does not exist

Let me emphasise again that this stack did not currently exist prior to running the deployment command. If I retry the command, it will work.

Reproduction Steps

Create a couple of CDK stacks that depend on each other and then create another stack that depends on one of the previously created ones.

Environment

• CLI Version : 1.32.2 (build e19e206)
• Framework Version: Not sure what this is
• OS : macOS Catalina
• Language : TypeScript

---

This is :bug: Bug Report

[winjer]

I just experienced this and found that the stack had been created, probably by an earlier invocation, but was in the "review" state because a changeset hadn't been applied.

Deleting the stack then rerunning cdk resolved this.

The error message could be improved.

[shivlaks]

@winjer - any chance it was in review because it was deployed with --no-execute flag? That would essentially create the changeset but would not apply it without manual intervention.

[shivlaks]

@Dzhuneyt I just tried to create a couple of stacks that were dependent on each other. Added another one after but didn't encounter the issue. Did they need to be deployed / partially deployed before the failure scenario materializes?

Marking this as a p2 for now, will bump up if it happens any-time a new dependent stack is added

[Dzhuneyt]

I also do not encounter it usually following the same steps. It's quite sporadic. Any chance I can check some low level logs when it happens? Where do I look?

[magJ]

I ran into this after running a --no-execute deployment.
Very confusing error message.
Deleting the review in progress stack fixes it.

Ironic that the solution to an error saying that a stack does not exist, is to delete the stack that exists.

[eladb]

This happens when the first rollback fails and the stack is in a ROLLBACK_FAILED state.

Here is another example:

 ❌  OidcStack6 failed: Error: The stack named OidcStack6 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_FAILED (The following resource(s) failed to delete: [ConnectMyOidcProviderC712F0F5]. )
The stack named OidcStack6 failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_FAILED (The following resource(s) failed to delete: [ConnectMyOidcProviderC712F0F5]. )

~/playground/oidc master* 1m 1s
❯ cdk deploy                                                                                                                                                                 ✘ 1
OidcStack6: deploying...
OidcStack6: creating CloudFormation changeset...

 ❌  OidcStack6 failed: ValidationError: Stack [OidcStack6] does not exist
Stack [OidcStack6] does not exist

We need a special case in the CLI for this.

[eladb]

BTW, a subsequent cdk deploy works.

[eladb]

OK, this gets interesting. If the stack creation fails with ROLLBACK_COMPLETE, a subsequent deploy will no-op with the following output:

❯ cdk deploy                                                                                                                                                                 ✘ 1
OidcStack6: deploying...

 ✅  OidcStack6 (no changes)

This is because our optimization that checks if the template changed does not take into account the state status. So it downloads the template and determines that nothing has changed.

This can be worked around using --force.

[Wingjam]

I ran into this after running a --no-execute deployment.
Very confusing error message.
Deleting the review in progress stack fixes it.

Ironic that the solution to an error saying that a stack does not exist, is to delete the stack that exists.

Same for me 🙂

[shivlaks]

@Wingjam @magJ @winjer I believe the issue you're running into is fixed by #7731

tl;dr we are trying to update ChangeSet, but since the stack has never been deployed.
The stack has no template and no resources and is in REVIEW_IN_PROGRESS STATE,
CloudFormation does not consider it to be a stack that exists.

So we need to create a new ChangeSet as opposed to update in that scenario.

@Dzhuneyt is your issue also with stacks that are in REVIEW_IN_PROGRESS status?

@eladb I haven't worked through your comments, but there might be more than one issue here...

[Dzhuneyt]

@shivlaks haven't really encountered the issue these days, given I don't create new CDK stacks all the time, but form what I remember from checking the CloudFormation console (UI) - the stack was not present there at all, not in a REVIEW_IN_PROGRESS state. Unless the UI "hides" stacks in this status?

[neg3ntropy]

I have set my stack names to some pretty strings and I get this error from time to time. Relaunching cdk immediatly after then works.

[shivlaks]

@Dzhuneyt @neg3ntropy @eladb the "does not exist" issue was addressed in #8164

The minimal repro is to create a stack:

export class Route53HostedzoneStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new r53.HostedZone(this, 'my-hosted-blar', {
      zoneName: 'again',
    });
  }
}

this will fail and end up in ROLLBACK_COMPLETE

two bugs that would be present on a subsequent deploy:
1.) when trying to deploy again, the 'skip deploy' optimization will skip the deploy and say everything was successful (which is incorrect)
2.) when making a change i.e. zoneName: another-attempt, this would try and create a changeSet against the stack that was deleted and get hit with an error like

Stack [Route53HostedzoneStack] does not exist

It would then work on a subsequent deploy (or if the --force flag was provided).

With the changes introduced in #8164
we will now output as a verbose log

Found existing stack Route53HostedzoneStack that had previously failed creation. Deleting it before attempting to re-create it.

It will then delete the stack, and then attempt to create the changeset.
It will be included in the next release (likely 1.42.0)

[shivlaks]

1.42.0 is released which addresses the outstanding items in this issue.
Feel free to reopen if the issue persists.