Aws-cdk: Upgrading to v1.19.0 causes errors with EIP tag update

Created on 18 Dec 2019 · 9Comments · Source: aws/aws-cdk

After upgraded to CDK v1.19.0, updating existing EIP for adding auto-generated Tag causes an error at CloudFormation

Reproduction Steps

Deploy stack with Vpc construct with CDK earlier v1.19.0
Upgrade to CDK v1.19.0 and deploy

Error Log

$ cdk deploy NetworkStackEnvStg

NetworkStackEnvStg 
NetworkStackEnvStg: deploying... 
NetworkStackEnvStg: creating CloudFormation changeset... 
 0/6 | 2:07:22 AM | UPDATE_IN_PROGRESS   | AWS::EC2::EIP                         | Vpc/PublicSubnet1/EIP (VpcPublicSubnet1EIPD7E02669)  
 1/6 | 2:07:24 AM | UPDATE_FAILED        | AWS::EC2::EIP                         | Vpc/PublicSubnet1/EIP (VpcPublicSubnet1EIPD7E02669) API: ec2:disassociateAddress You do not have permission to access the specified resource. 
    PublicSubnet.addNatGateway (/codebuild/output/src566679633/src/node_modules/@aws-cdk/aws-ec2/lib/vpc.ts:1538:21) 
    \_ NatGateway.configureNat (/codebuild/output/src566679633/src/node_modules/@aws-cdk/aws-ec2/lib/nat.ts:145:27) 
    \_ Vpc.createNatGateways (/codebuild/output/src566679633/src/node_modules/@aws-cdk/aws-ec2/lib/vpc.ts:1129:14) 
    \_ new Vpc (/codebuild/output/src566679633/src/node_modules/@aws-cdk/aws-ec2/lib/vpc.ts:1044:14) 
    \_ new NetworkStack (/codebuild/output/src566679633/src/src/network-stack.ts:20:16) 
    \_ newStack (/codebuild/output/src566679633/src/src/app.ts:52:28) 
    \_ Records.mapValues (/codebuild/output/src566679633/src/src/lib.ts:62:5) 
    \_ result.toEntries.map (/codebuild/output/src566679633/src/src/lib.ts:44:31) 
    \_ Array.map (<anonymous>) 
    \_ Function.mapValues (/codebuild/output/src566679633/src/src/lib.ts:43:43) 
    \_ Object.newStackGroup (/codebuild/output/src566679633/src/src/lib.ts:61:18) 
    \_ Object.<anonymous> (/codebuild/output/src566679633/src/src/app.ts:49:27) 
    \_ Module._compile (internal/modules/cjs/loader.js:778:30) 
    \_ Module.m._compile (/codebuild/output/src566679633/src/node_modules/ts-node/src/index.ts:536:23) 
    \_ Module._extensions..js (internal/modules/cjs/loader.js:789:10) 
    \_ Object.require.extensions.(anonymous function) [as .ts] (/codebuild/output/src566679633/src/node_modules/ts-node/src/index.ts:539:12) 
    \_ Module.load (internal/modules/cjs/loader.js:653:32) 
    \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12) 
    \_ Function.Module._load (internal/modules/cjs/loader.js:585:3) 
    \_ Function.Module.runMain (internal/modules/cjs/loader.js:831:12) 
    \_ main (/codebuild/output/src566679633/src/node_modules/ts-node/src/bin.ts:212:14) 
    \_ Object.<anonymous> (/codebuild/output/src566679633/src/node_modules/ts-node/src/bin.ts:470:3) 
    \_ Module._compile (internal/modules/cjs/loader.js:778:30) 
    \_ Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10) 
    \_ Module.load (internal/modules/cjs/loader.js:653:32) 
    \_ tryModuleLoad (internal/modules/cjs/loader.js:593:12) 
    \_ Function.Module._load (internal/modules/cjs/loader.js:585:3) 
    \_ Function.Module.runMain (internal/modules/cjs/loader.js:831:12) 
    \_ startup (internal/bootstrap/node.js:283:19) 
    \_ bootstrapNodeJSCore (internal/bootstrap/node.js:622:3) 
 1/6 | 2:07:25 AM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack            | NetworkStackEnvStg The following resource(s) failed to update: [VpcPublicSubnet1EIPD7E02669].  
 1/6 | 2:07:57 AM | UPDATE_IN_PROGRESS   | AWS::EC2::EIP                         | Vpc/PublicSubnet1/EIP (VpcPublicSubnet1EIPD7E02669)  

 ❌  NetworkStackEnvStg failed: Error: The stack named NetworkStackEnvStg is in a failed state: UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [VpcPublicSubnet1EIPD7E02669]. ) 
The stack named NetworkStackEnvStg is in a failed state: UPDATE_ROLLBACK_FAILED (The following resource(s) failed to update: [VpcPublicSubnet1EIPD7E02669]. ) 
error Command failed with exit code 1. 
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Environment

CLI Version : 1.19.0 (build 5597bbe)
Framework Version: 1.19.0
OS : Linux(on a CodeBuild project environment)
Language : TypeScript

Other

$ cdk diff NetworkStackEnvStg

Stack NetworkStackEnvStg 
Resources 
[~] AWS::EC2::EIP Vpc/PublicSubnet1/EIP VpcPublicSubnet1EIPD7E02669  
 └─ [+] Tags 
     └─ [{"Key":"Name","Value":"NetworkStackEnvStg/Vpc/PublicSubnet1"}]

This is :bug: Bug Report

@aws-cdaws-ec2 bug efforlarge efformedium p1

Source

idm-ryou

👍12

Most helpful comment

Workaround:

cdk.Tag.remove(this.vpc, "Name", {
  includeResourceTypes: ["AWS::EC2::EIP"]
});

idm-ryou on 18 Dec 2019

👍14

All 9 comments

Workaround:

cdk.Tag.remove(this.vpc, "Name", {
  includeResourceTypes: ["AWS::EC2::EIP"]
});

idm-ryou on 18 Dec 2019

👍14

I am having the same issue. After this upgrade, my VPC subnets are adding a new tag to each subnet. I am having the:
API: ec2:disassociateAddress You do not have permission to access the specified resource.

The rollback then proceeds to fail as each subnet fails to update.

danly on 18 Dec 2019

👍5

"UPDATE_ROLLBACK_FAILED state and can not be updated" I had to initiate "Continue update rollback" from the console. I had to skip changing those resources.

So now they are out of sync.

I still get the same API: ec2:disassociateAddress error on subsequent changes. With the workaround https://github.com/aws/aws-cdk/issues/5469#issuecomment-566846441 at least it can roll back.

So I will return to 1.18.0 for now.

crucialfelix on 18 Dec 2019

👍2

I am also seeing this bug after upgrading to 1.19.0

gbooth27 on 18 Dec 2019

I did get it to deploy still using 1.19.0 actually.

You also have to remove any other tags that might be assigned by parent constructs. In my case I had source: cdk and stage: staging so I now have code to remove all tags from EIP:

cdk.Tag.remove(this.vpc, "Name", {
  includeResourceTypes: ["AWS::EC2::EIP"]
});

cdk.Tag.remove(this.vpc, "source", {
  includeResourceTypes: ["AWS::EC2::EIP"]
});

cdk.Tag.remove(this.vpc, "stage", {
  includeResourceTypes: ["AWS::EC2::EIP"]
});

crucialfelix on 18 Dec 2019

👍2

idm-ryou on 19 Dec 2019

👍1

I am also seeing this bug after upgrading to 1.19.0.

Update 1: Confirming the recommendation by @idm-ryou to manually remove the tag is a good temp workaround.

Update 2: In complex environments, this is causing cascading errors for all tagged resources. I've reached a point where a multi-VPC application cannot be deployed even when forcibly removing all tags attached with the following code:

 Tag.remove(vpc, "Name", {
        includeResourceTypes: [],
        priority: 300
});

Update 3: In complex or multi-stack environments, I found the easiest way to find all issues with EIP tags is to run the command cdk diff in your local project. In my case, I found that an EKS cluster further downstream was also modifying custom k8s tags on the Subnet, inadvertently impacting the EIP resource.