Cloudformation-coverage-roadmap: Support Functions as DeletionPolicy/UpdateReplacePolicy Values

[jthomerson]

Currently the DeletionPolicy attribute for any resource can only be a string. This means we can't use functions like !If [ IsProduction, 'Retain', 'Delete' ].

I'd expect to be able to use conditions and other CloudFormation functions as a value for a DeletionPolicy the same as I can elsewhere in the template.

Others who have asked the same thing:

• https://forums.aws.amazon.com/message.jspa?messageID=560586
• https://github.com/aws/aws-cli/issues/3825

[bdrx312]

It would be nice if the DeletionPolicy also accepted a parameter from the Parameters section although you could work around that if DeletionPolicy at least supported !If

[PatMyron]

One workaround is having multiple duplicate resources with different policies and using Conditions to create only the resources with the correct policies if anyone needs a workaround until this is properly supported

[jthomerson]

Yeah, @PatMyron, that works, but I'm sure you'd agree that "copy and paste your resources and hope you always remember to keep them in sync" is a pretty bad pattern. 😜 Glad you added "until this is properly supported" ... that gives me hope!

[bdrx312]

That work around is really bad if you have an resources that have dependencies (DependsOn) on the resources.

[danieljamesscott]

I think that a better work around is to use AWS::Include and parameterise the name of the included snippet:

"Fn::Transform" : {
  "Name" : "AWS::Include",
  "Parameters" : {
    "Location" : {"Fn::Sub": "s3://${S3Bucket}/deletionPolicy/${RDSDeletionPolicy}.json" }
  }
}

It does mean you have to have snippets for all possible values (and a separate set for UpdateReplacePolicy), but better than duplicating resources.

[PatMyron]

@danieljamesscott due to the regional availability of AWS::Include, I've used Jinja to avoid maintaining duplicate source code

[bdrx312]

The AWS::Include that @danieljamesscott mentioned is a better work around than having duplicate resources but as @PatMyron mentioned is not available in all regions.

[danieljamesscott]

I've also just discovered that you can't use that work around for multiple policies (DeletionPolicy and UpdateReplacePolicy for example) because it results in duplicate keys in the JSON. The work around for that is to generate transforms with each combination of the 3 possibilities.

If AWS would just let us parameterise the policies it would save all this craziness... :)

[PatMyron]

@danieljamesscott the duplicate resources should have different Logical IDs
As @bdrx312 mentioned, that workaround can get painful if you have resources with DependsOn because of that

And very much agreed, the temporary workaround was not to say this isn't still one of the most important feature requests :)

[danieljamesscott]

@PatMyron I'll admit that I haven't tested it for 'real' (my editor and cfn-lint complained) but is this really valid CF?

"S3Bucket": {
  "Type": "AWS::S3::Bucket",
  "Properties": {
    "BucketName": "dan-test-bucket4",
    "VersioningConfiguration": {
      "Status": "Suspended"
    }
  },
  "Fn::Transform" : {
    "Name" : "AWS::Include",
    "Parameters" : {
      "Location" : {"Fn::Sub": "s3://cloudformation-partials/deletePolicy/${DeletePolicy}.json" }
    }
  },
  "Fn::Transform" : {
    "Name" : "AWS::Include",
    "Parameters" : {
      "Location" : {"Fn::Sub": "s3://cloudformation-partials/updateReplacePolicy/${UpdateReplacePolicy}.json" }
    }
  }
}

[mrowles]

This has a high level of user experience, automation, cost and security implications. This was first raised in 2014 and we, as paying customers, still have no way to set DeletionPolicy dynamically.

[joelb01-github]

Has there been any feedback from AWS on this topic and at least a tentative target date for completion?

[zetas]

@PatMyron I'll admit that I haven't tested it for 'real' (my editor and cfn-lint complained) but is this really valid CF?

"S3Bucket": {
  "Type": "AWS::S3::Bucket",
  "Properties": {
    "BucketName": "dan-test-bucket4",
    "VersioningConfiguration": {
      "Status": "Suspended"
    }
  },
  "Fn::Transform" : {
    "Name" : "AWS::Include",
    "Parameters" : {
      "Location" : {"Fn::Sub": "s3://cloudformation-partials/deletePolicy/${DeletePolicy}.json" }
    }
  },
  "Fn::Transform" : {
    "Name" : "AWS::Include",
    "Parameters" : {
      "Location" : {"Fn::Sub": "s3://cloudformation-partials/updateReplacePolicy/${UpdateReplacePolicy}.json" }
    }
  }
}

I'm not sure if using 2 transforms like this is necessary or works, I'm doing something similar and combining both policies in one include like so:

Resources:
  Instance:
    Type: 'AWS::RDS::DBInstance'
    Fn::Transform:
      Name: AWS::Include
      Parameters:
        Location: !Sub 's3://${S3SourceBucketName}/${CPVersion}/templates/customer/rds-instance-${StackType}-policy.yaml'
    Properties:
      Engine: MySQL
      EngineVersion: '5.7'
...

and then in the rds-instance-dev-policy.yaml:

DeletionPolicy: Delete
UpdateReplacePolicy: Delete

and all seems to work fine. I am however getting an error with cfn-lint: E3001 Invalid resource attribute Fn::Transform for resource Instance rds.yaml:64:5 which is super annoying because I use taskcat for CF testing and it relies on the cfn linter under the hood so I can't complete my CI pipeline now because it fails the linter. I've posted about it on their issue tracker here but I think this is an aws-cfn-bootstrap issue which is why I'm posting here. I hope someone with knowledge with that package can read this and help.

Also as an aside in that issue I linked there is the processed json template directly from the CF ui so you can see definitively that those directives are being included properly.

[PatMyron]

@zetas AWS::Include transform isn't fully supported by the Linter yet, but can workaround with resource-level ignores:

  Instance:
    Type: AWS::RDS::DBInstance
    Metadata:
      cfn-lint:
        config:
          ignore_checks:
          - E3001

[zetas]

@PatMyron omg you are amazing, thank you so much! That worked great.

[nefkor]

It would be important to fix this issue soon, since many customer projects required parametrize the deletion policies, and making "copies" with conditional is not an option when cfn template increases in size beyond the limit

[mariaines]

Kinda hacky workaround: specify 'Retain' for your DeletionPolicy, then use a custom resource to delete the resource(s) on stack deletion when desired. This avoids duplication and works without any transforms.

E.g.:

  Table:
    Type: AWS::DynamoDB::Table
    DeletionPolicy: Retain
    ...
  CleanupTableOnStackDelete:
    Type: AWS::Serverless::Function
    Condition: ShouldDeleteTableOnStackDelete
    Properties:
      InlineCode:
        !Sub |
          import json, boto3, logging, uuid
          import cfnresponse
          logger = logging.getLogger()
          logger.setLevel(logging.INFO)

          def lambda_handler(event, context):
              logger.info("event: {}".format(event))
              try:
                  tableName = event['ResourceProperties']['TableName']
                  logger.info("tableName: {}, event['RequestType']: {}".format(tableName,event['RequestType']))
                  if event['RequestType'] == 'Delete':
                      dynamodb = boto3.client('dynamodb')
                      dynamodb.delete_table(TableName=tableName)

                  sendResponseCfn(event, context, cfnresponse.SUCCESS)
              except Exception as e:
                  logger.info("Exception: {}".format(e))
                  sendResponseCfn(event, context, cfnresponse.FAILED)

          def sendResponseCfn(event, context, responseStatus):
              responseData = {}
              responseData['Data'] = {}
              physicalResourceId = event.get('PhysicalResourceId', str(uuid.uuid4()))
              cfnresponse.send(event, context, responseStatus, responseData, physicalResourceId)

      Handler: "index.lambda_handler"
      Runtime: python3.8
      MemorySize: 128
      Timeout: 60
      Role: "arn:aws:iam::...:role/DeleteTestTableRole"
  DoCleanupTableOnStackDelete:
    Type: Custom::CleanupTableOnStackDelete
    Condition: ShouldDeleteTableOnStackDelete
    Properties:
      ServiceToken: !GetAtt CleanupTableOnStackDelete.Arn
      TableName: !Ref Table
    DependsOn:
      - Table
      - CleanupTableOnStackDelete