Cloud-custodian: Only one periodic policy per .yaml file?

[lithiumoxide]

I'm not sure if this is a bug or if I'm just misunderstanding the functionality, so I just want to lay out what I'm seeing.

Short version

In short, if I have multiple periodic policies in a .yaml file under policies, only one Lambda function gets created. If I split all the policies into their own .yaml files (i.e. one policy per policies heading) I get the requisite number of Lambda functions deployed.

Longer version

Let's call my yaml file _my-policies.yaml_, which was laid out something like this (I've stripped out the details, but I'll go into those later on for some extra context):

---
policies:
  - name: policy-1
    description: This is policy number 1
    mode:
      type: periodic
      schedule: rate(1 hour)
      role: arn-to-lambda-role
    resource: security-group
    filters:
      - filters-in-here
    actions:
      - actions-in-here

  - name: policy-2
    description: This is policy number 2
    mode:
      type: periodic
      schedule: rate(1 hour)
      role: arn-to-lambda-role
    resource: security-group
    filters:
      - filters-in-here
    actions:
      - actions-in-here

  - name: policy-3
    description: This is policy number 3
    mode:
      type: periodic
      schedule: rate(1 hour)
      role: arn-to-lambda-role
    resource: security-group
    filters:
      - filters-in-here
    actions:
      - actions-in-here

Cloud Custodian seemed to run this without issue, but when I inspected the outcome, I could only find a Lambda function for _policy-3_. I redeployed a number of times and this seemed to happen consistently.

Next, I split the policies into three yaml files, let's call them _policy1.yaml_, _policy2.yaml_, _policy3.yaml_. When I ran Custodian against these, all three Lambda functions are successfully deployed.

The question

My question here is: why is only the last policy deployed when all three policies are contained within one yaml file? My expectation here was that there would have been three Lambda functions deployed as there are three policies.

Context

I gave a generalised outline above of the issue I'm seeing. I don't think the context will have an impact on the issue above but I just want to outline it just in case.

In more specific terms, I'm experimenting with using delayed operations (mark-for-op and marked-for-op) as outlined in this doc. The doc is for Azure, and I'm using AWS: again, I'm really just playing at this point to see if it works with AWS before trying another method. You will see in the doc that there are three policies under the policies header. I based my policies off this outline to use the periodic mode, hence why I'm expecting three Lambda functions.

My use case is to find unused security groups, tag them as unused, untag them if they're later found to be unused, then delete any that are still not used after a period of time. I know the aws.security-group resource has used and unused filters, but in my case there are legitimate cases where a security group is created and left unused for a short period (I'm employing a sandbox environment where people can experiment with AWS), so I don't want to remove unused security groups immediately. As the potential solution from the docs employs three policies, I'd like to maintain them in a single yaml file as they're all closely interconnected.

[PratMis]

You can have as many policies as you like in a policy file. We run hundreds of them per policy file. Few things here:

1. Custodian appends a prefix to the lambda function names defaulting to custodian-{policyname}.
2. Are you seeing any errors in the log as you execute the policy file. Asking because, you would have to configure the IAM roles in such a way that the central role should have the ability to pass in role: arn-to-lambda-role to the lambda. You should be able to see any such failure in the execution logs.
3. you can use mark-for-op in conjunction with unmark/marked-for-op for all the resources that support tagging. custodian schema aws should be a good way to find it. you can have those policies in one file.

[lithiumoxide]

Hi @PratMis, thanks for those pointers.

In the case of point 1, I only end up with one Lambda function for the last policy in the policy file. The other policies in the file have no Lambda functions created.

For point 2, I will need to investigate the logs more closely. However, the central role has no issue in passing the role to the Lambda when the policies are divided out into separate files, so my guess here is that it's fine, but I'll take a closer look to confirm.

On your final point 3, yes that's good to know. I actually got the tests working (when I separated the policies into different files) and mark-for-op works well here.

I use Terraform to grab the policy file names and to construct a custodian run command which is then run in a Terraform null_resource. I'm curious now if there's an issue in my Terraform logic that is causing it to, for some reason, "skip" policies in a file, so I'll investigate this too.

Thanks again for the tips.