Cloud-custodian: AWS S3 - Feature Request - Reduce Non-Essential API Calls

[jtroberts83]

Jamison Roberts @jtroberts83 12:02
@kapilt Would there be a way to limit what api calls are being made when running an s3 policy? The reason I ask is because we have a few S3 lambda based policies that run on a periodic basis, some as frequently as every 5 minutes for compliance, and even if the policy filters are only checking bucket encryption for example, the policy still queries all the other properties such as lifecycle, versioning, logging, bucket acl etc. This is generating hundreds of thousands of useless api calls and some of our customers are "not happy" about all the Cloud Custodian S3 api clutter in CloudTrail that they have to sift through. Would it be possible for custodian to only make the needed api calls for the filters provided? I know the odds are pretty slim but thought I would ask

Kapil Thangavelu @kapilt 12:16
It’s possible, file an issue with the use case, it would be additional config for the s3 resource policies, but yeah that could be done

My use case for this would be for an S3 policy which enables default bucket encryption for newly created buckets. Per our compliance regulations we have to have all S3 buckets encrypted ASAP after being created as files could start getting uploaded to new buckets within seconds or minutes after creation. We do have a policy that triggers on CreateBucket cloudtrail api and configures several aspects of of the new buckets but we can't apply encryption at the CreateBucket event because several customers had an issue with the policy as they were trying to apply their own custom KMS key encryption which would get overwritten by the S3 Cloud Custodian policy. Being that the CreateBucket event doesn't show if encryption is being enabled by the customer I can't use those events to determine if the customer is applying encryption. Currently I have the CreateBucket triggered policy tag the new buckets for a followup policy to check encryption which runs every 5 minutes and then the followup periodic policy will check the encryption and if still missing it will apply the default encryption. While this works as a solution, it is generating so many useless api calls as it queries every aspect of every bucket every 5 minutes and generating so many CloudTrail events. Limiting these to what the filters need or the users specifies would be nice. Here is the policy set for our usecase:

- name: s3-enable-standards-realtime
  resource: s3
  description: | 
    New S3 buckets get tagged with creators ID, enables versioning,
    enables logging, and it applies a deny public objects bucket policy.
  mode:
    type: cloudtrail
    events:
      - CreateBucket
    role: arn:aws:iam::{account_id}:role/Cloud_Custodian_Role
    timeout: 300
  filters:
    - type: event
      key: "region"
      op: in
      value:
          - us-east-1
          - eu-west-1
  actions:
    - type: auto-tag-user
      tag: CreatorName
    - type: toggle-versioning
      enabled: true
    - type: toggle-logging
      target_bucket: "{account_id}-{region}-s3-logs"
      target_prefix: "{source_bucket_name}/"
    - type: set-statements
      statements:
        - Sid: "DenyS3PublicObjects"
          Effect: "Deny"
          Action: "s3:PutObjectAcl"
          NotPrincipal:
             "AWS":
                 - "arn:aws:iam::001100:root"
                 - "arn:aws:iam::110011:user/logs"
                 - "arn:aws:iam::220022:user/logs"
                 - "arn:aws:iam::330033:root"
                 - "arn:aws:iam::440044:root"
          Resource:
             - "arn:aws:s3:::{bucket_name}/*"
             - "arn:aws:s3:::{bucket_name}"
          Condition:
            StringEqualsIgnoreCaseIfExists:
               's3:x-amz-acl':
                   - "public-read"
                   - "public-read-write"
                   - "authenticated-read"
    - type: tag
      key: c7n_CheckEncryption
      value: "Status"
    - type: tag
      key: c7n_CheckLifeCycle
      value: "Status"


##Followup Policy Generating way too many api calls below

- name: s3-check-encryption-status-tag-disabled
  resource: s3
  description: |
     This is part of a policy chain which checks to see if the bucket
     has encryption enabled and if not it enables AES-256 default key.
  mode:
      type: periodic
      schedule: "rate(5 minutes)"
      role: arn:aws:iam::{account_id}:role/Cloud_Custodian_Role
      timeout: 300
  filters:
    - "tag:c7n_CheckEncryption": present
    - not:
        - type: bucket-encryption
  actions:
    - type: set-bucket-encryption
    - type: unmark
      tags: ['c7n_CheckEncryption']

- name: s3-check-encryption-status-tag-enabled
  resource: s3
  description: |
     This is part of a policy chain which checks to see if the bucket
     has encryption enabled and if so it removed the check tag.
  mode:
      type: periodic
      schedule: "rate(30 minutes)"
      role: arn:aws:iam::{account_id}:role/Cloud_Custodian_Role
      timeout: 300
  filters:
    - "tag:c7n_CheckEncryption": present
    - type: bucket-encryption
  actions:
    - type: unmark
      tags: ['c7n_CheckEncryption']

[kapilt]

so i think we could probably do something here by extending query attribute of the policy, that would allow controlling which subdocuments are fetch. The extant default would remain, but for policies where you want to minimize the call volume, you could control the exact fetch. An alternative here is doing a mode: config-rule which effectively just evaluates ~15m after attribute change, but i think the feature makes sense.

something like

- name: s3-check-encrypted
  resource: s3
  query:
   - documents: [BucketEncryption]
  filters:
   - type: bucket-encryption

[jtroberts83]

yeah that example looks good. I assume if you leave the query item out it will default to all queries correct? That is pretty much what I was thinking. I don't understand the - documents: bit or where the values for the documents come from. I assume those are shortcuts that would be defined in the docs?

[kapilt]

query structure at the moment is array of dictionaries, wrt to the actual key documents its based on the s3 model of bucket attributes as separate documents (s3 control plane api is fairly REST oriented) although we could shorten to docs or attrs perhaps. The values here do need some documentation. The original mapping of s3 document names to custodian attributes was done very early in the project lifecycle (late 2015 if memory servces), and is already a bit arbitrary, on a clean redo we would have just kept the service's document names, unfortunately compatibility constrains us now. We should probably use the c7n names (the example was using the service document name) though since they likely have some familiarity to custodian users, and definitely those names need more explicit documenting.

[jtroberts83]

I'm cool with whatever way you think will work the best. Let me know if you need me to test anything and thanks again!