Cloud-custodian: tests - aws flight recording improvements

[kapilt]

to simplify aws flight recording as part of authoring tests, two additional features should be incorporated

• Automatically convert response metadata to an empty dictionary when saving results to disk
• Automatically replace account ids in recorded responses with known safe/dummy account ids.

[kapilt]

here's a quick standalone script for removing response metadata

import json
import fnmatch
import click
import os

@click.command()
@click.option('--path', type=click.Path())
def main(path):
    """cli tool to process flight recorded data."""

    shrink = 0
    for root, dirs, files in os.walk(path):
        if not files:
            continue
        for f in fnmatch.filter(files, "*json"):
            json_path = os.path.join(root, f)
            with open(json_path) as fh:
                data = json.load(fh)
                size = fh.tell()
            if not data.get('data', {}).get('ResponseMetadata'):
                continue
            data['data']['ResponseMetadata'] = {}
            serialized = json.dumps(data, indent=2)
            print("found aggregate result %s %d->%s" % (
                json_path, size, len(serialized)))
            shrink += size - len(serialized)
            with open(json_path, 'w') as fh:
                fh.write(json.dumps(data, indent=2))
    print("shrunk %d" % shrink)

if __name__ == '__main__':
    main()

[JohnHillegass]

Hey @kapilt, I started looking into this and we can automatically remove responsemetadata be redefining as an empty dict and likely obfuscate accountId by matching for 12 digit numbers in regex but will either need to:

(1) override the placebo save_response method / essentially create our own implementation for PillTest like was done in ZippedPill
(2) add a cleanup method that opens the file after the stop() and makes the edits and dumps it back out

Do you see any other paths forward? I am leaning towards #2 as it would likely be easier and doesn't obsolete the placebo lib as much at that point.

[kapilt]

1. overriding placebo is probably best.
2. cleanup methods rely on unit test, which is effectively a non starter as we're supporting tests using pytest. the other issue its also spooky action at a distance, ie poking at the internals of something else is actually implicit tight coupling that may or may not fail if the internals change.