Cloud-custodian: EBS.modify_volume crashing due to unrelated instances being present

[danmarcab]

Describe the bug
I was trying to call a simple policy (shown below) to modify a volume and run into this issue, custodian crashes and does not execute the action.

To Reproduce

1 Create an instance with an attached volume.
2 Create a volume (4GB gp2 in my case) and leave it unattached.
3 Run the policy shown below (modify the unattached volume to io1)

Expected behavior

The policy executes and the volume starts being modified.

Background (please complete the following information):

• OS: OSX 0.14.6
• Python Version: 3.7.6
• Custodian Version: 0.8.46.1
• Cloud Provider: aws
• Policy:

policies:
  - name: modify_volume_type
    resource: ebs
    filters:
      - VolumeId: vol-066023f2042374d9e
      - modifyable
    actions:
      - type: modify
        volume-type: io1

• Traceback:

Traceback (most recent call last):
  File "/Users/danmarcab/Projects/custodian/custodian/lib/python3.7/site-packages/c7n/cli.py", line 374, in main
    command(config)
  File "/Users/danmarcab/Projects/custodian/custodian/lib/python3.7/site-packages/c7n/commands.py", line 141, in _load_policies
    return f(options, list(policies))
  File "/Users/danmarcab/Projects/custodian/custodian/lib/python3.7/site-packages/c7n/commands.py", line 283, in run
    policy()
  File "/Users/danmarcab/Projects/custodian/custodian/lib/python3.7/site-packages/c7n/policy.py", line 1049, in __call__
    resources = mode.run()
  File "/Users/danmarcab/Projects/custodian/custodian/lib/python3.7/site-packages/c7n/policy.py", line 288, in run
    resources = self.policy.resource_manager.resources()
  File "/Users/danmarcab/Projects/custodian/custodian/lib/python3.7/site-packages/c7n/query.py", line 467, in resources
    resources = self.filter_resources(resources)
  File "/Users/danmarcab/Projects/custodian/custodian/lib/python3.7/site-packages/c7n/manager.py", line 108, in filter_resources
    resources = f.process(resources, event)
  File "/Users/danmarcab/Projects/custodian/custodian/lib/python3.7/site-packages/c7n/resources/ebs.py", line 1330, in process
    results.extend(instance_map.pop(i['InstanceId']))
KeyError: 'i-0baee84015734f898'

Additional context

Tried to debug (with my limited python, sorry if not helpful)

(Pdb) l 1317, 1330
1317            # Filter volumes attached to unsupported instance types
1318            ec2 = self.manager.get_resource_manager('ec2')
1319            instance_map = {}
1320            for v in attached:
1321                instance_map.setdefault(
1322                    v['Attachments'][0]['InstanceId'], []).append(v)
1323
1324            instances = ec2.get_resources(list(instance_map.keys()))
1325            for i in instances:
1326                if i['InstanceType'] in self.older_generation:
1327                    stats['instance-type'] += len(instance_map[i['InstanceId']])
1328                    filtered.extend([v['VolumeId'] for v in instance_map.pop(i['InstanceId'])])
1329                else:
1330 ->                 results.extend(instance_map.pop(i['InstanceId']))
(Pdb) instance_map
{}
(Pdb) instances = ec2.get_resources(list(instance_map.keys()))
(Pdb) instances
[{'AmiLaunchIndex': 0, 'ImageId': 'ami-0e38b48473ea57778', 'InstanceId': 'i-0baee84015734f898', 'InstanceType': 't2.micro', 'LaunchTime': datetime.datetime(2020, 3, 6, 11, 27, 29, tzinfo=tzutc()), 'Monitoring': {'State': 'disabled'}, 'Placement': {'AvailabilityZone': 'us-east-2a', 'GroupName': '', 'Tenancy': 'default'}, 'PrivateDnsName': 'ip-172-31-9-247.us-east-2.compute.internal', 'PrivateIpAddress': '172.31.9.247', 'ProductCodes': [], 'PublicDnsName': 'ec2-18-217-163-42.us-east-2.compute.amazonaws.com', 'PublicIpAddress': '18.217.163.42', 'State': {'Code': 16, 'Name': 'running'}, 'StateTransitionReason': '', 'SubnetId': 'subnet-8906e9e2', 'VpcId': 'vpc-2efb3c45', 'Architecture': 'x86_64', 'BlockDeviceMappings': [{'DeviceName': '/dev/xvda', 'Ebs': {'AttachTime': datetime.datetime(2020, 3, 6, 11, 27, 30, tzinfo=tzutc()), 'DeleteOnTermination': True, 'Status': 'attached', 'VolumeId': 'vol-0948b91be2da21749'}}], 'ClientToken': '', 'EbsOptimized': False, 'EnaSupport': True, 'Hypervisor': 'xen', 'NetworkInterfaces': [{'Association': {'IpOwnerId': 'amazon', 'PublicDnsName': 'ec2-18-217-163-42.us-east-2.compute.amazonaws.com', 'PublicIp': '18.217.163.42'}, 'Attachment': {'AttachTime': datetime.datetime(2020, 3, 6, 11, 27, 29, tzinfo=tzutc()), 'AttachmentId': 'eni-attach-04b0db08ffd66dd7d', 'DeleteOnTermination': True, 'DeviceIndex': 0, 'Status': 'attached'}, 'Description': '', 'Groups': [{'GroupName': 'launch-wizard-6', 'GroupId': 'sg-03b3b641888656bf1'}], 'Ipv6Addresses': [], 'MacAddress': '02:26:d3:ce:43:28', 'NetworkInterfaceId': 'eni-0e437feb92210b38b', 'OwnerId': '434399676812', 'PrivateDnsName': 'ip-172-31-9-247.us-east-2.compute.internal', 'PrivateIpAddress': '172.31.9.247', 'PrivateIpAddresses': [{'Association': {'IpOwnerId': 'amazon', 'PublicDnsName': 'ec2-18-217-163-42.us-east-2.compute.amazonaws.com', 'PublicIp': '18.217.163.42'}, 'Primary': True, 'PrivateDnsName': 'ip-172-31-9-247.us-east-2.compute.internal', 'PrivateIpAddress': '172.31.9.247'}], 'SourceDestCheck': True, 'Status': 'in-use', 'SubnetId': 'subnet-8906e9e2', 'VpcId': 'vpc-2efb3c45', 'InterfaceType': 'interface'}], 'RootDeviceName': '/dev/xvda', 'RootDeviceType': 'ebs', 'SecurityGroups': [{'GroupName': 'launch-wizard-6', 'GroupId': 'sg-03b3b641888656bf1'}], 'SourceDestCheck': True, 'VirtualizationType': 'hvm', 'CpuOptions': {'CoreCount': 1, 'ThreadsPerCore': 1}, 'CapacityReservationSpecification': {'CapacityReservationPreference': 'open'}, 'HibernationOptions': {'Configured': False}, 'MetadataOptions': {'State': 'applied', 'HttpTokens': 'optional', 'HttpPutResponseHopLimit': 1, 'HttpEndpoint': 'enabled'}, 'Tags': ()}]
(Pdb)

Seems that ec2.get_resources(list(instance_map.keys())) is retutning instances when list(instance_map.keys()) is empty. Then in line 1330 those instances are asummed to be in instance_map.

[kapilt]

thanks for the bug report and additional context. it looks like the action was assuming some of the modifiable volumes would always be attached to instances.

[kapilt]

also note to contributors, this would also be resolved by #5245

[JohnHillegass]

closed by #5545