Moto: EbsConfiguration Attribute for "Instances.InstanceGroups" not supported in EMR "run_job_flow"

[ndamclean]

Code Excerpt

    ...
    cluster_kwargs['Instances'] = { 
        'InstanceGroups': [
          {
            'InstanceCount': 1,
            'EbsConfiguration': {           
                'EbsBlockDeviceConfigs': [{     
                  'VolumeSpecification': {        
                    'SizeInGB': 32,           
                    'VolumeType': 'gp2'          
                  },           
                  'VolumesPerInstance': 1         
                }]
            },                 
            'InstanceRole': 'MASTER',       
            'InstanceType': 'm4.large',
            'Name': 'master'   
          },
          {
            'InstanceCount': 5,
            'EbsConfiguration': {           
                'EbsBlockDeviceConfigs': [{     
                    'VolumeSpecification': {        
                      'SizeInGB': 32,           
                      'VolumeType': 'gp2'          
                    },  
                    'VolumesPerInstance': 1         
                }]             
            },
            'InstanceRole': 'CORE',         
            'InstanceType': 'm4.large',  
            'Name': 'slave'    
          }                    
        ]
    }
    ...
    client.run_job_flow(**cluster_kwargs)

Error Message

test.py:50: in test_run_job_flow
    response = client.run_job_flow(**cluster_kwargs)
/usr/local/lib/python2.7/dist-packages/botocore/client.py:253: in _api_call
    return self._make_api_call(operation_name, kwargs)
/usr/local/lib/python2.7/dist-packages/botocore/client.py:544: in _make_api_call
    operation_model, request_dict)
/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py:141: in make_request
    return self._send_request(request_dict, operation_model)
/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py:170: in _send_request
    success_response, exception):
/usr/local/lib/python2.7/dist-packages/botocore/endpoint.py:249: in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
/usr/local/lib/python2.7/dist-packages/botocore/hooks.py:227: in emit
    return self._emit(event_name, kwargs)
/usr/local/lib/python2.7/dist-packages/botocore/hooks.py:210: in _emit
    response = handler(**kwargs)
/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py:183: in __call__
    if self._checker(attempts, response, caught_exception):
/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py:251: in __call__
    caught_exception)
/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py:269: in _should_retry
    return self._checker(attempt_number, response, caught_exception)
/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py:317: in __call__
    caught_exception)
/usr/local/lib/python2.7/dist-packages/botocore/retryhandler.py:223: in __call__
    attempt_number, caught_exception)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.retryhandler.ExceptionRaiser object at 0x7f8995f148d0>, attempt_number = 1
caught_exception = TypeError("__init__() got an unexpected keyword argument 'ebs_configuration._ebs_block_device_configs.member.1._volume_specification._volume_type'",)

    def _check_caught_exception(self, attempt_number, caught_exception):
        # This is implementation specific, but this class is useful by
        # coordinating with the MaxAttemptsDecorator.
        # The MaxAttemptsDecorator has a list of exceptions it should catch
        # and retry, but something needs to come along and actually raise the
        # caught_exception.  That's what this class is being used for.  If
        # the MaxAttemptsDecorator is not interested in retrying the exception
        # then this exception just propogates out past the retry code.
>       raise caught_exception
E       TypeError: __init__() got an unexpected keyword argument 'ebs_configuration._ebs_block_device_configs.member.1._volume_specification._volume_type'

See boto3 docs for the specs for this attribute.
https://boto3.readthedocs.io/en/latest/reference/services/emr.html#EMR.Client.run_job_flow

[spulec]

Thanks for opening. Will mark this as a feature request.

[delijati]

Adding kwargs fixed it for me:

class FakeInstanceGroup(BaseModel):

    def __init__(self, instance_count, instance_role, instance_type,
                 market='ON_DEMAND', name=None, id=None, bid_price=None,
                 **kwargs):

[ndamclean]

I don't think this is a good fix as it will accept any arbitrary keyword arguments. We should be checking for valid values in the instance group AWS API.
https://docs.aws.amazon.com/emr/latest/APIReference/API_InstanceGroup.html

[trentgerman]

Would love for someone to work on this!

[weldpua2008]

I am getting the same error.
I had checked my code with the debugger and after I executed the code - it runs but tests are failing
'Got error __init__() got an unexpected keyword argument \'ebs_configuration._ebs_block_device_configs.member.1._volume_specification._volume_type\' None'

the configuration part:

{
                    "Name": "Master nodes",
                    "Market": "ON_DEMAND",
                    "InstanceRole": "MASTER",
                    "InstanceType": "m4.2xlarge",
                    "InstanceCount": 1,
                    "EbsConfiguration": {
                        'EbsBlockDeviceConfigs': [
                            {
                                'VolumeSpecification': {
                                    "VolumeType": "gp2",
                                    "SizeInGB": 100
                                },
                                'VolumesPerInstance': 1
                            },
                        ]
                    }
                }