Boto3: emr run_job_flow lacks option to enable debugging

[asafcombo]

aws cli has an option to enable debugging:

aws emr create-cluster 
--auto-scaling-role EMR_AutoScaling_DefaultRole 
--applications Name=Hadoop Name=Hive Name=Pig Name=Spark 
--ebs-root-volume-size 10 
--ec2-attributes 'xxxxx' 
--service-role EMR_DefaultRole 
--enable-debugging
--release-label emr-5.11.0
....

whereas boto3 seems to lack this option.

[stealthycoin]

Boto3 has set_stream_logger which does the same thing. You can set to log the most possible with boto3.set_stream_logger('')

http://boto3.readthedocs.io/en/latest/reference/core/boto3.html#boto3.set_stream_logger

[asafcombo]

@stealthycoin , this does not achieve the desired functionality of enable debugging.

enable debugging sets the EMR cluster with debugging mode.

I have found a workaround:
https://github.com/boto/boto3/issues/690

[CollinRea]

@asafcombo Does adding a debugging step actually enable debugging on the AWS console? I'd still love to be able to pass the --enable-debugging flag in to the config and use it like the AWS CLI allows.

aws emr create-cluster --name "Test cluster" --release-label emr-4.1.0 --log-uri s3://mybucket/logs/ --enable-debugging --applications Name=Hadoop Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-type m4.large --instance-count 3

[igorborgest]

@asafcombo @CollinRea

Actually --enable-debugging is not a native AWS EMR API feature.
That is achieved in console/CLI silently adding a extra first step that enables the debugging.

So, we can do that using Boto3 doing the some strategy and add this step to your run_job_flow call:

Steps=[
    {
        'Name': 'Setup Hadoop Debugging',
        'ActionOnFailure': 'TERMINATE_CLUSTER',
        'HadoopJarStep': {
            'Jar': 'command-runner.jar',
            'Args': ['state-pusher-script']
        }
    }...

@stealthycoin do you think that worth add this feature in Boto3?

P.S. AWS CLI implementation