Aws-cli: Firehose creation from json configuration doesn't work

[nitrocode]

My Command

aws --profile etl firehose create-delivery-stream \
    --delivery-stream-name my_long_stream \
    --cli-input-json file://my_long_config.json

json configuration

{
  "RoleARN": "arn:aws:iam::...",
  "ClusterJDBCURL": "jdbc:redshift://...:5439/dev",
  "CopyCommand": {
    "DataTableName": "...",
    "CopyOptions": "delimiter '\\t' emptyasnull"
  },
  "Username": "dataloader",
  "RetryOptions": {
    "DurationInSeconds": 3600
  },
  "S3Configuration": {
    "RoleARN": "arn:aws:iam::...",
    "BucketARN": "arn:aws:s3:::...",
    "Prefix": "...",
    "BufferingHints": {
      "SizeInMBs": 60,
      "IntervalInSeconds": 128
    },
    "CompressionFormat": "UNCOMPRESSED",
    "EncryptionConfiguration": {
      "NoEncryptionConfig": "NoEncryption"
    },
    "CloudWatchLoggingOptions": {
      "Enabled": true,
      "LogGroupName": "/aws/kinesisfirehose/...",
      "LogStreamName": "..."
    }
  },
  "CloudWatchLoggingOptions": {
    "Enabled": true,
    "LogGroupName": "...",
    "LogStreamName": "..."
  }
}

Output

Parameter validation failed:
Unknown parameter in input: "Username", must be one of: DeliveryStreamName, S3DestinationConfiguration, RedshiftDestinationConfiguration, ElasticsearchDestinationConfiguration
Unknown parameter in input: "S3Configuration", must be one of: DeliveryStreamName, S3DestinationConfiguration, RedshiftDestinationConfiguration, ElasticsearchDestinationConfiguration
Unknown parameter in input: "RetryOptions", must be one of: DeliveryStreamName, S3DestinationConfiguration, RedshiftDestinationConfiguration, ElasticsearchDestinationConfiguration
Unknown parameter in input: "RoleARN", must be one of: DeliveryStreamName, S3DestinationConfiguration, RedshiftDestinationConfiguration, ElasticsearchDestinationConfiguration
Unknown parameter in input: "ClusterJDBCURL", must be one of: DeliveryStreamName, S3DestinationConfiguration, RedshiftDestinationConfiguration, ElasticsearchDestinationConfiguration
Unknown parameter in input: "CopyCommand", must be one of: DeliveryStreamName, S3DestinationConfiguration, RedshiftDestinationConfiguration, ElasticsearchDestinationConfiguration
Unknown parameter in input: "CloudWatchLoggingOptions", must be one of: DeliveryStreamName, S3DestinationConfiguration, RedshiftDestinationConfiguration, ElasticsearchDestinationConfiguration

am I doing something wrong?

[stealthycoin]

I'm not sure exactly what you are trying to accomplish here. The --cli-input-json just takes a json object and maps keys to command line arguments. the aws firehose create-delivery-stream only has the arguments:

--delivery-stream-name <value>
[--s3-destination-configuration <value>]
[--extended-s3-destination-configuration <value>]
[--redshift-destination-configuration <value>]
[--elasticsearch-destination-configuration <value>]

Which map to the json keys DeliveryStreamName, S3DestinationConfiguration, RedshiftDestinationConfiguration, ElasticsearchDestinationConfiguration which is says in the error messages you are getting. The JSON file you provided has a lot of top level keys that don't make sense in this context.

[nitrocode]

ohh so there isn't a way to input that json blob in order to create a Kinesis Firehose Delivery Stream ?

[stealthycoin]

No I am saying the JSON blob you are supplying does not match the arguments for that function.

Just run aws firehose create-delivery-stream --generate-cli-skeleton input > file.json which will generate a skeleton in the exact format you need. Then modify file.json so the values are all correct, remove any optional top level keys you do not want to specify. Once it is satisfactory you can run:
aws firehose create-delivery-stream --cli-input-json file://file.json

[balazs-endresz]

I had the same issue because I started out with the json examples from the docs: http://docs.aws.amazon.com/cli/latest/reference/firehose/create-delivery-stream.html
And those don't include the top level keys that --generate-cli-skeleton does. Now it makes sense but it's a bit misleading for those who are new to the CLI.

[JordonPhillips]

The docs display what's needed for using the parameters on the command line. The cli skeleton also includes all of those parameters.

[kolasr]

When I am trying to create firehose delivery stream, I am seeing below error:

An error occurred (InvalidArgumentException) when calling the CreateDeliveryStream operation: Firehose is unable to assume role arn:aws:iam::12345:role/testing. Please check the role provided.

and here is the input which I am using:
aws firehose create-delivery-stream --profile spokeniam-iwdev --region=us-west-2 --cli-input-json file:///testfirehose.json
and here is the testfirehose.json file:
{
"DeliveryStreamName": "SampleDeliveryStream",
"DeliveryStreamType": "KinesisStreamAsSource",
"KinesisStreamSourceConfiguration": {
"KinesisStreamARN": "arn:aws:kinesis:us-west-2:12345:stream/testtestEventStream",
"RoleARN": "arn:aws:iam::12345:role/testing"
},
"S3DestinationConfiguration": {
"RoleARN": "arn:aws:iam::12345:role/testing",
"BucketARN": "arn:aws:s3:::test-events",
"Prefix": "",
"BufferingHints": {
"SizeInMBs": 5,
"IntervalInSeconds": 300
},
"CompressionFormat": "GZIP",
"EncryptionConfiguration": {
"NoEncryptionConfig": "NoEncryption"
}
}
}

[kolasr]

I am using the KinesisStreamARN and the role which I have used to create the kinesis. Am I doing anything wrong? I got the template from --generate-cli-skeleton and deleted couple of them because I am using only source as kinesis and target as S3.