Aws-cli: Unable to pass complex JSON file with multiple nestings for assigning permission to the SNS topic to send messages to the SQS queue

[rosscar]

Following command issued:

aws sqs set-queue-attributes --queue-url https://sqs.us-west-1.amazonaws.com/123456789012/MyQueue --attributes file:sqspolicy.json

Note that the JSON file (sqspolicy.json) was first created within the SQS console using the following instructions - Step 2. Give permission to the SNS topic to send messages to the SQS queue. (http://docs.aws.amazon.com/sns/latest/gsg/SendMessageToSQS.html#SendMessageToSQS.sqs.permissions)

Contents of JSON file:

{
  "Version": "2008-10-17",
  "Id": "arn:aws:sqs:us-west-1:123456789012:MyQueue/SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "Sid1363024842688",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:us-west-1:123456789012:MyQueue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-west-1:123456789012:MyTopic"
        }
      }
    }
  ]
}

[rosscar]

Hi, any idea when this will be fixed?

I tried this with the latest version. Here's the results:

Error parsing parameter --attributes, should be: --attributes key_name=string,key_name2=string
Where valid key names are:
Policy
VisibilityTimeout
MaximumMessageSize
MessageRetentionPeriod
ApproximateNumberOfMessages
ApproximateNumberOfMessagesNotVisible
CreatedTimestamp
LastModifiedTimestamp
QueueArn
ApproximateNumberOfMessagesDelayed
DelaySeconds
ReceiveMessageWaitTimeSeconds

Here's the version info:
C:>aws --version
aws-cli/0.15.0 Python/2.7.3 Windows/7

[garnaat]

I think what you really want to be able to do is this:

aws sqs set-queue-attributes --queue-url --attributes Policy=file://sqspolicy.json

Unfortunately, we don't support the file:// syntax for values of embedded fields, only for the top-level fields. One solution would be to customize this command and turn all of the possible attribute names into first-class options. Then, you could do something like this:

aws sqs set-queue-attributes --queue-url <queue_url> --policy file://sqspolicy.json

Does that seem like a reasonable solution?

[ehammond]

Something definitely needs to be done here, if just clarifying with a complete example in the documentation. I have been URL encoding as requested and still can't get past this error:

A client error (InvalidAttributeValue) occurred when calling the SetQueueAttributes operation: Invalid value for the parameter Policy.

Requiring form-url-encoding is making it overly difficult for the user. Let us specify the policy and do whatever encoding is required in the code.

[ehammond]

When I try to use the JSON format listed in the "help" output, it returns the error

Error parsing parameter '--attributes': should be: --attributes key_name=string,key_name2=string

which kind of implies that it isn't even looking for a JSON alternative.

[ehammond]

Ah, if the JSON is formatted perfectly, then it accepts it silently.

[Neocid]

Hi there,

I do have the issue as well. As @garnaat specified enclosed [] properties (second level) are not being able to be parsed. As the policy must contains it, we can not automate the create of policy via CLI directly from a valid json file. Any idea when this is going to be possible to parse that?

Thanks :)

[jamesls]

I think the best idea I've heard so far is here: https://github.com/aws/aws-cli/issues/1803#issuecomment-189027369, but we're open to other ideas.

We can probably consolidate feature requests and track this work over in #1803.

[Neocid]

Good idea.
Side note: for those who are looking for a solution, I got mine here:
https://alestic.com/2014/12/s3-bucket-notification-to-sqssns-on-object-creation/
(with the modification specified in the comments)

[CamelShikai]

I met the policy invalid type error,too. I converted policy dict to a json string and it was accepted. Hope that could help you.

[CamelShikai]

By the way, I was calling boto3's api 'set_queue_attributes' function to do that.

[bedge]

As this is so completely brain-dead I have to add my nasty solution to save the next poor sot that ends up having to deal with this revolting API that someone in AWS actually signed off on:

cat >sqs.json <<-EOT
{
   "Policy" : "{ \"Statement\" : [ { \"Action\" : \"SQS:SendMessage\", \"Effect\" : \"Allow\", \"Sid\": \"AllowPESends\", \"Principal\" : { \"AWS\" : \"*\" }, \"Condition\" : { \"ArnEquals\" : { \"aws:SourceArn\" : \"${sns_topic_arn}\" } }, \"Resource\" : \"${sqs_arn}\" } ], \"Id\" : \"SQSPESendPolicy\", \"Version\" : \"2012-10-17\" }"
 }
EOT

aws sqs set-queue-attributes --region ${region} --queue-url ${sqs_url} --attributes file://sqs.json

Yes folks you have to embed a json file as a string, in ...wait for it... another json file, and there's more, write all that to a file, and put that file on the command line.

And, yes, you have to spec the Principal as "AWS":"" because the account that sends the SQS message is *not yours.

AWS has some cool stuff, but Jeez, this was not finished. Even my ugliest debug crap is leaner than that.

[TalosThoren]

Oh man I need some help with this. Your example doesn't work for me, bedge, though I'm sure it's because I'm doing something different somehow, and the most this thing tells me is "Invalid value for the parameter Policy".

AWS just needs to have some mechanism for automatically applying the appropriate policy to a queue when you subscribe it to a topic. It should almost certainly just happen as part of the subscription process. This process is ridiculous.

[bedge]

Hey @TalosThoren, I eventually shoved the whole mess into a couple of shell functions.

The connect_sqs is the one you want, but I included the create_sqs_with_dead_drop it calls in case there's something else in there that I'm doing that you're not.

#
# SQS can have a dead leter SQS for messages that fail many times.
# This has to be created prior to the primary SQS
# This method creates both SQSs
# reauired param:
#   $1 base_name
# optional params:
#   $2 maxReceiveCount - before going to dead-letter sqs, default:10
#   $4 visibility_timeout

# if eval'd, sets:
# sqs_name
# sqs_url
# sqs_name_dead
# sqs_url_dead
#
create_sqs_with_dead_drop() {
  sqs_name=$1
  max_receive_count=${2:-10}
  visibility_timeout=${3:-60}

  # Temp dir deleted on exit
  tmpdir=$(mktemp -d /tmp/d.ec2-common.XXXXXXXX) && cd "$dir" || exit 99
  trap 'rm -rf "$tmpdir"' EXIT

  # SQS's can be recreated nondestructively

  while true ; do
      sqs_url_dead=$(aws sqs create-queue --queue-name "${sqs_name}-dead" \
        --region $region --output text)
      [ $DEBUG ] && sqs_dead_url=$sqs_dead_url
      sqs_arn_dead=$(aws sqs get-queue-attributes --queue-url ${sqs_url_dead} \
        --attribute-names QueueArn --region $region  --output text | awk "{print \$2}") \
        && break
      sleep 5
  done

  cat << EOF > ${tmpdir}/attributes-dead.json
  {
      "DelaySeconds": "10",
      "MaximumMessageSize": "262144",
      "MessageRetentionPeriod": "1209600",
      "ReceiveMessageWaitTimeSeconds": "0",
      "VisibilityTimeout": "60"
  }
EOF
  aws sqs set-queue-attributes --queue-url ${sqs_url_dead} --attributes file://${tmpdir}/attributes-dead.json --region $region

  while true ; do
      sqs_url=$(aws sqs create-queue --queue-name "${sqs_name}" \
        --region $region --output text)
      [ $DEBUG ] && sqs_url=$sqs_url
      sqs_arn=$(aws sqs get-queue-attributes --queue-url ${sqs_url} \
        --attribute-names QueueArn --region $region  --output text | awk "{print \$2}") \
        && break
      sleep 5
  done

  cat << EOF > ${tmpdir}/attributes.json
  {
      "DelaySeconds": "10",
      "MaximumMessageSize": "262144",
      "MessageRetentionPeriod": "1209600",
      "ReceiveMessageWaitTimeSeconds": "0",
      "RedrivePolicy": "{\"deadLetterTargetArn\":\"${sqs_arn_dead}\",\"maxReceiveCount\":\"${max_receive_count}\"}",
      "VisibilityTimeout": "${visibility_timeout}"
  }
EOF
  aws sqs set-queue-attributes --queue-url ${sqs_url} --attributes file://${tmpdir}/attributes.json --region $region

  cat << EOT
  export sqs_url="${sqs_url}"
  export sqs_name="${sqs_name}"
  export sqs_arn="${sqs_arn}"
  export sqs_url_dead="${sqs_url_dead}"
  export sqs_name_dead="${sqs_name}-dead"
  export sqs_arn_dead="${sqs_arn_dead}"
EOT
}


function connect_sqs() {

    sns_arn=$1
    sqs_name=$2
    max_rc=$3
    vis_to=$4

    tmpfile=$(mktemp /tmp/sns_sub.sh.$?.XXXXXXXX)
    eval $(create_sqs_with_dead_drop "${sqs_name}" $max_rc $vis_to)

    # Poorly documented mechanism for allowing the prod publication-events topic to write to out sqs.
    # Note the json encoded string as an embedded json parameter. Someone at AWS signed off on this??
    cat >${tmpfile} <<-EOT
    {
    "Policy" : "{ \"Statement\" : [ { \"Action\" : \"SQS:SendMessage\", \"Effect\" : \"Allow\", \"Sid\": \"AllowPESends\", \
    \"Principal\" : { \"AWS\" : \"*\" }, \"Condition\" : { \"ArnEquals\" : { \"aws:SourceArn\" : \"${sns_arn}\" } }, \
    \"Resource\" : \"${sqs_arn}\" } ], \"Id\" : \"SQSPESendPolicy\", \"Version\" : \"2012-10-17\" }"
    }
EOT
    aws sqs set-queue-attributes --region ${region} --queue-url ${sqs_url} --attributes file://${tmpfile}

    # Subscribe to prod's publication events
    if ! aws sns subscribe --topic-arn ${sns_arn} --region ${region} \
    --protocol sqs \
    --notification-endpoint ${sqs_arn} ; then
    cat <<-EOT
    =============================================
        SNS $1 TOPIC NOT ATTACHED
    =============================================
EOT
        return 1
    fi
    if [ ! "${NO_CLEAN_UP}" ] && [ "${PURGE}" ] ; then
        aws sqs purge-queue --region ${region} --queue-url ${sqs_url} || true
    fi
}

[TalosThoren]

Thanks @bedge!

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.