Aws-cli: `aws sqs get-queue-url` returns deprecated SQS queue url

[rbroemeling]

This is a straight repost/migration of the problem report that was originally made at https://forums.aws.amazon.com/thread.jspa?messageID=688330.

When we fetch an SQS queue url via aws sqs get-queue-url, we receive a deprecated URL.

To whit:

$ aws sqs get-queue-url --queue-name MY_QUEUE --region us-west-2 --output text
https://us-west-2.queue.amazonaws.com/MY_ACCOUNT_ID/MY_QUEUE

According to the SQS web interface, the URL for this queue is actually:
https://sqs.us-west-2.amazonaws.com/MY_ACCOUNT_ID/MY_QUEUE

Normally we wouldn't particularly care about this, except for that when we take the old/deprecated style queue URL (the one that is returned from the aws sqs get-queue-url command) and try to use it from an Amazon EC2 instance, we get the following error:

[Aws::SQS::Client 403 0.143666 0 retries] receive_message(max_number_of_messages:1,queue_url:"https://us-west-2.queue.amazonaws.com/MY_ACCOUNT_ID/MY_QUEUE",wait_time_seconds:20) Aws::SQS::Errors::SignatureDoesNotMatch Credential should be scoped to a valid region, not 'queue'.

So, basically, we have the aws-cli tools spitting out an SQS queue URL that is invalid/unsable in at least some cases.

We work-around the problem by "crafting" our own queue url as it is currently pretty predictable, but we really think that this should be fixed with aws-cli being updated to return a correct/usable queue url. The Amazon documentation explicitly warns about crafting our own URL, but at this point we don't have a lot of reasonable options left.

For the reference that makes me claim that the queue.amazonaws.com style has been deprecated, see this thread (specifically, look for the post by joel@AWS on 25-Feb-2013): https://forums.aws.amazon.com/thread.jspa?messageID=425255

Thanks!

[JordonPhillips]

I think I see the source of the issue. So in older versions of Python there's a bug where SSL fails if you have to check alternative names rather than the common name. As a result, we have modifications in place for a handful of services where the common name doesn't match the {service}.{region}.amazonaws.com scheme, SQS included. It seems that when you query SQS with the {region}.queue.amazonaws.com scheme, it will return in kind. You then take that url into what appears to be Ruby code, whose SDK is assuming the default scheme when it performs signing.

Unfortunately, we won't be able to remove this behavior from boto3 unless and until we drop support for Python 2.6. Can I ask why you're performing get-queue-url from the CLI rather than the SDK that you're using?

[rbroemeling]

@JordonPhillips I have a shellscript that configures the user-data for an instance launch configuration -- part of what it does is (given an SQS queue name) seed the URL for that specific SQS queue. At the time of execution, it is just a BASH script and there isn't really an SDK involved. As such, I use aws sqs get-queue-url to map the name to a URL.

The relevant portion of the bash script currently looks something like this:

SQS_QUEUE_URL="https://sqs.${EC2_REGION}.amazonaws.com/OUR_ACCOUNT_ID/${CLOUD_BACKUP_SQS_QUEUE_NAME}"

... when we would _like_ it to look something like this (currently we can't do this, due to this outstanding issue):

SQS_QUEUE_URL="$(aws sqs get-queue-url --queue-name "${CLOUD_BACKUP_SQS_QUEUE_NAME}" --output text --region "${EC2_REGION}")"

I could pass in just the SQS queue name itself and then within the instance itself map it to an SQS URL, but that would require enough changes on the instance side (of which the changes carry enough risk) that I am far from eager to go that path.

[rbroemeling]

It seems that when you query SQS with the {region}.queue.amazonaws.com scheme, it will return in kind.

Is there some way that I can configure the CLI to query SQS using {service}.{region}.amazonaws.com?

[JordonPhillips]

@rbroemeling Yes, you can customize the endpoint url like so:

aws sqs get-queue-url --queue-name test --endpoint-url https://sqs.us-west-2.amazonaws.com/

[anilfunde]

boto3 sqs client still returns the deprecated URL. How can I force boto3 client to return new format URL?

[anilfunde]

I just went though boto source and found a solution for boto client. If you want get_queue_url to return URL in format {service}.{region}.amazonaws.com , use "endpoint_url" argument while creating client

    sqs_client = boto3.client(
        'sqs',
        aws_access_key_id=#####,
        aws_secret_access_key=#####,
        region_name=####,
        endpoint_url='https://sqs.' + {region_name} + '.amazonaws.com'
    )

sqs client created this way will return queue url in the desired format

[sworisbreathing]

Should this be reopened, given that the default behavior of the CLI is still producing an unusable result?

Python 2.6 has been EOL since 2013, and 2.7 is only a couple months from being EOL itself (hopefully for real this time).

[jordanst3wart]

Why is this closed? This is still a bug. Could you re-open this please?

Recreate:

queue_url=$(aws sqs get-queue-url  --queue-name "$name" | jq '.QueueUrl')
aws sqs receive-message --queue-url "$queue_url"

An error occurred (InvalidAddress) when calling the ReceiveMessage operation: The address "<<redacted>>" is not valid for this endpoint.

Add 'sqs' behind 'https://', and it works.

aws --version               
aws-cli/1.16.257 Python/3.7.4 Darwin/18.7.0 botocore/1.12.247