Botocore: Update SQS endpoints

Created on 23 Mar 2018 · 8Comments · Source: boto/botocore

botocore as of 1.9.15 returns queue URLs according to the {region}.queue.{suffix} pattern:

In [19]: boto3.client('sqs').get_queue_url(QueueName='hello')['QueueUrl']
Out[19]: 'https://eu-west-1.queue.amazonaws.com/1234..../hello'

In [20]: boto3.__version__
Out[20]: '1.6.15'

In [21]: botocore.__version__
Out[21]: '1.9.15'

According to this page they are considered legacy endpoints, and from the wording it seems that they are only tolerated for AWS CLI and Python libraries. Today I've experienced a problem with this when Ruby SDK refused to work with a URL obtained through boto3 client: https://github.com/aws/aws-sdk-ruby/issues/1620.

Is there any hard blocker for this to be fixed?

Source

immerrr

Most helpful comment

this just dawned on me. We have to use the old endpoints on Py26, yes, but why do the remaining 99% of developers running on Py2.7 and Py3+ have to suffer?

For example, an easy approach would be to decide on whether or not to use the old endpoint with

(sys.version_info.major, sys.version_info.minor) == (2, 6)

Or even, following the EAFP principle, the code could start with the new endpoint as the default, try using that, and then if it fails with some expected SAN-related error it would reset the default endpoint to the old one.

@jamesls WDYT?

immerrr on 9 Jul 2019

👍5

All 8 comments

Yes, the main blocker is python2.6 support. py26 can't read the SAN on the certs for the sqs.<region>.amazonaws.com endpoints so as long we support py26 we have to use the old endpoints. Once we're able to drop py26 support (not sure on the timing) then we stop using the older endpoints.
More info here: https://bugs.python.org/issue13034

Let me know if you have any other questions.

jamesls on 29 Mar 2018

This is sadly still the case but I got around it by specifying the endpoint_url manually. For example SQS VPC endpoint only works with the new host names so boto3 doesn't work in all environments.
See https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-sending-messages-from-vpc.html

I used: boto3.client('sqs', endpoint_url = 'https://sqs.' + os.getenv('AWS_REGION') + '.amazonaws.com')

jurajseffer on 9 Jul 2019

this just dawned on me. We have to use the old endpoints on Py26, yes, but why do the remaining 99% of developers running on Py2.7 and Py3+ have to suffer?

For example, an easy approach would be to decide on whether or not to use the old endpoint with

(sys.version_info.major, sys.version_info.minor) == (2, 6)

@jamesls WDYT?

immerrr on 9 Jul 2019

👍5

@jamesls : Any chance to reopen this issue and make the change, now that Python 2.6 has been dropped? We just stumbled upon this issue and it is a pain to diagnose (we just have something which is timeouting and solving it with @jurajseffer workaround is not obvious. We are in a VPC without internet access and the SQS VPC Endpoint is not working with legacy URLs. We are for instance not even capable of listing the queues or finding a queue by name.
Thanks.

benoit74 on 27 Jan 2020

👍1

This becomes a more significant issue when using PrivateLink VPC endpoints, which do not support the legacy queue URLs. With PrivateLink becoming more and more common, this feels like an issue which really needs to be resolved, or at least called out specifically in the documentation.