Kombu: Connection fails to parse sqs:// URLs with `/` character in secret key

[jwilges]

With kombu version 4.2.1, creating a kombu.Connection with a sqs:// broker URL that has an AWS secret key with a / (forward slash) character raises a ValueError during parse_url().

Steps to reproduce

Assume we have an AWS access key of access_key and an AWS secret key of secret/key.

sqs_test.py

import kombu

broker_url = 'sqs://access_key:secret/key@'
kombu.Connection(broker_url)

Executing this yields:

$ python sqs_test.py
Traceback (most recent call last):
  File "sqs_test.py", line 4, in <module>
    kombu.Connection(broker_url)
  File ".venv/lib64/python3.7/site-packages/kombu/connection.py", line 181, in __init__
    url_params = parse_url(hostname)
  File ".venv/lib64/python3.7/site-packages/kombu/utils/url.py", line 34, in parse_url
    scheme, host, port, user, password, path, query = _parse_url(url)
  File ".venv/lib64/python3.7/site-packages/kombu/utils/url.py", line 52, in url_to_parts
    parts.port,
  File "/usr/lib64/python3.7/urllib/parse.py", line 169, in port
    port = int(port, 10)
ValueError: invalid literal for int() with base 10: 'secret'

Obvious workaround

Regenerating a new key that does not contain a / character works around this issue, but that is a sub-optimal solution.

[shekhar-kamble]

The problem seems to be in the function _splitnetloc used in urlsplit, it uses "/?#" as delim to extract netloc. can I give a try to solve the issue?

[mengjuleu]

This is related to this issue https://github.com/celery/kombu/issues/777

import kombu
from urllib.parse import quote_plus

endpoint_url = 'sqs://{}:{}@'.format(
    quote_plus(aws_access_key),
    quote_plus(aws_access_secret_key)
)

[shekhar-kamble]

But shouldn't the bug in the library be fixed in itself, rather than expecting the user to handle it.

[georgepsarakis]

Literal / characters are interpreted as path delimiters and thus URL parsing cannot work correctly. As far as I can see there is no bug to be fixed.

[jwilges]

Literal / characters are interpreted as path delimiters and thus URL parsing cannot work correctly. As far as I can see there is no bug to be fixed.

It was not obvious at the time, but it is mostly just a simple URL parse call during kombu.Connection().

So, yes, URL encoding the username and password are the correct and safe steps. Fortunately, this caveat is now mentioned up front in some of the updated documentation as of https://github.com/celery/celery/commit/57dbd63113f7722a1ee548cd24fe0f1bd84d7073.