Smart_open: open can not open s3url which key value contains ':@'

[vuryleo]

Reproduce code:

from boto3 import client
c = client('s3')
c.put_object(Key='oss-playground', Key=':@', Body=b'')

from smart_open import open
f = open('s3://oss-playground/:@')

from urllib import parse
f = open('s3://oss-playground/' + parse.quote(':@'))

Expected result:

file can be opened

Reality:

ValueError: not enough values to unpack (expected 2, got 1)
ValueError: '%3A%40' does not exist in the bucket 'oss-playground', or is forbidden for access

There should be at least someway to access 's3://oss-playground/:@'

[mpenkov]

Thanks for reporting this. Sounds like a bug in the parser. Are you able to debug and make a PR?

[vuryleo]

I would think it's a design decision. If you wish to support something like credentials in URL, you may need to support url-encoded URL. So, the bucket/key part need to be URL-encoded.

For example,

s3://user/name:pass/word@buc:ket/ke@y

needs to be encoded as

s3://user/name:pass/word@buc%3Aket/ke%40y

It's a breaking change for user used to use

s3://bucket/100%25

to access

s3://bucket/100%25

but now it will try to get

s3://bucket/100%

instead. (urllib.parse.quote('%') == '%25')

Or, you could just drop support for username/password/host/port things for it could be configured in another way.

[vuryleo]

FYI:

aws s3 ls s3://oss-playground/:@

works.

aws s3 ls s3://oss-playground/%3A%40

does not.

[vuryleo]

So, what do you think? URL-encode? or drop username/password/host/port config via URL?

[mpenkov]

I think URL-encoding is the way to go.

The Amazon CLI allows @ in URL paths, but that breaks the RFC, so we shouldn't be following their example. Could you please confirm whether this reasoning is true?

[mrk-its]

I agree - percent encoding to conform RFC is the way to go (for example it will allow to remove terrible _my_urlsplit hack)
Also - what about dropping support for host/port in S3 URLs (endpoint_url may be still provided in transport_params)? It seems this is rarely used feature and these URLs also break RFC rules (like two @ characters in url). Parsing will be much easier without that (parse.urlsplit will be enough).