Smart_open: Example with custom S3 isnt working

[bolkedebruin]

The examples list

>>> # using a completely custom s3 server, like s3proxy:
>>> for line in smart_open.smart_open('s3u://user:secret@host:port@mybucket/mykey.txt'):
...    print line

But this doesn't work. smart_open does not parse the host/port bits.

https://github.com/RaRe-Technologies/smart_open/blob/master/smart_open/smart_open_lib.py#L383

[menshikh-iv]

Thanks for report @bolkedebruin
CC @mpenkov

[mpenkov]

The code itself is "fine", but the example is misleading. If you use an integer as a port number, everything is cool. If you use the literal "port" for the port number, you'll get an exception.

(smart_open) misha@cabron:~/git/smart_open$ ipython
Python 3.6.5 (default, Apr  1 2018, 05:46:30)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from smart_open.smart_open_lib import _parse_uri

In [2]: _parse_uri('s3u://user:secret@host:1234@mybucket/mykey.txt')
Out[2]: Uri(scheme='s3u', uri_path=None, bucket_id='mybucket', key_id='mykey.txt', port=1234, host='host', ordinary_calling_format=True, access_id='user', access_secret='secret')

In [3]: _parse_uri('s3u://user:secret@host:port@mybucket/mykey.txt')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/git/smart_open/smart_open/smart_open_lib.py in _parse_uri_s3x(parsed_uri)
    506                 if len(server) == 2:
--> 507                     port = int(server[1])
    508             else:

ValueError: invalid literal for int() with base 10: 'port'

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-3-dd6e6977bfa9> in <module>
----> 1 _parse_uri('s3u://user:secret@host:port@mybucket/mykey.txt')

~/git/smart_open/smart_open/smart_open_lib.py in _parse_uri(uri_as_string)
    446         return _parse_uri_webhdfs(parsed_uri)
    447     elif parsed_uri.scheme in smart_open_s3.SUPPORTED_SCHEMES:
--> 448         return _parse_uri_s3x(parsed_uri)
    449     elif parsed_uri.scheme in ('file', '', None):
    450         return _parse_uri_file(parsed_uri)

~/git/smart_open/smart_open/smart_open_lib.py in _parse_uri_s3x(parsed_uri)
    516         # Bucket names can contain lowercase letters, numbers, and hyphens.
    517         # Each label must start and end with a lowercase letter or a number.
--> 518         raise RuntimeError("invalid S3 URI: %s" % str(parsed_uri))
    519
    520     return Uri(

RuntimeError: invalid S3 URI: SplitResult(scheme='s3u', netloc='user:secret@host:port@mybucket', path='/mykey.txt', query='', fragment='')

The solution is to:

• Fix the example to use an integer port number
• Improve the exception handling

I'll take care of this tomorrow :)

[eromoe]

How can I pass AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as arguments in smart_open ?
There is no example for this.

[mpenkov]

@eromoe Specifying those parameters directly is not possible.

However, you can achieve the same thing by passing a boto3 Session that includes your credentials:

# untested code!
import boto3
import smart_open
session = boto3.Session(aws_access_key_id=..., aws_secret_access_key=...)
with smart_open.smart_open(..., s3_session=session) as fin:
   ...