Smart_open: In some environments (not all), I see that smart_open adds some characters to bucket name to cause error

[devangmehta123]

Hi,

Please see error message below. This started happening after my upgrade from 1.5.3 to 1.5.5. It is a strange problem. In my local VM environment, which uses the exact same bucket, these additional characters are not added.

botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "('afdsoftware.energylinx.co.uk": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$"

I used to have a monkey patch for boto2 to prevent a similar but not identlical error which did some modification to ssl behaviour. But that is not needed anymore in local environment and has no effect inside the environment causing this problem.

The difference is the additional "('" prefixed to the bucket name after this change.

[menshikh-iv]

Hi @devangmehta123, thanks for the report, can you show code example (needed for reproducing)?

[devangmehta123]

@menshikh-iv , thanks your quick response. Please see my code snippet below. I am formulating a URL in this code and I have checked it thorugh debugger. It looks correct and one example is this:

's3://some_access_key:[email protected]/paf-sample-subset/2017-10-25/Y17M10/PAF MAIN FILE/fpmainfl.c02.gz'

    def _update_dynamodb_table(self, main_file_series_name, start_line=0):
        directory = self._not_yet_built_dir if main_file_series_name == 'not_yet_built' else self._main_file_dir
        file_list = [
            's3://{}:{}@{}/{}/{}/{}.gz'.format(
                settings.PAF['s3_access_key'],
                settings.PAF['s3_secret_key'],
                settings.PAF['s3_url_prefix'],
                self._today_date_str,
                directory,
                x)
            for x in settings.PAF[main_file_series_name]['main_file_names']
        ]
        model = paf_table_map[main_file_series_name]['model']
        num_lines = 0
        for f in file_list:
            with smart_open.smart_open(f) as main_file:

[devangmehta123]

Also, this entire traceback can be useful:

app@dced75c07b2c:~/app$ DJANGO_SETTINGS_MODULE=elexapis2.settings_master python3 /srv/app/manage.py upsert_paf_tables --upsert-organisation --date=2017-11-25 > /tmp/paf_upsert_organisation.log
Traceback (most recent call last):
  File "/srv/app/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/srv/app/common/management/commands/upsert_paf_tables.py", line 358, in handle
    self._update_dynamodb_table(main_file_series_name=m, start_line=start_line)
  File "/srv/app/common/management/commands/upsert_paf_tables.py", line 268, in _update_dynamodb_table
    with smart_open.smart_open(f) as main_file:
  File "/usr/local/lib/python3.5/dist-packages/smart_open/smart_open_lib.py", line 178, in smart_open
    return s3_open_uri(parsed_uri, mode, **kw)
  File "/usr/local/lib/python3.5/dist-packages/smart_open/smart_open_lib.py", line 260, in s3_open_uri
    fobj = smart_open_s3.open(parsed_uri.bucket_id, parsed_uri.key_id, s3_mode, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/smart_open/s3.py", line 61, in open
    fileobj = SeekableBufferedInputBase(bucket_id, key_id, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/smart_open/s3.py", line 256, in __init__
    self._raw_reader = SeekableRawReader(self._object)
  File "/usr/local/lib/python3.5/dist-packages/smart_open/s3.py", line 96, in __init__
    self._content_length = self._object.content_length
  File "/usr/local/lib/python3.5/dist-packages/boto3/resources/factory.py", line 339, in property_loader
    self.load()
  File "/usr/local/lib/python3.5/dist-packages/boto3/resources/factory.py", line 505, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 589, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/usr/local/lib/python3.5/dist-packages/botocore/client.py", line 641, in _convert_to_request_dict
    params=api_params, model=operation_model, context=context)
  File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.5/dist-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.5/dist-packages/botocore/handlers.py", line 220, in validate_bucket_name
    raise ParamValidationError(report=error_msg)
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid bucket name "('afdsoftware.energylinx.co.uk": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$"

[mpenkov]

@devangmehta123 Let's try to isolate this issue further.

Does this reproduce the bug in the "broken" environment?

key = 'your_key'
secret = 'your_secret'
bucket = 'your_bucket'
prefix ='/p/r/e/f/i/x'
file_list = ['your', 'files']
for f in file_list:
    url = 's3://{}:{}@{}/{}/{}.gz'.format(key, secret, bucket, prefix, f)
    parsed = smart_open.ParseUri(url)
    print(parsed.bucket_id)

What do you see when you run the above code? Is the bucket ID correct, or does it start with a parens?

[devangmehta123]

@mpenkov , I feel very, very silly right now. Because I simply did not look at the settings file correctly and there is one typo which crept through.

The error message is not the best because it is showing somehow the string representation of a tuple but not the closing parantheses.

But I found this very small mistake in my settings file (see typo of trailing comma):

PAF["s3_url_prefix"] = "afdsoftware.energylinx.co.uk/paf",

I am quite confident that this is the cause, I will test it in 10 minutes.

[mpenkov]

@devangmehta123 Glad you figured it out. Please let us know the result of your tests.

If the issue persists, please give us the full URL your are passing to smart_open(). Please redact sensitive info (key, secret, etc). Be sure to use repr() to print the full representation.

[devangmehta123]

Thank you and yes, the problem was in the typo and smart_open is working correctly now. I am happy to use the latest version with boto3 and now I do not need that boto2 monkey patch.

The monkey patch was needed because my bucket had dot characters in the name and with boto2, there was a bug which created problems with SSL.

Thank you very much and sorry to report a problem which was not a real problem to begin with.