Moto: BaseResponse method `uri_to_regexp` leaves dashes in the final string

Created on 18 Dec 2020 · 3Comments · Source: spulec/moto

Context

Hi. I'm using moto to mock aws pinpoint client in a similar way to the solution described here

The main issue is that botocore specifies some uris with dashes in named groups, like application-id here: https://github.com/boto/botocore/blob/d479a37ce2c9d1832b578c627e92886e8219c1de/botocore/data/pinpoint/2016-12-01/service-2.json#L65

This ends up with and re.error exception later on when a regex match is attempted using this string.

Process

[ordered list the process to finding and recreating the issue, example below]

Create the package structure as described in here
replace url_bases and url_paths in pinpoint_urls.py by the following:

url_bases = ['https?://pinpoint.(.+).amazonaws.com']
url_paths = {r'{0}/v1/apps/(.+)/messages$': PinPointResponse.dispatch,}

Add a send_message method to pinpoint_responses.py:

def send_messages(self):
  return 200, {}, {}

Instantiate a pinpoint client and call send_message fromit in test.py:

class TestMyModule:
@mock_pinpoint
def test(self):
  client = boto3.client('pinpoint', region_name='eu-central-1')
  client.send_messages(
    ApplicationId='asdqwe1234',
    MessageRequest={
      'Addresses': {
        'some_address': {
          'ChannelType': 'EMAIL'
        }
      },
      'MessageConfiguration': {
        'EmailMessage': {
          'FromAddress': 'my_address',
          'RawEmail': {
            'Data": "string"
          }
        }
      }
    }
  )

Expected result

When running the tests you should get a re.error with the message re.error: bad character in group name 'application-id' at position 14
The full traceback:

../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/client.py:357: in _api_call
    return self._make_api_call(operation_name, kwargs)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/client.py:663: in _make_api_call
    operation_model, request_dict, request_context)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/client.py:682: in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/endpoint.py:102: in make_request
    return self._send_request(request_dict, operation_model)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/endpoint.py:137: in _send_request
    success_response, exception):
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/endpoint.py:256: in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/retryhandler.py:183: in __call__
    if self._checker(attempts, response, caught_exception):
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/retryhandler.py:251: in __call__
    caught_exception)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/retryhandler.py:269: in _should_retry
    return self._checker(attempt_number, response, caught_exception)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/retryhandler.py:317: in __call__
    caught_exception)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/retryhandler.py:223: in __call__
    attempt_number, caught_exception)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/retryhandler.py:359: in _check_caught_exception
    raise caught_exception
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/endpoint.py:197: in _do_get_response
    responses = self._event_emitter.emit(event_name, request=request)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/hooks.py:356: in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/hooks.py:228: in emit
    return self._emit(event_name, kwargs)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/botocore/hooks.py:211: in _emit
    response = handler(**kwargs)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/moto/core/models.py:323: in __call__
    request, request.url, request.headers
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/moto/core/responses.py:202: in dispatch
    return cls()._dispatch(*args, **kwargs)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/moto/core/responses.py:312: in _dispatch
    return self.call_action()
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/moto/core/responses.py:392: in call_action
    action = camelcase_to_underscores(self._get_action())
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/moto/core/responses.py:380: in _get_action
    return self._get_action_from_method_and_request_uri(self.method, self.path)
../../.local/share/virtualenvs/my_project/lib/python3.7/site-packages/moto/core/responses.py:365: in _get_action_from_method_and_request_uri
    match = re.match(regexp, request_uri)
../../.local/share/virtualenvs/my_project/lib/python3.7/re.py:175: in match
    return _compile(pattern, flags).match(string)
../../.local/share/virtualenvs/my_project/lib/python3.7/re.py:288: in _compile
    p = sre_compile.compile(pattern, flags)
../../.local/share/virtualenvs/my_project/lib/python3.7/sre_compile.py:764: in compile
    p = sre_parse.parse(p, flags)
../../.local/share/virtualenvs/my_project/lib/python3.7/sre_parse.py:924: in parse
    p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0)
../../.local/share/virtualenvs/my_project/lib/python3.7/sre_parse.py:420: in _parse_sub
    not nested and not items))

Current result

My current workaround this issue is to simply define the method_urls from my Response object manually as following:

method_urls = {
  'POST': {
    '^/v1/apps/(?P<application_id>.*)/messages$': 'SendMessages',
  },
}

Possible Fix

I tried changing the uri_to_regexp method from BaseResponse class in moto/core/responses.py and it seems to work in my case, but I'm not sure if this is a definitive solution or if this could cause issues somewhere else. Anyways, the change I tested was simply to add one more replace call in line 327:
replace('-', '_').

bug

Source