Moto: @mock_s3 permanently breaks requests library

[samfrances]

At first look, I thought this was a duplicate of https://github.com/spulec/moto/issues/303.

However, on further inspection, the problem is only present with the mock_s3 decorator, not with the mock_s3_deprecated decorator. Therefore, if my inspection of the code base is right, this looks like a problem related to the responses library or the use of it in moto, whereas https://github.com/spulec/moto/issues/303 concerned the HTTPretty library.

Anyway, the meat of the issue:

After using @mock_s3 in django tests, other test classes that use LiveserverTestCase and the requests library stop working, and fail with the error: ConnectionError: Connection refused.

This is not an inherent problem with responses itself, as the following test demonstrates:

python -c "from moto.packages.responses.responses import mock; import requests; mock.start(); mock.stop(); print requests.get('http://google.com')"
<Response [200]>

However, if mock.stop() is omitted, you see the same error as in the django test:

python -c "from moto.packages.responses.responses import mock; import requests; mock.start(); print requests.get('http://google.com')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/moto/packages/responses/responses.py", line 302, in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/moto/packages/responses/responses.py", line 244, in _on_request
    raise response
requests.exceptions.ConnectionError: Connection refused: GET http://google.com/

This suggests to me that, one way or another, moto is failing to call mock.stop().

moto version 1.0.1
Django version 1.11.3

[samfrances]

In addition, the tracebacks from the failing django unit tests themselves confirms that the requests are going through the responses library. For example:

Traceback (most recent call last):
  File "/home/REDACTED/tests.py", line 1099, in test_REDACTED
    result='passed')
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/sam/REDACTED/env/local/lib/python2.7/site-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/moto/packages/responses/responses.py", line 302, in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
  File "/home/REDACTED/env/local/lib/python2.7/site-packages/moto/packages/responses/responses.py", line 244, in _on_request
    raise response
ConnectionError: Connection refused: POST http://localhost:48507/api/REDACTED

This is from a test method in a test class using LiveserverTestCase and the requests library. This test class does not itself use moto, however, another test class that did use @mock_s3 as a class decorator seems to have failed to turn off responses when the test class finished.

[marcelchastain]

I can confirm similar behavior, but using @mock_s3_deprecated.
python 2.7.11 OSX
moto==1.0.1

[austinbyers]

Can confirm similar behavior, and I've tried both the decorator and raw usage (explicitly calling mock.start() and mock.stop()).

In my case, it is Lambda which is not properly restored:

# Python 3.6
import boto3, moto
mocks = [moto.mock_lambda(), moto.mock_s3()]
for mock in mocks:
    mock.start()
for mock in mocks:
    mock.stop()
boto3.client('lambda').get_function(FunctionName='...')

Traceback (most recent call last):
  File "venv/lib/python3.6/site-packages/botocore/client.py", line 310, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "venv/lib/python3.6/site-packages/botocore/client.py", line 586, in _make_api_call
    operation_model, request_dict)
  File "venv/lib/python3.6/site-packages/botocore/endpoint.py", line 141, in make_request
    return self._send_request(request_dict, operation_model)
  File "venv/lib/python3.6/site-packages/botocore/endpoint.py", line 170, in _send_request
    success_response, exception):
  File "venv/lib/python3.6/site-packages/botocore/endpoint.py", line 249, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "venv/lib/python3.6/site-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "venv/lib/python3.6/site-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 251, in __call__
    caught_exception)
  File "venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 317, in __call__
    caught_exception)
  File "venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "venv/lib/python3.6/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "venv/lib/python3.6/site-packages/botocore/endpoint.py", line 204, in _get_response
    proxies=self.proxies, timeout=self.timeout)
  File "venv/lib/python3.6/site-packages/botocore/vendored/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "venv/lib/python3.6/site-packages/moto/packages/responses/responses.py", line 302, in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
  File "venv/lib/python3.6/site-packages/moto/packages/responses/responses.py", line 244, in _on_request
    raise response
requests.exceptions.ConnectionError: Connection refused: GET https://lambda.REGION.amazonaws.com/2015-03-31/functions/FUNCTION_NAME

[austinbyers]

A workaround for this is to manually restore the HTTP adapter which is mocked out in responses.py:

import botocore
real_http_adapter = botocore.vendored.requests.adapters.HTTPAdapter.send

# run tests using moto

botocore.vendored.requests.adapters.HTTPAdapter.send = real_http_adapter
# boto3 operations work correctly now

[dbfr3qs]

I'm having this issue also. I'm pretty sure the problem occurs when you use more than one mock from moto, for example decorating a test with mock_s3 and mock_lambda. My guess is something is going on with disabling patching. This will fail:

    def _process_lamda(pfunc):
        zip_output = io.BytesIO()
        zip_file = zipfile.ZipFile(zip_output, 'w', zipfile.ZIP_DEFLATED)
        zip_file.writestr('lambda_function.zip', pfunc)
        zip_file.close()
        zip_output.seek(0)
        return zip_output.read()

    def get_test_zip_file():
        pfunc = """
    def lambda_handler(event, context):
        return event
    """
    return _process_lamda(pounce)

    @mock_lambda
    @mock_apigateway
    def test_lambda_invoke(self):
        conn = boto3.client('lambda', 'us-west-2')
        conn.create_function(
            FunctionName='testMotoFunction',
            Runtime='python2.7',
            Role='test-iam-role',
            Handler='lambda_function.handler',
            Code={
                'ZipFile': get_test_zip_file(),
            },
            Description='test lambda function',
            Timeout=3,
            MemorySize=128,
            Publish=True,
            )

    def test_request(self):
        import requests
        r = requests.get('https://www.google.com')
        self.assertEqual(r.status_code, 200)

Traceback (most recent call last):
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/app/tests/common_tests/test_aws_lambda.py", line 36, in test_request
    r = requests.get('https://www.google.com')
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/moto/packages/responses/responses.py", line 308, in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
  File "/Users/chris.keogh/Documents/Workspace/report-card-app/flask/lib/python3.5/site-packages/moto/packages/responses/responses.py", line 250, in _on_request
    raise response
requests.exceptions.ConnectionError: Connection refused: GET https://www.google.com/

Both tests will pass when removing the @mock_apigateway decorator.

[kingbuzzman]

I can confirm using @mock_iam as well. Anything NOT mocked gets this requests.exceptions.ConnectionError: Connection refused: METHOD https://domain.name/ exception

[aop]

I'll confirm this issue is still live in 1.1.24.

For me the issue occurs when trying to use the responses library separately in addition with mock_s3

[terrycain]

Ok so this line mock = _default_mock = RequestsMock(assert_all_requests_are_fired=False, pass_through=False), is the culprit.

Hmm, I have a feeling if i set pass_through to True, things will break. Lets see.

[terrycain]

Ok well it seems I can change pass_through to True, without breaking any tests, results.

@JackDanger any thoughts on this, according to the tests it wont break anything. Only downside is it will allow non supported services to go to AWS but people should really know that some things are not supported as they would have had to @decorate somewhere.

Other option would be to do something like @mock_s3(pass_through=True)

[aop]

Is it so that moto can't now be used in conjuction with the pure responses library? I see that moto carries it's own "package" of the responses library. There seems to be a problem that the both patch the same target namely here: https://github.com/spulec/moto/blob/master/moto/packages/responses/responses.py#L312

I'm not sure what double patching the same target does but it probably kills the other patching meaning it can't be stopped, thus the target remains patched after. What is the reason moto carries it's own responses and doesn't use the responses package for instance? Earlier versions didn't have this problem.

[JackDanger]

@terrycain I'm kinda okay with non-supported services passing through to AWS. It seems reasonable to me that if you @mock_ec2 it goes to Moto and if you don't you get AWS, always (If I understand the decision we're trying to make correctly). @spulec WDYT?

[g-io]

Any news on this? I'm experiencing the same for requests to unrelated domains with the requests library while using @mock_...
Setting pass_through to True (like proposed) fixes the issue for me, but obviously that's out of my hands in most ci/cd scenarios.

[joarleymoraes]

the pass_through won't work when using start() and stop() approach:

TypeError: start() got an unexpected keyword argument 'pass_through'

[amagee]

@g-io I have worked around the problem with the following monkeypatch:

# https://github.com/spulec/moto/issues/1026#issuecomment-343525897
# https://github.com/terrycain/moto/blob/0e451fc1d1954f4fae750563f9cb290276b2e5c2/moto/packages/responses/responses.py
from moto.packages.responses import responses

responses.mock = responses._default_mock = responses.RequestsMock(
    assert_all_requests_are_fired=False,
    pass_through=True
)
for __attr in (a for a in dir(responses._default_mock) if not a.startswith('_')):
    setattr(responses, __attr, getattr(responses._default_mock, __attr))

[hoangelos]

So, maybe I'm missing this. However, how is this even a solution? Doesn't this just instead of fixing the monkey patching make the mock able to connect to other URLs? We are getting this error too. But we only get it when you use two together. I still don't see a solution that seems to fix this without adding an ability to make the mocked request connect, which doesn't seem like a solution. Also, the issue is worse. We have other requests failing in other testing files altogether because requests is being permanently set to Moto's request. The other test doesn't use moto at all. It does use httpretty. All is fixed by removing the additional mock. Not using two just one.

[killthrush]

This issue is biting us as well - we're unable to use AWS services hosted in LocalStack docker containers if moto mocks are used anywhere in our test suite. I tried @amagee 's workaround successfully and got past my initial problem which was preventing me from talking to docker at all. However now in botocore's urllib connection pool, I'm running into an odd error with an isinstance check on a Timeout class of all things - is it possible that this is a mocked resource that isn't being cleaned up either? Haven't been able to piece it together yet.

[hoangelos]

@killthrush This is the same issue. Two things are patched. requests and botocore.

[dmulter]

seeing this too...

[hoangelos]

Sadly I haven't been able to come back to this, but the interesting problem is the depreciated methods, which use httpretty work. It's the "new" ones that are broke. :-D

[haidaraM]

Same issue here when using multiple annotations on class with unittest. My tests include some tests with moto and some without. Using moto breaks the other tests

[martindurant]

There are some workarounds suggested here to modify the moto patches. Is there a simple way to absolutely ensure that moto's patching is simply removed, so that, thereafter, requests works as before. I have tried reloading some requests submodules, but to no avail.

[lsutic]

I've had many problems with shared use of the global responses mock object. Using a local RequestsMock has always fixed it.

Replacing the RequestsAWSMock with a version that doesn't use the global responses mock solves this problem for me. Note that S3 uses its own decorator (S3Backend.decorator), whereas other moto modules uses moto.core.base_decorator(backend_class). This is a drop-in solution for S3, but not for the other APIs, although I think the other APIs can be fixed using the same method.

import moto

class LocalResponsesMockAWS(moto.core.models.BaseMockAWS):

    def __init__(self, *args, **kwargs):
        # Used by constructor, must use version of RequestsMock in moto.packages
        self.responses = moto.packages.responses.responses.RequestsMock(
            assert_all_requests_are_fired=False)
        super(LocalResponsesMockAWS, self).__init__(*args, **kwargs)

    def reset(self):
        self.responses.reset()

    def enable_patching(self):
        self.responses.start()
        for method in moto.core.models.RESPONSES_METHODS:
            for backend in self.backends_for_urls.values():
                for key, value in backend.urls.items():
                    callback = moto.core.utils.convert_flask_to_responses_response(value)
                    self.responses.add_callback(
                        method=method,
                        url=re.compile(key),
                        callback=callback,
                    )

        for pattern in self.responses._urls:
            pattern['stream'] = True

    def disable_patching(self):
        try:
            self.responses.stop()
        except AttributeError:
            pass
        self.responses.reset()


class LocalMockS3(moto.s3.models.S3Backend):

    @property
    def _url_module(self):
        # I'm sure this can be done in a more generic fashion, but now we solve
        # the problem with S3. 
        backend_module = moto.s3.models.S3Backend.__module__
        backend_urls_module_name = backend_module.replace("models", "urls")
        backend_urls_module = __import__(backend_urls_module_name, fromlist=[
                                         'url_bases', 'url_paths'])
        return backend_urls_module

    def decorator(self, func=None):
        mocked_backend = LocalResponsesMockAWS({'global': self})

        if func:
            return mocked_backend(func)
        else:
            return mocked_backend


mock_s3 = LocalMockS3().decorator

[spulec]

I've created a PR that should address this: https://github.com/spulec/moto/pull/1553

If anyone can test and confirm it works, it would be appreciated.

[spulec]

Closed with #1553

[yhvh]

Installed from master, this still fails. We have calls to different services interleaved with calls to aws, can you confirm this minimal use case is supported?

import moto, requests

@moto.mock_s3
def test_foo():
    print requests.get('https://www.google.co.uk')

    @moto.mock_s3
    def test_foo():
>       r = requests.get('https://www.google.co.uk')

test_foo.py:6: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/requests/api.py:72: in get
    return request('get', url, params=params, **kwargs)
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/requests/api.py:58: in request
    return session.request(method=method, url=url, **kwargs)
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/requests/sessions.py:508: in request
    resp = self.send(prep, **send_kwargs)
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/requests/sessions.py:618: in send
    r = adapter.send(request, **kwargs)
../../../.virtualenvs/testmoto/lib/python2.7/site-packages/responses.py:572: in unbound_on_send
    return self._on_request(adapter, request, *a, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <responses.RequestsMock object at 0x7f52e2515fd0>, adapter = <requests.adapters.HTTPAdapter object at 0x7f52df88dd90>, request = <PreparedRequest [GET]>, kwargs = {'cert': None, 'proxies': OrderedDict(), 'stream': False, 'timeout': None, ...}, match = None, resp_callback = None, error_msg = 'Connection refused: GET https://www.google.co.uk/'
response = ConnectionError(u'Connection refused: GET https://www.google.co.uk/',)

    def _on_request(self, adapter, request, **kwargs):
        match = self._find_match(request)
        resp_callback = self.response_callback

        if match is None:
            if request.url.startswith(self.passthru_prefixes):
                logger.info(
                    'request.allowed-passthru', extra={
                        'url': request.url,
                    })
                return _real_send(adapter, request)

            error_msg = 'Connection refused: {0} {1}'.format(
                request.method, request.url)
            response = ConnectionError(error_msg)
            response.request = request

            self._calls.add(request, response)
            response = resp_callback(response) if resp_callback else response
>           raise response
E           ConnectionError: Connection refused: GET https://www.google.co.uk/

../../../.virtualenvs/testmoto/lib/python2.7/site-packages/responses.py:543: ConnectionError

[JoostvDoorn]

@yhvh A workaround for this for now is:

import requests
import responses
import moto
import unittest

class TestRequests(unittest.TestCase):
    @moto.mock_s3
    def test_requests(self):
        responses.add_passthru('https://')
        response = requests.get('https://www.google.com')
        self.assertEqual(response.status_code, 200)

unittest.main()

[martindurant]

Explicitly calling the following upon the end of the mocked portion also appears to fix the issue.

            httpretty.HTTPretty.disable()
            httpretty.HTTPretty.reset()

[Bennyelg]

same problem on mock_emr

[spulec]

Follow along on #1567 for more discussion of changing requests during tests.

[kingbuzzman]

As of moto 1.3.5 this works:

    import responses

    responses.add_passthru('https://')
    responses.add_passthru('http://')

[martindurant]

Note that as of the new version of botocore, it no longer uses responses, so likely the solution above will fail too, whenever moto is rewritten to follow the change in botocore.