Moto: Test cases written in moto using Latest version of boto3 fails

[krishnamee2004]

Test cases written in moto makes actual AWS API calls to botocore instead of mocking them. This happens with the latest version of boto3 (1.8.). It used to work fine without issues with 1.7. versions.

Sample code to reproduce error

import boto3
import json
from moto import mock_s3

@mock_s3
def test_mock_s3():
    client = boto3.client('s3', region_name='us-east-1')
    client.create_bucket(Bucket='testbucket')
    response = client.list_buckets()
    print json.dumps(response, default=str)

if __name__ == "__main__":
    test_mock_s3()

Expected result

Method should return the ListBuckets response. It should look something like:

{"Owner": {"DisplayName": "webfile", "ID": "bcaf1ffd86f41161ca5fb16fd081034f"}, "Buckets": [{"CreationDate": "2006-02-03 16:45:09+00:00", "Name": "testbucket"}], "ResponseMetadata": {"RetryAttempts": 0, "HTTPStatusCode": 200, "HTTPHeaders": {"Content-Type": "text/plain"}}}

Actual error

botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

Full stack trace

Traceback (most recent call last):
  File "testcases.py", line 14, in <module>
    test_mock_s3()
  File "/private/tmp/virtualenv2/lib/python2.7/site-packages/moto/core/models.py", line 71, in wrapper
    result = func(*args, **kwargs)
  File "testcases.py", line 8, in test_mock_s3
    client.create_bucket(Bucket='testbucket')
  File "/private/tmp/virtualenv2/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/private/tmp/virtualenv2/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

Library versions

moto : 1.3.4
boto3 : 1.8.1 - fails
boto3 : 1.7.84 - succeeds

[yingyi-nauto]

Similar problem with mock_sqs. Got the following error:
ClientError: An error occurred (AccessDenied) when calling the CreateQueue operation: Access to the resource https://queue.amazonaws.com/ is denied.

[rouge8]

It looks like the issue is actually with botocore >= 1.11.0 which no longer uses requests and instead directly uses urllib3: https://github.com/boto/botocore/pull/1495. This means moto probably can't use responses anymore...

[garyd203]

Obviously we want to update moto so that it can work with the new implementation of botocore.

Whilst we figure out what this looks like, is it worthwhile restricting moto's install_requires in setup.py to require only compatible versions of botocore? Clearly this isn't a perfect solution, but if it saves some pain for some of our users then it could be worthwhile.

EDIT: I have added a PR for this as a discussion starter

[grudelsud]

pinning to boto3<1.8 is not really a solution as it would outdate the library pretty soon and doesn't solve when installing dependencies relying on newer versions either, as this will cause incompatibilities.
I appreciate this comment doesn't bring much to the table, but can we have an idea of how big of an effort would be to update moto to reflect the current status of libraries?

[garyd203]

@grudelsud I agree that pinning boto3 is not a viable long-term solution, and introduces problems of its own. But given that users are currently being affected by tests silently falling back to making real boto3 calls (which can have significant unexpected side-effects if you happen to have valid AWS credentials available), it seems worthwhile to consider a quick solution as well a permanent solution. I feel that, for now, the problems from pinning boto3 are lower severity than doing nothing.

That said, I would be a million times happier if someone put forward a genuine fix. But I lack the time and knowledge to do that myself at this point, or even to guess how much work would be involved.

[joguSD]

@garyd203 I've been driving a lot of the changes to get botocore tracking upstream dependencies instead of using our vendored versions. What does moto require so that you guys wouldn't need to monkey patch our dependencies?

[garyd203]

Hi @joguSD , thanks for asking. I'm just an occasional contributor to moto myself, and I'm not very familiar with the internals of moto. That said, I will try to provide some comments...

The goal of moto is to provide a fake implementation of specified AWS services which are accessed via boto3, as if they were the live services. FWIW, my understanding of the current implementation is that we have achieved this by mocking out HTTPAdapter.send in botocore's vendored version of requests,so that we can inspect each request and either pass it off to our internal handler for the fake service, or pass it through to the original send. You can see this in moto.core.models, with botocore_mock and ResponsesMockAWS

Moving forward, there seem to be a few choices for how we could do this better. I suspect we are constrained to work with the HTTP request rather than some other part of the boto3/botocore stack. So just to start the discussion, here's a couple of options:

1. Use a pluggable HTTP client backend for botocore, so that moto can wrap the standard HTTP backend with it's own interceptor functionality
2. Add a filter/interceptor chain in botocore's HTTP request handling, where moto can inject it's own filter early on and modify the behaviour based on what request is being made.

Do you have any opinion on these (or other) implementation choices, from botocore's perspective? My personal preference would be a pluggable backend with a standard implementation that is extensible and/or wrappable.

Again, don't take any of this discussion as definitive (I can't speak for the moto project maintainers), but I hope it helps.

[jbaum-cmcrc]

Can we have a release for the temporary work-around, please?

Everyone has to pin to boto3<1.8 until the proper fix is made in any case; it would be better to have that pin in one place (moto) rather than in every project that depends on it.

[spulec]

As a temporary measure, I've merged #1794 and released version 1.3.5.

If anyone does still want to use the newer boto3, it should work properly with the deprecated decorators (@mock_ecs_deprecated, etc). Those decorators mock at the socket level. We had been wanting to move away from that because they are fragile, but we may need to reconsider now.

Another option is to do a comparable thing to what we were doing before, but for urllib3. There looks to be at least one library out there: https://github.com/florentx/urllib3-mock

Finally another option is to stop patching all together and switch to moto only supporting standalone mode. I don't particularly love this, but if it means avoiding hacky patching issues like this in the future, I think we seriously need to consider it.

So, we've released a temporary fix for now, but we need to think through some bigger picture questions in the near-term so people aren't locked to old boto3/botocore. Since this is probably as good a place as any, feel free to comment here with thoughts.

[joguSD]

@spulec I'd rather we reach some sort of compromise that allows you to continue doing what you are now without having to patch anything.

@garyd203 I personally really like option 1, and is a large reason I've begun working on refactoring how botocore makes HTTP requests. I'd like to eventually get to a place where it's relatively easy to provide alternative HTTP clients, but I'm not sure how far out that is as we'll have to solidify a lot of interfaces we've considered internal to botocore.

As for option 2, that seems a little more reasonable in the short term. We've been playing with the idea of a before-send event in botocore that would allow custom handlers to return an HTTP response instead of going through botocore's default HTTP layer.

[joguSD]

I can't make any promises yet but something like this is what I was thinking for option 2:

from botocore.session import Session
from botocore.awsrequest import AWSResponse

class RawResponse(object):
    # a stubbed out urllib3-like response object
    def stream(self):
        yield b''

def stub_http(event_name, request, **kwargs):
    # return None here to let the botocore actually send the request
    return AWSResponse(request.url, 200, {}, RawResponse())

client = Session().create_client('s3')
# NOTE: this event doesn't exist
client.meta.events.register('before-send', stub_http)
# gets a stubbed response
print(client.list_buckets())
client.meta.events.unregister('before-send', stub_http)
# gets a real response
print(client.list_buckets())

To actually get something like this pushed through we would need to finalize and make the AWSRequest and AWSResponse interfaces public, and make what we expect from a raw response solid.

I think the last blocker would be injecting the event handler without patching which might not be possible given moto's current interface (no direct reference to the client/session).

[will-ockmore]

Just ran into this issue with an internal project, the dependencies were not pinned so when running on the CI server the newer version was installed and ran against our infrastructure. Thankfully nothing serious was changed by the tests.

The sub-dependency pinning fixes (#1794 , #1800 ) are possibly not sufficient - and I think for such a potentially damaging / expensive problem further steps should be taken (error on import if the versions are not correct perhaps?)

Obviously best practice is to pin everything, but sadly this doesn't happen all the time!

With the latest releases available on PyPI, steps to access your real AWS resources with no errors / pip install warnings (using python 3.6.4 on my mac):

# set up environment
$ python -m venv env && source env/bin/activate && pip install boto3 moto
$ python
Python 3.6.4 (default, Mar 12 2018, 13:30:09)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> import moto
>>> with moto.mock_s3():
...     s3 = boto3.resource('s3')
...     print([x for x in s3.buckets.all()])

# see the list of all your real s3 buckets

Install order matters, if you install moto first (eg. pip install moto boto3) it correctly pins botocore; but as this can't be depended on the problem remains.

[michaelPotter]

I agree with @will-ockmore; dependency management can be a tricky thing to get right. Checking the botocore version and raising an error would not only be more reliable in preventing access to real systems, but it would also inform users as to why their tests might have suddenly stopped working.

[michael-k]

more reliable in preventing access to real systems

You should never rely on moto or any other library to prevent (accidental) access to your production resources. The only way to prevent this is to not use production credentials outside of production (dev, test, stage, …).

[will-ockmore]

You should never rely on moto or any other library to prevent (accidental) access to your production resources. The only way to prevent this is to not use production credentials outside of production (dev, test, stage, …).

Totally agreed; but you have to account for the human factor (like junior developers!).

Another thing to bear in mind is that moto doesn't require credentials in the same way as boto, so if you have your local aws credentials set up (eg. in ~/.aws) and you run the tests locally on a project with no credentials override, you will encounter the same problem.

Thinking out loud - but perhaps moto should require credentials - and warn that they shouldn't be your real creds?

[2rs2ts]

I mean, we don't do development in production environments, but it's still expensive when your test suite spins up a couple dozen VPCs and EC2 instances.

[luyun-aa]

I ran into this issue when using moto with aws-xray-sdk. I'm working on a fix: https://github.com/spulec/moto/pull/1808

[luyun-aa]

Hi there,

I was rushing to a working implementation and wasn't aware of the discussion above until I've completed most part of the patch, in PR #1808. Basically it took an intuitive approach to mock botocore.httpsession.URLLib3Session.send instead of requests. To make it work, I need to:

• introduce a modified version of the responses library, tailored for URLLib3Session and AWSRequest
• test the vendored responses library
• update tests to send http request by URLLib3Session
• wrap request and response object to make them similar to that of requests

I've also tried another two approaches:

• patching requests and URLLib3Session at the same time: doesn't work because their request objects are way too different
• patching at urllib3 level:
  
  
  
  • it's not too much different from patching URLLib3Session because URLLib3Session did little to massage the response of urllib3
  
  
  • as commented in the docstring, URLLib3Session is equivalent to HTTPAdapter in requests, their send method are comparable
  
  

For a longer term solution, I'd prefer @garyd203's option 2 as it requires less engineering effort from both botocore and moto. It would be something like wsgi application that moto can return a response of its own.

[joguSD]

@luyun-aa As you found out, the new classes in botocore are similar to requests but not quite the same so I don't think switching the patch target will work completely. If the short term solution is just "switch the patch target" then using the deprecated decorators will work.

Also, to reiterate, I strongly suggest not implementing moto's new mechanism by patching botocore's HTTP client library as we cannot guarantee that urllib3 (or even httplib) will be used to carry out all HTTP requests in the future.

As an update, I've been playing with @garyd203's option 2 and have a proof of concept that mostly works. A few issues I'm running into are:

• It's difficult to be as dynamic as patching is in terms of when the mock is enabled / disabled and when clients are created. With the event approach, clients created before the mock is enabled are unaffected.
• The InstanceMetadataFetcher and ContainerMetadataFetcher were previously implemented using requests but now use botocore's URLLib3Session, meaning you won't pick up on the mocked credentials if you were relying on that. And the way things are currently structured it would be fairly difficult to add an event to stub these requests in similar manner.

[bernardgardner]

The fix applied in moto 1.3.5 appears to be incomplete - when using Pipenv on a clean environment to install moto, dependency resolution fails, because moto wants a version of botocore < 1.11 while the current version of boto3 (1.9.4) has a requirement for botocore >= 1.12.4 (so you still have to pin boto3 < 1.8 - which wanted botocore 1.11.0)

I think you probably need to also pin boto3 in your install_requires.

Steps to repro:

• Install pipenv
• mkdir moto-repro && cd moto-repro
• pipenv install moto # installs packages and errors out trying to generate Pipfile.lock
• pipenv graph # shows installed versions and dependency tree

Note pipenv install 'boto3<1.8.0' moto works

This might (also) be considered a failure in pipenv's dependency resolution but at the moment, moto isn't installable using pipenv without doing a bunch of troubleshooting

[yan12125]

@bernardgardner That is indeed a moto issue and a fix is proposed in https://github.com/spulec/moto/pull/1801

[4lph4-Ph4un]

What's the current situation with this? :)
The long term solution, to be exact!

[joguSD]

@4lph4-Ph4un I've opened a PR that begins the work to move moto off of patching libraries to implement the core functionality and instead use the before-send event we recently added in botocore. There's still a few kinks that need to be worked out, however.

@spulec Have you had a chance to look at #1847?

[4lph4-Ph4un]

@joguSD, you have no idea how happy I'm to hear that it's been worked on! 👍 😄

[lhufnagel]

It looks like one can get the test-suite to pass when mocking out the responses calls (like done before), on top of @joguSD's fixes https://github.com/joguSD/moto/pull/1

[MartinThoma]

What is the currently recommended solution for this problem?

I have the following installed:

boto==2.48.0
boto3==1.9.27
botocore==1.12.27
moto==1.3.6

and still mock_s3 is not mocking s3:

>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the CreateBucket operation: The AWS Access Key Id you provided does not exist in our records.

/home/math/.local/lib/python3.5/site-packages/botocore/client.py:623: ClientError

If I set the version of boto3 to 1.7.84, I get:

E           requests.exceptions.ConnectionError: Connection refused: POST https://sts.amazonaws.com/

/home/math/.local/lib/python3.5/site-packages/responses.py:543: ConnectionError

which looks similar to https://github.com/spulec/moto/issues/1026

Other combinations which DONT work for me:

boto==2.49.0
boto3==1.9.27
botocore==1.12.27
moto==1.3.6

boto==2.49.0
boto3==1.7.84
botocore==1.10.84
moto==1.3.6

[danmaas]

This Pip spec works for me at the moment (for SQS - S3 not tested):

boto3>=1.7.84,<1.8
moto>=1.3.4

As mentioned above, it might be necessary to have some AWS credentials in the environment, even if they are just "dummy" ones, e.g.

AWS_ACCESS_KEY_ID=dummy-access-key
AWS_SECRET_ACCESS_KEY=dummy-access-key-secret
AWS_DEFAULT_REGION=us-east-1

[mbelang]

Same here with

boto==2.49.0
boto3==1.7.84
botocore==1.10.84
moto==1.3.6

Have to set environment variables for AWS otherwise it doesn't work.

[spulec]

This is fixed with https://github.com/spulec/moto/pull/1907

[gdoron]

@spulec Thanks a lot, for the fix and for this amazing package!
When can we expect it to be on pypi?

[gilbsgilbs]

@gdoron it's shipped with moto 1.3.7 which is already available on PyPi. You might have forgotten to specify dummy credentials in your environment variables (like me). See https://github.com/spulec/moto/pull/1907/files#diff-354f30a63fb0907d4ad57269548329e3R26 .

[mikegrima]

Noticing an issue with PynamoDB (https://github.com/pynamodb/PynamoDB).

It appears to be making use of the botocore vendored requests library: https://github.com/pynamodb/PynamoDB/blob/master/pynamodb/connection/base.py#L19-L20

Unfortunately, with the latest moto I'm getting:

 pynamodb.exceptions.VerboseClientError: An error occurred (UnrecognizedClientException) on request (...) on table (...) when calling the DescribeTable operation: The security token included in the request is invalid.

@joguSD: what happens now for libraries that are still utilizing the vendored requests in botocore? Are these easy fixes?

EDIT: I just tried overriding their Requests session library as documented here: https://github.com/pynamodb/PynamoDB/issues/558, and it now works 👏 .

In the meantime, I am going to submit a PR to PynamoDB that swaps out the vendored Requests for the normal Requests library and hopefully that will work 🤞 .

[alexcasalboni]

Any updates on this?

I've updated to

boto==2.49.0
boto3==1.9.47
botocore==1.12.47
moto==1.3.7

and I am still seeing @mikegrima's error when using @mock_ssm:

ClientError: An error occurred (UnrecognizedClientException) when calling the PutParameter operation: The security token included in the request is invalid.

Plus, I get another error when using @mock_secretsmanager:

ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "https://secretsmanager.eu-west-1.amazonaws.com/".

[johnnyplaydrums]

Also checking in on this, I thought the 1.3.7 release fixed this, maybe I'm missing something. Thank you in advance for the help! I have a file test.py and a fresh virtualenv where I just installed moto with pip install moto (1.3.7):

import boto3

from moto import mock_ec2

@mock_ec2
def test_mock_ec2():
    client = boto3.client('ec2', region_name='us-east-1')
    print(client.describe_images()['Images'][0]['ImageId'])

if __name__ == "__main__":
    test_mock_ec2()

Running this gives me:

(venv) ~/sandbox $ python test.py
botocore.exceptions.NoCredentialsError: Unable to locate credential

With fake credentials, I am able to get the correct response:

(venv) ~/sandbox $ AWS_ACCESS_KEY_ID=fake AWS_SECRET_ACCESS_KEY=fake python test.py
ami-03cf127a

Relevant dependencies from my virtualenv:

boto3==1.9.48
botocore==1.12.48
moto==1.3.7

I got confused by some of the discussion above: Is supposed supposed to need fake credentials in order to run? I figured it would mock all that out. Thanks again!

[lhufnagel]

Yeah, currently you need to export fake credentials - I opened https://github.com/spulec/moto/pull/1952 to do that for you, but am hoping someone has a better fix

[johnnyplaydrums]

thanks for the prompt response @lhufnagel. I'll keep my eye on it and lend a hand if I can free up some time.

[jethrolam]

Just reporting that I can still reproduce the error with boto3==1.9.70 and moto==1.3.7 by the following:

import boto3
boto3.client('s3')  # ***By having this line, mock_s3 failed and real S3 was created***
import json
from moto import mock_s3

@mock_s3
def test_mock_s3():
    client = boto3.client('s3', region_name='us-east-1')
    client.create_bucket(Bucket='testbucket')
    response = client.list_buckets()
    print json.dumps(response, default=str)

if __name__ == "__main__":
    test_mock_s3()

This runs on

boto3==1.9.70
botocore==1.12.40
moto==1.3.7

Hope this helps.

[luisdemarchi]

Hello,

Just registering that with all libraries in the latest version Dynamodb in other regions works only with @mock_dynamodb2_deprecated

[luisdemarchi]

I discovered what it is, the problem is apparently in Pynamodb, but with the deprecated function it works. Here's my solution for @mock_dynamodb2:

model > config.py

from pynamodb.connection import Connection
_conn = Connection(region='us-west-2')
session_cls = _conn.session_cls

model > user.py

from model.config import session_cls
...

class User(Model):
    class Meta:
        table_name = '{}-user'.format(PROJECT_PREFIX)
        region = 'us-west-2'
        session_cls = session_cls

    user_id = UnicodeAttribute(hash_key=True, default=utils.generate_uuid)

test.py

from model import config
config.session_cls = requests.Session
from moto import mock_sqs, mock_dynamodb2
from model.user import User

@mock_dynamodb2
def setUp():
    User.create_table(read_capacity_units=1, write_capacity_units=1, wait=True)

[mikegrima]

@luisdemarchi This might be related to the requests version that is being used by PynamoDB. Can you try testing the fix outlined here: https://github.com/pynamodb/PynamoDB/issues/558 ?

[nebi-frame]

Any solution so far?

[mikegrima]

@nebi-frame Have you tried the solution outlined in pynamodb/PynamoDB#558 (assuming you are talking about the Dynamo issue)

[nebi-frame]

no not tried yet @mikegrima what about u?

[halfdan]

Just wanted to report that the issue @jethrolam posted is still valid: https://github.com/spulec/moto/issues/1793#issuecomment-449475823

This has to do with import order! We fixed all of our issues by upgrading moto to latest master _and_ running from moto import mock_s3 as early as possibly (before running any boto3.client/boto3.resource calls).

```
import boto3
import json
from moto import mock_s3
boto3.client('s3') # Moving this line to _after_ the import works. Whatever boto3.client('s3') does internally is creating an instance that is not mocked by moto

@mock_s3
def test_mock_s3():
client = boto3.client('s3', region_name='us-east-1')
client.create_bucket(Bucket='testbucket')
response = client.list_buckets()
print json.dumps(response, default=str)

if __name__ == "__main__":
test_mock_s3()

[mikegrima]

@halfdan Just curious, what is the purpose of having a boto3.client in the same file defined outside of the test cases? It would make more sense to me to have the client defined within each unit test itself (of course after the from moto import ... at the top of the file) (or even through a pytest fixture).

[jethrolam]

@mikegrima In my case, the line boto3.client resides in an external dependency foo that I don't have immediate control. So import foo broke mock_s3 silently.

[mikegrima]

@jethrolam Would you be able to import your dependency locally in the unit test first?

I also wonder if you can mock.patch() the boto3.client in your unit test with the moto mocked one.

[jethrolam]

Once we discovered import foo was the culprit, we resolved the problem easily by modifying foo. We did not attempt other solutions.

With that said, I want to highlight that it was the discovery of the culprit (import foo in my case) that is the most difficult.

[chris-erickson]

Can you or anyone highlight some preferred strategies to structuring project with Boto involved (isolating it and consistently importing it or something)? Seems like I never find a structure that doesn't cause me these sorts of troubles at some point.

[mikegrima]

Personally, I always use local imports for the code I want to test in my test functions. This helps to isolate package import issues. I'm not totally sure how "proper Pythonic" it is, but it works really well without giving me any headaches.

[nebi-frame]

@jethrolam can you talk about how you did modify import foo thing, because I have the same exact problem, I do import boto3 in a local file, and it broke everything

[jethrolam]

In my case it is not import boto3 but boto3.client('s3') that broke the test. My solution was just to move the latter from module level into function level so it would not be called at all during import foo.

[mikegrima]

@jethrolam Can you run your import foo from within def test_thing(): ? I'm wondering if that is a better fix than making a new boto3 client each time.

[kadusumilli1]

Hi,

I am getting the following error when I was writing tests to mock SQS queue using moto.

botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the CreateQueue operation: The security token included in the request is invalid.

Has anyone seen this before? I'm using:

moto==1.3.8
boto3==1.9.134
botocore==1.12.134

Appreciate your inputs. Thanks!

[mikegrima]

@kadusumilli1 You'll need to have a fixture or an equivalent that mocks out the AWS credentials via an environment variable, like so: https://github.com/Netflix-Skunkworks/swag-api/blob/master/swag_api/tests/conftest.py#L16-L22

You will need to have that before you declare your mocks.

[vkartik97]

I solved it using

botocore==1.12.86
git+https://github.com/spulec/moto.git@df493ea18de22b4533073eee3e296686019911c2

Installing via commit till moto>1.3.8

[dci-aloughran]

I'm using identical code to what was running yesterday, and am now seeing this issue:
botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the CreateBucket operation: The AWS Access Key Id you provided does not exist in our records.

I've tried with decorator, with 'with', and with mock.start(), and each time (when running in pytest) i get an error:

@pytest.fixture(scope="function", autouse=True)
def create_buckets():
"""Creates some mock s3 buckets before the start of every test"""
mock = mock_s3()
print(mock)
mock.start()
conn = boto3.resource("s3", region_name=settings.REGION_NAME)
conn.create_bucket(Bucket=settings.RUN_BUCKET)
conn.create_bucket(Bucket=settings.INGESTION_BUCKET)
s3 = boto3.client("s3", region_name=settings.REGION_NAME)
response = s3.list_buckets()
# Output the bucket names
print("Existing buckets:")
for bucket in response["Buckets"]:
print(f' {bucket["Name"]}')
mock.stop()

[dci-aloughran]

```@pytest.fixture(autouse=False, scope="function")
def create_buckets():
"""Creates some mock s3 buckets before the start of every test"""
mock = mock_s3()
mock.start()
s3 = boto3.resource('s3')

  s3.create_bucket(Bucket=settings.RUN_BUCKET)

conftest.py:21:

---

venv/lib/python3.6/site-packages/boto3/resources/factory.py:520: in do_action
response = action(self, *args, kwargs)
venv/lib/python3.6/site-packages/boto3/resources/action.py:83: in __call__
response = getattr(parent.meta.client, operation_name)(params)
venv/lib/python3.6/site-packages/botocore/client.py:357: in _api_call
return self._make_api_call(operation_name, kwargs)

---

self =
operation_name = 'CreateBucket'
api_params = {'Bucket': 'example-local-product-bucket'}

def _make_api_call(self, operation_name, api_params):
    operation_model = self._service_model.operation_model(operation_name)
    service_name = self._service_model.service_name
    history_recorder.record('API_CALL', {
        'service': service_name,
        'operation': operation_name,
        'params': api_params,
    })
    if operation_model.deprecated:
        logger.debug('Warning: %s.%s() is deprecated',
                     service_name, operation_name)
    request_context = {
        'client_region': self.meta.region_name,
        'client_config': self.meta.config,
        'has_streaming_input': operation_model.has_streaming_input,
        'auth_type': operation_model.auth_type,
    }
    request_dict = self._convert_to_request_dict(
        api_params, operation_model, context=request_context)

    service_id = self._service_model.service_id.hyphenize()
    handler, event_response = self.meta.events.emit_until_response(
        'before-call.{service_id}.{operation_name}'.format(
            service_id=service_id,
            operation_name=operation_name),
        model=operation_model, params=request_dict,
        request_signer=self._request_signer, context=request_context)

    if event_response is not None:
        http, parsed_response = event_response
    else:
        http, parsed_response = self._make_request(
            operation_model, request_dict, request_context)

    self.meta.events.emit(
        'after-call.{service_id}.{operation_name}'.format(
            service_id=service_id,
            operation_name=operation_name),
        http_response=http, parsed=parsed_response,
        model=operation_model, context=request_context
    )

    if http.status_code >= 300:
        error_code = parsed_response.get("Error", {}).get("Code")
        error_class = self.exceptions.from_code(error_code)

      raise error_class(parsed_response, operation_name)

E botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the CreateBucket operation: The AWS Access Key Id you provided does not exist in our records.
```

[pmatias]

I've run into the same problem (moto not actually mocking the requests), upgrading to 1.3.10 did the trick.

[monkut]

blindly upgraded boto3/moto, and just accidentally deleted buckets from my aws account trying to run testcases locally where mock_s3() was being used... (luckily I noticed some funky bucket names and managed to _ctrl-c_ before too much damage was done...

:(

reminder to self, use switchrole and don't have your default aws access account have access to anything valuable..

I think I'll be migrating my testcases from using/calling moto's mock_s3() directly in my testcases and use a localstack (which includes moto) container. At least that way I can set the endpoint_url and be certain (?) that boto3 isn't accessing my real account...

[mikegrima]

It think we should add some documentation to the readme to help clarify how to avoid issues similar to @monkut 's. It's not great to hear that your unit tests manipulated your real environment, and that is a major problem.

I'm a big fan of pytest and pytest fixtures. For all of my moto tests, I have a conftest.py file where I define the following fixtures:

@pytest.fixture(scope='function')
def aws_credentials():
    """Mocked AWS Credentials for moto."""
    os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
    os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
    os.environ['AWS_SECURITY_TOKEN'] = 'testing'
    os.environ['AWS_SESSION_TOKEN'] = 'testing'

@pytest.fixture(scope='function')
def s3(aws_credentials):
    with mock_s3():
        yield boto3.client('s3', region_name='us-east-1')


@pytest.fixture(scope='function')
def sts(aws_credentials):
    with mock_sts():
        yield boto3.client('sts', region_name='us-east-1')


@pytest.fixture(scope='function')
def cloudwatch(aws_credentials):
    with mock_cloudwatch():
        yield boto3.client('cloudwatch', region_name='us-east-1')

... etc.

All of the AWS/mocked fixtures take in a parameter of aws_credentials, which sets the proper fake environment variables -- which is needed. Then, for when I need to do anything with the mocked AWS environment, I do something like:

def test_create_bucket(s3):
    # s3 is a fixture defined above that yields a boto3 s3 client.
    # Feel free to instantiate another boto3 S3 client -- Keep note of the region though.
    s3.create_bucket(Bucket="somebucket")

    result = s3.list_buckets()
    assert len(result['Buckets']) == 1
    assert result['Buckets'][0]['Name'] == 'somebucket'

Taking this approach works for all of my tests. I have had some issues with Tox and Travis CI occasionally -- and typically I need to do something along the lines of touch ~/.aws/credentials for those to work. However, using the latest moto, boto3, and the fixtures I have above seems to always work for me without issues.

Also Protip:
This might be controversial, but I always make use of in-function import statements _for my unit tests_. This ensures that the mocks are run before any of the actual code is.
Example:

def test_something(s3):
   from some.package.that.does.something.with.s3 import some_func # <-- In function import for unit test
   # ^^ Importing here ensures that the mock has been established.      

   sume_func()  # The mock has been established from the fixture, so this function that uses
                # an S3 client will properly use the mock and not reach out to AWS.

[Freyja-Folkvangr]

It think we should add some documentation to the readme to help clarify how to avoid issues similar to @monkut 's. It's not great to hear that your unit tests manipulated your real environment, and that is a major problem.

I'm a big fan of pytest and pytest fixtures. For all of my moto tests, I have a conftest.py file where I define the following fixtures:

@pytest.fixture(scope='function')
def aws_credentials():
    """Mocked AWS Credentials for moto."""
    os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
    os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
    os.environ['AWS_SECURITY_TOKEN'] = 'testing'
    os.environ['AWS_SESSION_TOKEN'] = 'testing'

@pytest.fixture(scope='function')
def s3(aws_credentials):
    with mock_s3():
        yield boto3.client('s3', region_name='us-east-1')


@pytest.fixture(scope='function')
def sts(aws_credentials):
    with mock_sts():
        yield boto3.client('sts', region_name='us-east-1')


@pytest.fixture(scope='function')
def cloudwatch(aws_credentials):
    with mock_cloudwatch():
        yield boto3.client('cloudwatch', region_name='us-east-1')

... etc.

All of the AWS/mocked fixtures take in a parameter of aws_credentials, which sets the proper fake environment variables -- which is needed. Then, for when I need to do anything with the mocked AWS environment, I do something like:

def test_create_bucket(s3):
    # s3 is a fixture defined above that yields a boto3 s3 client.
    # Feel free to instantiate another boto3 S3 client -- Keep note of the region though.
    s3.create_bucket(Bucket="somebucket")

    result = s3.list_buckets()
    assert len(result['Buckets']) == 1
    assert result['Buckets'][0]['Name'] == 'somebucket'

Taking this approach works for all of my tests. I have had some issues with Tox and Travis CI occasionally -- and typically I need to do something along the lines of touch ~/.aws/credentials for those to work. However, using the latest moto, boto3, and the fixtures I have above seems to always work for me without issues.

Finally!

[pskowronek]

I would say, that people shouldn't be afraid to run unit tests that use _moto_ risking their real AWS envs being modified. I don't suspect that there are some folks that would like to run tests that use both real AWS and still mock some bits of AWS.
What I'm trying to say is that currently to use _moto_ requires a lot of attention and even that may go in vain with a newer version of AWS libs or _moto_. Maybe we should all together put some pressure on Amazon so they expose some mocking points to make everyone's life easier? Amazon has their feedback page IIRC.

[yitzikc]

I can still reproduce the issue with S3 and SQS tests on
boto3 = "==1.9.189" botocore = "==1.12.189" moto = "==1.3.13"
Same with moto = "==1.3.10"

[mikegrima]

@yitzikc Are your mocks running before your code is executed?

[mikegrima]

I just added a Protip to my post above, which should assist users when designing their tests with moto.

[EdwynPeignon]

I got the same problem and my solution was to import the moto librairies before the boto3 librairie.
There are certainly some conflicts between the librairies.
Hope it'll help some people :)

[yitzikc]

I got the same problem and my solution was to import the moto librairies before the boto3 librairie.
There are certainly some conflicts between the librairies.
Hope it'll help some people :)

Indeed when care is taken to import Moto before any imports of Boto3 or Botocore, the mocking works properly. I had to watch for imported modules which were importing Boto. Also, when running Pytest on multiple files, imports for tests in one file would interfere with the ones run subsequently. I had to import Moto in any test files that might import modules which ultimately import Boto.

[mikegrima]

I just made a PR to introduce more AWS Config features (#2363), and I updated the readme with the wording in my post above. Please review and let me know if there is anything else I should add:
https://github.com/spulec/moto/blob/1c268e3580b0976c9867b50124f665320c188148/README.md#very-important----recommended-usage

[jayai2014]

I got the same problem and my solution was to import the moto librairies before the boto3 librairie.
There are certainly some conflicts between the librairies.
Hope it'll help some people :)

Indeed when care is taken to import Moto before any imports of Boto3 or Botocore, the mocking works properly. I had to watch for imported modules which were importing Boto. Also, when running Pytest on multiple files, imports for tests in one file would interfere with the ones run subsequently. I had to import Moto in any test files that might import modules which ultimately import Boto.

Changing import order still not working to me 😢 . Using same version as boto3 = "==1.9.189" botocore = "==1.12.189" moto = "==1.3.13" / "==1.3.10"

[mikegrima]

The problem is not the import order -- it's the order upon which a boto client is instantiated. If it is instantiated BEFORE a mock is established, it won't work.

Please review: https://github.com/spulec/moto#very-important----recommended-usage

[pskowronek]

Hi guys, not sure if I understand everything correctly, but I allowed myself to report an issue to boto3 to make mocking easier - can anyone from _moto_ core team comment on https://github.com/boto/boto3/issues/2123 - maybe there's something _boto_ team could do to avoid such problems? This bug is not the only one reported to _moto_ - some of those issues are more than 1 year old and people still have problems that tests hit AWS servers.

[mikegrima]

I'm actually curious if I _permanently_ fixed this issue #2578

[mikegrima]

Can you all verify if the latest master fixes this issue? The changes in #2578 seems like it should fix this issue once and for all.

[karthikvadla]

@mikegrima, Thank you very much for your responses.
i have below exception even after mocking my dynamodb with pytest fixtures.

[CPython37:setup:stdout] >           raise error_class(parsed_response, operation_name)
[CPython37:setup:stdout] E           botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the DescribeTable operation: The security token included in the request is invalid.
[CPython37:setup:stdout] 

This is my source code file named db_utils.py

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import json
from dynamodb_json import json_util as db_json

import boto3
from botocore.exceptions import ClientError

from .converters import DecimalEncoder


class DynamoDbOperations:
    """
    Class to perform dynamo_db common operations.
    By default connects to us-west-2 (Oregon) region.
    """
    def __init__(self, table_name: str, region_name: str = 'us-west-2'):
        dynamo_db_resource = boto3.resource('dynamodb', region_name=region_name)

        # Use client to handle exceptions and resolve the error.
        dynamo_db_client = boto3.client('dynamodb', region_name=region_name)
        try:
            self._table = dynamo_db_resource.Table(table_name)
            print("{} table created on {}".format(table_name, self._table.creation_date_time))
        except dynamo_db_client.exceptions.ResourceNotFoundException as e:
            # Log error
            print("Error: {}".format(e))

    def insert_item(self, json_item):
        """
        Inserts item into dynamo_db table

        :param json_item: Item as Json object

        :return:
        """
        if type(json_item) is not dict:
            raise ValueError("Insert Item: {} must be json object".format(json_item))

        try:
            response = self._table.put_item(Item=json_item)
        except ClientError as ce:
            print(ce.response['Error']['Message'])
        else:
            print("PutItem succeeded")
            clean_response = db_json.loads(response)
            print(json.dumps(clean_response, indent=4, cls=DecimalEncoder))
            return clean_response

    def get_item(self, primary_key: dict):
        """
        Get item from table based on primary_key

        :param primary_key: Dictionary of partition_key and sort_key(optional)

        :return: Returns json object with primary key
        """
        if type(primary_key) is not dict:
            raise ValueError("primary_key: {} must be dictionary \
            of partition_key and sort_key(optional)".format(primary_key))

        try:
            response = self._table.get_item(Key=primary_key)
        except ClientError as ce:
            print(ce.response['Error']['Message'])
        else:
            item = response['Item']
            print("GetItem succeeded")
            clean_response = db_json.loads(item)
            print(json.dumps(clean_response, indent=4, cls=DecimalEncoder))
            return clean_response

    def modify_item(self, primary_key: dict,
                    update_expression: str,
                    expression_attribute_values: dict,
                    condition_expression: str = None,
                    return_values: str = "UPDATED_NEW"
                    ):
        """
        Update/Modify item based on primary_key.
        You can update values of existing attributes, add new attributes, or remove attributes,

        More info:
        Update Expression: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html
        Condition Expression: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html#Expressions.ConditionExpressions.SimpleComparisons

        :param primary_key: Dictionary of partition_key and sort_key(optional)
        :param update_expression: An update expression consists of one or more clauses.
        Each clause begins with a SET, REMOVE, ADD, or DELETE keyword. You can include any of these clauses
        in an update expression, in any order. However, each action keyword can appear only once
        :param condition_expression: To perform a conditional update
        :param expression_attribute_values: Attribute values to be updated/add/delete
        :param return_values: Return type after performing update

        :return: Returns updated json attributes
        """

        if type(primary_key) is not dict:
            raise ValueError("primary_key: {} must be dictionary \
            of partition_key and sort_key(optional)".format(primary_key))

        if 0 < len(primary_key) > 2:
            raise Exception("primary_key: {} must contain \
            partition_key and sort_key(optional) only".format(primary_key))

        if type(expression_attribute_values) is not dict:
            raise ValueError("expression_attribute_values: {} must be dictionary".format(expression_attribute_values))

        try:
            if condition_expression:
                response = self._table.update_item(
                    Key=primary_key,
                    UpdateExpression=update_expression,
                    ConditionExpression=condition_expression,
                    ExpressionAttributeValues=expression_attribute_values,
                    ReturnValues=return_values
                )
            else:
                response = self._table.update_item(
                    Key=primary_key,
                    UpdateExpression=update_expression,
                    ExpressionAttributeValues=expression_attribute_values,
                    ReturnValues=return_values
                )
        except ClientError as e:
            if e.response['Error']['Code'] == "ConditionalCheckFailedException":
                print(e.response['Error']['Message'])
            else:
                raise
        else:
            print("UpdateItem succeeded:")
            clean_response = db_json.loads(response)
            print(json.dumps(clean_response, indent=4, cls=DecimalEncoder))
            return clean_response

    def delete_item(self, primary_key:dict,
                    expression_attribute_values: dict = None,
                    condition_expression: str = None):
        """
        Deletes an item from table

        :param primary_key: Dictionary of partition_key and sort_key(optional)
        :param expression_attribute_values: Items with matching Attribute values to be deleted
        :param condition_expression: To perform a conditional delete
        :return:
        """
        if type(primary_key) is not dict:
            raise ValueError("primary_key: {} must be dictionary \
            of partition_key and sort_key(optional)".format(primary_key))

        if condition_expression:
            if expression_attribute_values is None:
                raise ValueError("expression_attribute_values: {} \
                must be provided".format(expression_attribute_values))
            elif type(expression_attribute_values) is not dict:
                raise ValueError("expression_attribute_values: {} must be a dictionary")

        try:
            if condition_expression:
                response = self._table.delete_item(
                    Key=primary_key,
                    ConditionExpression=condition_expression,
                    ExpressionAttributeValues=expression_attribute_values
                )
            else:
                response = self._table.delete_item(Key=primary_key)

        except ClientError as ce:
            if ce.response['Error']['Code'] == "ConditionalCheckFailedException":
                print(ce.response['Error']['Message'])
            else:
                raise
        else:
            print("DeleteItem succeeded:")
            clean_response = db_json.loads(response)
            print(json.dumps(clean_response, indent=4, cls=DecimalEncoder))
            return clean_response

here is my test file

from db_utils import DynamoDbOperations
from moto import mock_dynamodb2
import pytest
import boto3
import os

TEST_DYNAMO_TABLE_NAME = 'test'


@pytest.fixture(scope='function')
def aws_credentials():
    """Mocked AWS Credentials for moto."""
    os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
    os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
    os.environ['AWS_SECURITY_TOKEN'] = 'testing'
    os.environ['AWS_SESSION_TOKEN'] = 'testing'
    os.environ['AWS_DEFAULT_REGION'] = 'us-west-2'


@pytest.fixture
def dynamo_db_table(aws_credentials):
    def _table(table_name):
        with mock_dynamodb2():
            boto3.client('dynamodb').create_table(
                AttributeDefinitions=[
                    {'AttributeName': 'id', 'AttributeType': 'S'}
                ],
                TableName=f'{table_name}',
                KeySchema=[{'AttributeName': 'id', 'KeyType': 'HASH'}],
                ProvisionedThroughput={
                    'ReadCapacityUnits': 5,
                    'WriteCapacityUnits': 5,
                },
            )
            yield boto3.resource('dynamodb').Table(f'{table_name}')
    yield _table

def test_dynamo_db_utils_init(dynamo_db_table):
    DynamoDbOperations(TEST_DYNAMO_TABLE_NAME)

def test_dynamo_db_utils_insert_item(dynamo_db_table):
    json_item = {
        'id': '123',
        'name': 'karthik'
    }
    dynamo_db_table(TEST_DYNAMO_TABLE_NAME)
    db = DynamoDbOperations(TEST_DYNAMO_TABLE_NAME)
    response = db.insert_item(json_item)
    assert 200 in response

Anything i'm missing here. Appreciate your help.

[karthikvadla]

I used decorator around

@mock_dynamodb2
def test_dynamo_db_utils_init(dynamo_db_table):
    DynamoDbOperations(TEST_DYNAMO_TABLE_NAME)

The above invalid security token went off.
but now i see

test_dynamo_db_utils.py Error: An error occurred (ResourceNotFoundException) when calling the DescribeTable operation: Requested resource not found

Any idea how to mock this line
print("{} table created on {}".format(table_name, self._table.creation_date_time)) in fixtures

Can we mock self._table.creation_date_time???

[bblommers]

Hi @karthikvadla, where in your test are you calling the 'DescribeTable' operation, i.e. on which line is it failing? I can't see it in the code you provided

The creation time can be accessed like this:

table_description = conn.describe_table(TableName=name)
created_on = table_description["Table"]["CreationDateTime"]

[drewmullen]

Hi all im getting the same issue reported above. any advice would be greatly appreciated:

from moto import mock_s3
import boto3
import pytest
import os

@pytest.fixture(scope='function')
def aws_credentials():
    """Mocked AWS Credentials for moto."""
    os.environ['AWS_ACCESS_KEY_ID'] = 'testing'
    os.environ['AWS_SECRET_ACCESS_KEY'] = 'testing'
    os.environ['AWS_SECURITY_TOKEN'] = 'testing'
    os.environ['AWS_SESSION_TOKEN'] = 'testing'


@pytest.fixture(scope='function')
def s3(aws_credentials):
    with mock_s3():
        yield boto3.client('s3', region_name='us-east-1')


def test_create_bucket(s3):
    # s3 is a fixture defined above that yields a boto3 s3 client.
    # Feel free to instantiate another boto3 S3 client -- Keep note of the region though.
    s3.create_bucket(Bucket="somebucket")

    result = s3.list_buckets()
    assert len(result['Buckets']) == 1
    assert result['Buckets'][0]['Name'] == 'somebucket'

$ pytest moto.py

output:

______ERROR collecting moto.py _________________
ImportError while importing test module '/Users/dmullen/scratch/py-serverless/fixture-tests/moto.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
moto.py:1: in <module>
    from moto import mock_s3
E   ImportError: cannot import name 'mock_s3' from partially initialized module 'moto' (most likely due to a circular import) (/Users/dmullen/scratch/py-serverless/fixture-tests/moto.py)

versions:

• moto 1.3.14
• boto3 1.11.16
• botocore 1.14.16
• pytest 5.4.1

[msmolens]

______ERROR collecting moto.py _________________
ImportError while importing test module '/Users/dmullen/scratch/py-serverless/fixture-tests/moto.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
moto.py:1: in <module>
    from moto import mock_s3
E   ImportError: cannot import name 'mock_s3' from partially initialized module 'moto' (most likely due to a circular import) (/Users/dmullen/scratch/py-serverless/fixture-tests/moto.py)

@drewmullen The name of your test file conflicts with the moto package. Rename your file to something other than moto.py.

[bjmc]

Edit: I think I just need to be more careful with import order

Is this meant to be fixed? I'm seeing what looks to be a similar issue trying to mock out DynamoDB calls.

moto==1.3.14
boto==2.49.0
boto3==1.14.7
botocore==1.17.7

Partial example:

def setup_table():
    ddb_client = boto3.client('dynamodb')
    ddb_client.create_table(
        AttributeDefinitions=[
            {'AttributeName': 'email', 'AttributeType': 'S'},
            {'AttributeName': 'timestamp', 'AttributeType': 'S'},
        ],
        TableName='contact-form-submissions',
        KeySchema=[
            {'AttributeName': 'email', 'KeyType': 'HASH'},
            {'AttributeName': 'timestamp', 'KeyType': 'RANGE'},
        ],
        ProvisionedThroughput={'ReadCapacityUnits': 1, 'WriteCapacityUnits': 1},
    )

@mock_dynamodb2
def test_save_to_db():
    setup_table()
    result = save_to_db(DATA)
    assert result is True

I'm getting an error:

botocore.errorfactory.ResourceInUseException: An error occurred (ResourceInUseException) when calling the CreateTable operation: Table already exists: contact-form-submissions

And it looks like that table is getting created on real AWS, not mocked as I'd expected. Am I doing something wrong or is there a regression?

[mickog]

This is happening for me with moto 1.3.16 - botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the Publish operation: No account found for the given parameters

rolling back and using 1.13.10 seems the resolve the issue

[bblommers]

Hi @mickog, this might fix your issue:
https://github.com/spulec/moto/blob/master/README.md#very-important----recommended-usage

[mickog]

Cheers @bblommers will take a look at refactoring my tests,