moto and pynamodb don't play nicely together

[tom-dalton-fanduel]

As the title, I'm struggling to be able to use moto to mock out dynamodb[2] when using the pynamodb package. I've also posted this same bug report to pynamodb here: https://github.com/pynamodb/PynamoDB/issues/569

Steps to reproduce:

car.py:

from pynamodb.attributes import UnicodeAttribute
from pynamodb.models import Model
class CarModel(Model):
    class Meta:
        table_name = "cars"
    manufacturer = UnicodeAttribute(hash_key=True)
    serial_number = UnicodeAttribute(range_key=True)
    car_data_json = UnicodeAttribute()

test_car.py:

import moto
from car import CarModel
def test_save_load():
    with moto.mock_dynamodb2():
        m = CarModel(
            manufacturer="Cars Inc",
            serial_number="12345",
            car_data_json='{"wheels":4}',
        )
        m.save()
        m2 = m.load("Cars Inc", "12345")
        assert m2 == m

pip install pynamodb moto pytest

Gives me (amongst other things):

boto==2.49.0
boto3==1.9.58
botocore==1.12.58
mock==2.0.0
moto==1.3.7
pynamodb==3.3.1
pytest==4.0.1

I now hope the tests will run. Instead, we get an error that the table doesn't exist:

```~/tmp (moto-pynamo) › pytest
=============================================================================================================================== test session starts ===============================================================================================================================
platform linux -- Python 3.6.6, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /home/tom/tmp, inifile:
collected 1 item

test_car.py F [100%]

==================================================================================================================================== FAILURES =====================================================================================================================================
_________________________________________________________________________________________________________________________________ test_save_load __________________________________________________________________________________________________________________________________

self = Connectionhttps://dynamodb.us-east-1.amazonaws.com, table_name = 'cars', refresh = False

def get_meta_table(self, table_name, refresh=False):
    """
    Returns a MetaTable
    """
    if table_name not in self._tables or refresh:
        operation_kwargs = {
            TABLE_NAME: table_name
        }
        try:

          data = self.dispatch(DESCRIBE_TABLE, operation_kwargs)

../.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/connection/base.py:504:

---

self = Connectionhttps://dynamodb.us-east-1.amazonaws.com, operation_name = 'DescribeTable', operation_kwargs = {'TableName': 'cars'}

def dispatch(self, operation_name, operation_kwargs):
    """
    Dispatches `operation_name` with arguments `operation_kwargs`

    Raises TableDoesNotExist if the specified table does not exist
    """
    if operation_name not in [DESCRIBE_TABLE, LIST_TABLES, UPDATE_TABLE, DELETE_TABLE, CREATE_TABLE]:
        if RETURN_CONSUMED_CAPACITY not in operation_kwargs:
            operation_kwargs.update(self.get_consumed_capacity_map(TOTAL))
    self._log_debug(operation_name, operation_kwargs)

    table_name = operation_kwargs.get(TABLE_NAME)
    req_uuid = uuid.uuid4()

    self.send_pre_boto_callback(operation_name, req_uuid, table_name)

  data = self._make_api_call(operation_name, operation_kwargs)

../.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/connection/base.py:313:

---

self = Connectionhttps://dynamodb.us-east-1.amazonaws.com, operation_name = 'DescribeTable', operation_kwargs = {'TableName': 'cars'}

def _make_api_call(self, operation_name, operation_kwargs):
    """
    This private method is here for two reasons:
    1. It's faster to avoid using botocore's response parsing
    2. It provides a place to monkey patch requests for unit testing
    """
    operation_model = self.client._service_model.operation_model(operation_name)
    request_dict = self.client._convert_to_request_dict(
        operation_kwargs,
        operation_model
    )
    prepared_request = self._create_prepared_request(request_dict, operation_model)

    for i in range(0, self._max_retry_attempts_exception + 1):
        attempt_number = i + 1
        is_last_attempt_for_exceptions = i == self._max_retry_attempts_exception

        response = None
        try:
            proxies = getattr(self.client._endpoint, "proxies", None)
            # After the version 1.11.0 of botocore this field is no longer available here
            if proxies is None:
                proxies = self.client._endpoint.http_session._proxy_config._proxies

            response = self.requests_session.send(
                prepared_request,
                timeout=self._request_timeout_seconds,
                proxies=proxies,
            )
            data = response.json()
        except (requests.RequestException, ValueError) as e:
            if is_last_attempt_for_exceptions:
                log.debug('Reached the maximum number of retry attempts: %s', attempt_number)
                if response:
                    e.args += (str(response.content),)
                raise
            else:
                # No backoff for fast-fail exceptions that likely failed at the frontend
                log.debug(
                    'Retry needed for (%s) after attempt %s, retryable %s caught: %s',
                    operation_name,
                    attempt_number,
                    e.__class__.__name__,
                    e
                )
                continue

        if response.status_code >= 300:
            # Extract error code from __type
            code = data.get('__type', '')
            if '#' in code:
                code = code.rsplit('#', 1)[1]
            botocore_expected_format = {'Error': {'Message': data.get('message', ''), 'Code': code}}
            verbose_properties = {
                'request_id': response.headers.get('x-amzn-RequestId')
            }

            if 'RequestItems' in operation_kwargs:
                # Batch operations can hit multiple tables, report them comma separated
                verbose_properties['table_name'] = ','.join(operation_kwargs['RequestItems'])
            else:
                verbose_properties['table_name'] = operation_kwargs.get('TableName')

            try:

              raise VerboseClientError(botocore_expected_format, operation_name, verbose_properties)

E pynamodb.exceptions.VerboseClientError: An error occurred (ResourceNotFoundException) on request (GOV4H3FGT5P4ST5G7VENS5UK6VVV4KQNSO5AEMVJF66Q9ASUAAJG) on table (cars) when calling the DescribeTable operation: Requested resource not found: Table: cars not found

../.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/connection/base.py:399: VerboseClientError

During handling of the above exception, another exception occurred:

def test_save_load():
    with moto.mock_dynamodb2():
        m = CarModel(
            manufacturer="Cars Inc",
            serial_number="12345",
            car_data_json='{"wheels":4}',
        )

      m.save()

test_car.py:13:

---

../.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/models.py:457: in save
return self._get_connection().put_item(args, *kwargs)
../.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/connection/table.py:115: in put_item
return_item_collection_metrics=return_item_collection_metrics)
../.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/connection/base.py:959: in put_item
operation_kwargs.update(self.get_identifier_map(table_name, hash_key, range_key, key=ITEM))
../.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/connection/base.py:755: in get_identifier_map
tbl = self.get_meta_table(table_name)

---

self = Connectionhttps://dynamodb.us-east-1.amazonaws.com, table_name = 'cars', refresh = False

def get_meta_table(self, table_name, refresh=False):
    """
    Returns a MetaTable
    """
    if table_name not in self._tables or refresh:
        operation_kwargs = {
            TABLE_NAME: table_name
        }
        try:
            data = self.dispatch(DESCRIBE_TABLE, operation_kwargs)
            self._tables[table_name] = MetaTable(data.get(TABLE_KEY))
        except BotoCoreError as e:
            raise TableError("Unable to describe table: {0}".format(e), e)
        except ClientError as e:
            if 'ResourceNotFound' in e.response['Error']['Code']:

              raise TableDoesNotExist(e.response['Error']['Message'])

E pynamodb.exceptions.TableDoesNotExist: Table does not exist: Requested resource not found: Table: cars not found

../.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/connection/base.py:510: TableDoesNotExist
================================================================================================================================ warnings summary =================================================================================================================================
/home/tom/.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/attributes.py:632
/home/tom/.virtualenvs/moto-pynamo/lib/python3.6/site-packages/pynamodb/attributes.py:632: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
attribute_args = getargspec(Attribute.__init__).args[1:]

-- Docs: https://docs.pytest.org/en/latest/warnings.html
====================================================================================================================== 1 failed, 1 warnings in 1.58 seconds =======================================================================================================================
~/tmp (moto-pynamo) ›
```

If I try to create the table inside the moto context manager, I get an AcessDenied error from aws/boto, suggesting that it's not actually mocking the calls as expected.

Can anyone help here? AFAIK pynamodb uses boto3, so it's not clear why the mocks aren't working in this case. I'm sure i'm not the only one who would be super happy if these two excellent packages would play nicely with each other!

[dmulter]

This would probably be better on StackOverflow as moto and pynamodb already work nicely together for quite some time.

[bpandola]

They used to work nicely together, but they no longer do after the recent botocore changes. I think pynamodb bypasses boto3 and makes some API calls directly, which are no longer captured by moto after the recent changes (#1793).

[tom-dalton-fanduel]

@dmulter do you get different results from me with my small test example in the description? Or is my example doing something incorrectly?

[dmulter]

My code using moto and pynamodb works fine using moto 1.3.6. I would suggest you verify things work with that version first. Then if you have a small definitive example that shows the same thing breaks in moto 1.3.7, then open an issue here. Clearly you have to create the table first as described in the docs. SO is a better place to get the basics working as it's not clear to me what the issue is that you're reporting. I am not on this project, but felt it would help to point you in the likely right direction.

[tom-dalton-fanduel]

Thanks @dmulter.

I have given a small example (car.py/test_car.py) that fails for me (in the issue description). I'm not sure if it classes as "definitive", but it demonstrates the issue I have. I've included the failure traceback and the package versions I'm using (including moto 1.3.7). Perhaps I am using moto or pynamodb incorrectly? If you're able to see an obvious difference wrt. how I'm using it vs how you're using it, info would be appreciated.

Alternatively, if my example works for you (with moto 1.3.7 or any other version) then any info (especially what versions of python/boto/boto3/botocore/pytest you are using) would be much appreciated!

[tom-dalton-fanduel]

FWIW I've just tried the example with moto 1.3.6 and get the same error.

[ororog]

Hi, I created another example and it does not work well. the code is below.

from moto import mock_dynamodb2

from pynamodb.models import Model
from pynamodb.attributes import UnicodeAttribute


class UserModel(Model):
    """
    A DynamoDB User
    """
    class Meta:
        table_name = 'dynamodb-user'
    email = UnicodeAttribute(hash_key=True)
    first_name = UnicodeAttribute()
    last_name = UnicodeAttribute()


@mock_dynamodb2
def test():
    UserModel.create_table(
        read_capacity_units=1, write_capacity_units=1, wait=True)
    user = UserModel(
        '[email protected]', first_name='Samuel', last_name='Adams')
    user.save()
    print(UserModel.count())


test()

(the UserModel part is just copied from official pynamodb tutorial.)
I created venv and I just installed moto and pynamodb.
I run the script, it outputs 0 and dynamodb-user table is created on AWS.

After that, I installed moto 1.3.6 (pip install moto==1.3.6) and run again, it prints 1 and no table is created on AWS.
I dont see the changes from 1.3.6 but the change might affect the pynamo's behavior.
I will use moto 1.3.6 as a workaround.

You can see my whole code and requirements here https://github.com/ororog/moto-pynamo-test.

[kimvais]

This can be closed as it's a bug in PynamoDB as it bypasses boto3 and uses direct a direct HTTP request to create table in Dynamo

[garrettheel]

Good news! I plan to support moto as of the 4.0.0 release of pynamodb. You can test it out in an alpha version (pynamodb==4.0.0a1) today and open an issue on pynamodb if you have any problems.

[tom-dalton-fanduel]

Closing as it looks like this is now working due to changes in a new version of pynamodb :-)