Moto: DynamoDB update_item using ReturnValues='UPDATED_NEW' fails with "Value must be a nonempty dictionary whose key is a valid dynamodb type"

[pstDnb]

The following code runs fine towards DynamoDB, but fails with moto:

import time

import boto3
import moto


@moto.mock_dynamodb2()
def test_update_does_not_work():
    dynamo_client = boto3.resource("dynamodb", region_name="eu-west-1")
    dynamo_client.create_table(TableName="ZzTable3",
                               KeySchema=[{'AttributeName': 'HashKey1', 'KeyType': 'HASH'}],
                               AttributeDefinitions=[{'AttributeName': 'HashKey1', 'AttributeType': 'S'}],
                               ProvisionedThroughput={'ReadCapacityUnits': 5, 'WriteCapacityUnits': 5})

    time.sleep(5)

    table_data = [{
        "HashKey1": "HashKeyValue1",
        "listValuedAttribute1": [
            "a",
            "b"
        ]}]

    dynamodb_table = dynamo_client.Table("ZzTable3")
    for record in table_data:
        dynamodb_table.put_item(Item=record)

    table_data[0]["listValuedAttribute1"][1] = "c"

    dynamodb_table.update_item(
        Key={"HashKey1": "HashKeyValue1"},
        UpdateExpression=f"SET listValuedAttribute1=:a",
        ExpressionAttributeValues={':a': table_data[0]["listValuedAttribute1"]},
        ReturnValues='UPDATED_NEW')

The following error is shown in the output:

self = <boto3.dynamodb.types.TypeDeserializer object at 0x000002D3CD64DB70>
value = None

    def deserialize(self, value):
        """The method to deserialize the DynamoDB data types.

        :param value: A DynamoDB value to be deserialized to a pythonic value.
            Here are the various conversions:

            DynamoDB                                Python
            --------                                ------
            {'NULL': True}                          None
            {'BOOL': True/False}                    True/False
            {'N': str(value)}                       Decimal(str(value))
            {'S': string}                           string
            {'B': bytes}                            Binary(bytes)
            {'NS': [str(value)]}                    set([Decimal(str(value))])
            {'SS': [string]}                        set([string])
            {'BS': [bytes]}                         set([bytes])
            {'L': list}                             list
            {'M': dict}                             dict

        :returns: The pythonic value of the DynamoDB type.
        """

        if not value:
>           raise TypeError('Value must be a nonempty dictionary whose key '
                            'is a valid dynamodb type.')
E           TypeError: Value must be a nonempty dictionary whose key is a valid dynamodb type.

Debugging the issue showed that there might be a bug in moto/dynamodb2/responses.py, possibly in the method _build_updated_new_attributes(). This causes None elements to be included in the list at indexes where the updated value equals the original value.

We've currently worked around the issue by omitting ReturnValues='UPDATED_NEW', since we didn't need the returned values after all.

[bblommers]

Thanks for raising this @pstDnb! Will mark it as a bug.

You're right, looks like the _build_updated_new_attributes() has a bug. As far as I can tell, that method should always return the changed value, even if they are the same.

[pstDnb]

I think you're right @bblommers. When I run the same test towards DynamoDB both list elements are returned, including the unmodified one.

[oidebrett]

Hi @pstDnb and @bblommers
I think I fixed this bug. I am very new to this project and relatively new to contributing to github projects. I have read the contribution guide and I am very keen to learn. I picked this bug as it seemed fairly straightforward. I created a Pull request and I will gladly take any feedback to help me improve and contribute to this great project. I hope this helps!

[bblommers]

This is now fixed as of moto >= 1.3.16.dev125