Boto: Dynamodb save with float raises exception DynamoDBNumberError

[ushma]

with python 2.7 and boto 2.9.4

item = Item.get_item(key='key')
item['number'] = 26.34
item.save()

raises:

Traceback (most recent call last):
  File "boto.py", line 9, in <module>
    item.save()
  File "/home/user/boto/boto/dynamodb2/items.py", line 365, in save
    final_data = self.prepare_full()
  File "/home/user/boto/boto/dynamodb2/items.py", line 268, in prepare_full
    final_data[key] = self._dynamizer.encode(value)
  File "/home/user/boto/boto/dynamodb/types.py", line 234, in encode
    return {dynamodb_type: encoder(attr)}
  File "/home/user/boto/boto/dynamodb/types.py", line 250, in _encode_n
    raise DynamoDBNumberError(msg)
boto.dynamodb.exceptions.DynamoDBNumberError: BotoClientError: Inexact numeric for `26.34`

It appears that the internals are converting floats to Decimal datatype with 38 digits of precision.
https://github.com/boto/boto/blob/develop/boto/dynamodb/types.py#L243

However, aws docs suggest that numbers can have up to 38 digits after decimal points, but not to convert all the floats to that precision.
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataModel.html#DataModelDataTypes

[toastdriven]

Unfortunately, this is both somewhat of a known issue & not something I'm sure we can fix (at least I don't have the knowledge to fix it). https://github.com/boto/boto/blob/develop/tests/unit/dynamodb/test_types.py#L61-L68 has some explanation.

I modified the tests (https://github.com/boto/boto/blob/develop/tests/integration/dynamodb2/test_highlevel.py#L68-L73) & added in a 'score': 23.5,, which worked fine. But changing it to 'score': 26.34, was enough for it to fail.

I think the only guidance we can offer is that you should convert to Decimal before attempting to store numbers in DDB.

[depristo]

I believe I have a fix for this issue, even though it's now closed:

# Current Dynamizer is busted for floating point types, and can be trivially fixed
# by not trapping the inexact and round messages
#
# from dynamodb/types.py:
#  DYNAMODB_CONTEXT = Context(
#    Emin=-128, Emax=126, rounding=None, prec=38,
#    traps=[Clamped, Overflow, Inexact, Rounded, Underflow])
#
# We simply allow Inexact and Rounded to fix the issue
#
FIXED_DYNAMODB_CONTEXT = Context(
    Emin=-128, Emax=126, rounding=None, prec=38,
    traps=[Clamped, Overflow, Underflow])

This will allow your test to pass in the latest BOTO sources. The problem is that if you feed fixed floating point numbers like 2.2 at Decimal, it produces an equivalent of 2.20000000000000000xxxxxxx where the xs are meaningless garbage. It then throws an Inexact conversion because of the noise values in x aren't exactly represented in the Decimal encoding. But that's what you are expecting here (you will have inexact and rounding issues due to 38 digit restriction), so you shouldn't trap them. I'm sorry I cannot just issue a pull request but I figured the change was sufficiently simply to just have someone from the team implement.

[craiga]

For what it's worth, I found that numbers larger than ~1048575.4 save just fine. Numbers less than this encounter this error.

>>> t
<boto.dynamodb2.table.Table object at 0x1004a2450>
>>> t.put_item({'number': 1048575.5}, overwrite=True)
True
>>> t.put_item({'test', 'number': 1048575.4}, overwrite=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/virtual-environment/lib/python2.7/site-packages/boto/dynamodb2/table.py", line 821, in put_item
    return item.save(overwrite=overwrite)
  File "/virtualenv/lib/python2.7/site-packages/boto/dynamodb2/items.py", line 448, in save
    final_data = self.prepare_full()
  File "/virtualenv/lib/python2.7/site-packages/boto/dynamodb2/items.py", line 329, in prepare_full
    final_data[key] = self._dynamizer.encode(value)
  File "/virtualenv/lib/python2.7/site-packages/boto/dynamodb/types.py", line 279, in encode
    return {dynamodb_type: encoder(attr)}
  File "/virtualenv/lib/python2.7/site-packages/boto/dynamodb/types.py", line 295, in _encode_n
    raise DynamoDBNumberError(msg)
boto.dynamodb.exceptions.DynamoDBNumberError: BotoClientError: Inexact numeric for `1048575.4`
None

This explains why the return value of time.time() saves to DynamoDB (unless you're a time traveller to early January 1970), but 1.234 doesn't.

[lbenitez000]

This is the same solution @depristo suggests, but as a monkey patch. If you are comfortable with allowing inexact representation of floats, then try this out.

https://gist.github.com/lbenitez000/85fb43e40a7839e13405

# Monkey patch Decimal's default Context to allow 
# inexact and rounded representation of floats
import decimal
from boto.dynamodb.types import DYNAMODB_CONTEXT
# Inhibit Inexact Exceptions
DYNAMODB_CONTEXT.traps[decimal.Inexact] = 0
# Inhibit Rounded Exceptions
DYNAMODB_CONTEXT.traps[decimal.Rounded] = 0

[joaoportela]

@craiga 1836016.789451258 doesn't save correctly.