Boto3: Add ability to disable Decimal usage for DynamoDB number type.
Hi,
Is there any strong reason why using a DynamoDB's Table resource will convert the number type "N" to a Decimal() object?
Shouldn't it try looking up the right python type, such as int or float or long?
I am trying to unpack a record's data (a mapping) into a function call, specifically: next_execution = now() + datetime.timedelta(**dynamo_record['frequency']) but datetime.timedelta will not accept the Decimal object into its arguments, although it does accept long and float.
TypeError: unsupported type for timedelta days component: Decimal
jonapich
All 69 comments
This was one of the lessons learned from boto. If I remember correctly, there were issues with round tripping float values and that the built in float() type could not handle the 38 digits of precision supported in dynamodb's numeric types. This would result in not being able to delete items in the table:
>>> d = Decimal('1234567890123.12345678901234567890')
>>> d
Decimal('1234567890123.12345678901234567890')
>>> float(d)
1234567890123.1235
More background info:
https://github.com/boto/boto/issues/873
PR:
https://github.com/boto/boto/pull/1183
Perhaps we could support using ints() if there's no floating point in the number. Would need to investigate what the impact of that would be. Would that help in your scenario?
jamesls
on 16 Nov 2015
After a trip to python's doc on floating point limitations, I can see where this came from. It's a shame that the standard library doesn't support Decimal() in place of a float.
To answer your question specifically, it will not help my scenario. The timedelta supports floats, it's perfectly valid to do a timedelta(hours=1.5). I think it would only add to the confusion if boto3 would change between int and Decimal on a per-record basis as its default behavior.
Perhaps a small option somewhere to tell boto we don't really care about the added precision, so it can return floats and ints across the board?
jonapich
on 16 Nov 2015
FWIW, I use this little function to recurse into Python objects returned by the boto3 DynamoDB resource layer and convert any Decimal values to int or float. It is by no means foolproof and doesn't in any way solve the problem of lack of precision in Python's float type but it solves my problem which is mainly to turn the data into something that can be returned to API Gateway via a Python Lambda function.
def replace_decimals(obj):
if isinstance(obj, list):
for i in xrange(len(obj)):
obj[i] = replace_decimals(obj[i])
return obj
elif isinstance(obj, dict):
for k in obj.iterkeys():
obj[k] = replace_decimals(obj[k])
return obj
elif isinstance(obj, decimal.Decimal):
if obj % 1 == 0:
return int(obj)
else:
return float(obj)
else:
return obj
garnaat
on 17 Nov 2015
Thanks for the code @garnaat, that certainly works.
This issue is a big PITA.
astewart-twist
on 18 Nov 2015
Would adding some sort of use_decimal=False option to the config object when creating clients/resources be helpful?
jamesls
on 18 Nov 2015
Of course that would be useful :)
jonapich
on 18 Nov 2015
Ok, let's mark this as a feature request. I'll update the title.
jamesls
on 19 Nov 2015
:+1: for this feature
niallrobinson
on 10 Feb 2016
This particular problem likes to creep into my code in the most unusual places. Converting from Decimal to int/float is a thing, but it seems boto won't take my python floats anymore (did it take them before? i'm not sure) so I created the following function to prepare all of my data before sending it to dynamodb (kinda like @garnaat's method, but the other way around):
def _sanitize(data):
""" Sanitizes an object so it can be updated to dynamodb (recursive) """
if not data and isinstance(data, (basestring, Set)):
new_data = None # empty strings/sets are forbidden by dynamodb
elif isinstance(data, (basestring, bool)):
new_data = data # important to handle these one before sequence and int!
elif isinstance(data, Mapping):
new_data = {key: _sanitize(data[key]) for key in data}
elif isinstance(data, Sequence):
new_data = [_sanitize(item) for item in data]
elif isinstance(data, Set):
new_data = {_sanitize(item) for item in data}
elif isinstance(data, (float, int, long, complex)):
new_data = Decimal(data)
else:
new_data = data
return new_data
+1 for this
mr337
on 9 Apr 2016
+1 for this feature request, I've now run into a similar problem to the one @jonapich experienced.
jonathanwcrane
on 24 May 2016
665
jonapich
on 31 May 2016
+1 for this feature request
sridharrajagopal
on 1 Jun 2016
+1
ustroetz
on 15 Jun 2016
+1
josepvalls
on 2 Aug 2016
any update on this ?
ghost
on 4 Oct 2016
+1
jong-eatsa
on 11 Nov 2016
+1
BorsosWyatt
on 23 Nov 2016
+1
johnjjung
on 14 Dec 2016
+1
blieber
on 23 Jan 2017
+1
itssiva
on 8 Feb 2017
+1
pmranade
on 10 Feb 2017
+1
mwada
on 13 Feb 2017
+1
cdmbr
on 13 Feb 2017
Seriously, this is not okay. Using python3.6 I can store math.floor(time.time()) in DynamoDB. Using python2.7 I cannot. A database is expected to be able to receive numbers. That's pretty basic.
RichardBronosky
on 1 Mar 2017
+1
Manelmc
on 4 May 2017
Just used the code by @garnaat and updated it to Python 3.6:
def replace_decimals(obj):
if isinstance(obj, list):
for i in range(len(obj)):
obj[i] = replace_decimals(obj[i])
return obj
elif isinstance(obj, dict):
for k, v in obj.items():
obj[k] = replace_decimals(v)
return obj
elif isinstance(obj, decimal.Decimal):
if obj % 1 == 0:
return int(obj)
else:
return float(obj)
else:
return obj
flomotlik
on 17 May 2017
+1
sillmnvg
on 21 Aug 2017
+1
lkuffo
on 13 Sep 2017
The code below is for saving to DynamoDB; use @flomotlik's code in https://github.com/boto/boto3/issues/369#issuecomment-302137290 to load floats from DynamoDB.
To allow rounding and inexact values and still prevent over/underflow and clamping, I'd recommend using a decimal.Context such as the one in boto3/dynamodb/types.py but drop the decimal.Inexact and decimal.Clamped traps. I'd also use numeric for the type check in the sanitizer instead of just checking for Decimal or float. The following serializer should be a bit more robust:
from collections.abc import Iterable, Mapping, ByteString, Set
import numbers
import decimal
context = decimal.Context(
Emin=-128, Emax=126, rounding=None, prec=38,
traps=[decimal.Clamped, decimal.Overflow, decimal.Underflow]
)
def dump_to_dynamodb(item):
# don't catch str/bytes with Iterable check below;
# don't catch bool with numbers.Number
if isinstance(item, (str, ByteString, bool)):
return item
# ignore inexact, rounding errors
if isinstance(item, numbers.Number):
return context.create_decimal(item)
# mappings are also Iterable
elif isinstance(item, Mapping):
return {
key: dump_to_dynamodb(value)
for key, value in item.values()
}
# boto3's dynamodb.TypeSerializer checks isinstance(o, Set)
# so we can't handle this as a list
elif isinstance(item, Set):
return set(map(dump_to_dynamodb, item))
# may not be a literal instance of list
elif isinstance(item, Iterable):
return list(map(dump_to_dynamodb, item))
# datetime, custom object, None
return item
\Float type which links to this issue. \
numberoverzero
on 18 Sep 2017
+1
kashoory
on 21 Sep 2017
Thanks for all above post and code you guys provided and my problem is solved. But I still want to ask why don't we have a setting parameter to disable / enable decimal, use float / int instead?
wangpin34
on 26 Sep 2017
+1
gibbymt
on 16 Nov 2017
+1
valeriyaxref
on 29 Nov 2017
+1
gschlabitz
on 13 Mar 2018
+1
sb89
on 16 Mar 2018
+1
maayanlahav
on 21 Mar 2018
+1
smthfrmn
on 23 Mar 2018
+1
shilpad
on 5 Apr 2018
+1
c--m
on 9 Apr 2018
+1
tomhillable
on 30 Apr 2018
+1
gibbymt
on 1 May 2018
+1
lerebel103
on 3 May 2018
+1
weiwongfaye
on 1 Jun 2018
+1
nikhilym
on 15 Jun 2018
+1
rogergzou
on 12 Jul 2018
+1
nvsit
on 20 Jul 2018
@jamesls Any updates on this issue?
aalvrz
on 23 Jul 2018
+1
linds14sr20det
on 4 Sep 2018
+1
himanshu219
on 21 Nov 2018
any updates ?
himanshu219
on 21 Nov 2018
I adapted @flomotlik 's code to support NumberSets (dicts) when reading DynamoDB items :
# Helper class to Decimals in an arbitrary object
def replace_decimals(obj):
if isinstance(obj, list):
for i in range(len(obj)):
obj[i] = replace_decimals(obj[i])
return obj
elif isinstance(obj, dict):
for k, v in obj.items():
obj[k] = replace_decimals(v)
return obj
elif isinstance(obj, set):
return set(replace_decimals(i) for i in obj)
elif isinstance(obj, decimal.Decimal):
if obj % 1 == 0:
return int(obj)
else:
return float(obj)
else:
return obj
gileri
on 19 Dec 2018
+1 on this feature request too
oitan
on 19 Feb 2019
Why is this still open in 2019?
+1
miroslawmajka
on 27 Feb 2019
+1
caiounderscore
on 15 May 2019
If the main issue you guys are having is JSON serializing the Decimal values, simplejson can handle that. That's what I use in my Lambda functions.
nft2
on 31 May 2019
If the main issue you guys are having is JSON serializing the Decimal values, simplejson can handle that. That's what I use in my Lambda functions.
Thanks, nft2! The simplejson package works great.
I use:
import simplejson as json
Instead of:
import json
rbk
on 14 Sep 2019
Hello @jamesls , @garnaat.
It's been 4 years+ since the opening of issue. What is the current take of the boto3 team on this? Are there plans to add support of native python int/float for DynamoDB?
vfilimonov
on 12 Jan 2020
+1 what a pain... sigh
matt-theguyw1cat
on 18 Apr 2020
+1 one more here
Ati59
on 28 Apr 2020
+1 one more here
Tomos-Evans
on 28 Apr 2020
Why is this still open in 2019?
2020*
riless
on 30 Apr 2020
TBH, this Decimal() fiasco has cost me hours (with the net result that I now just use strings for everything and have the app figure it out). I doubt I'll be using dynamodb for any future projects.
Its "type system" is too stupid to live.
cwells
on 17 May 2020
If the main issue you guys are having is JSON serializing the Decimal values, simplejson can handle that. That's what I use in my Lambda functions.
Thanks, nft2! The simplejson package works great.
I use:
import simplejson as jsonInstead of:
import json
man ! you saved lot of time. Thanks
MayurBMahajan
on 21 May 2020
2021
maradwan
on 20 Jun 2020
hopefulfor2022
syedautherabbas
on 10 Jul 2020
+1
Closing in on 5 years since open and the problem is still around :/
egalvan10
on 11 Oct 2020
Massive +1
This is really a user experience nightmare. Seems like hundreds of users around the web are spending time writing functions (or troubleshooting issues with the json approach, such as NaN handling). Really should be handled inside boto itself.
aerogt3
on 9 Nov 2020
Here's an updated version of @garnaat's function for Python 3. In my case, I want them as an int, not a float:
def replace_decimals(obj):
"""
Convert all whole number decimals in `obj` to integers
"""
if isinstance(obj, list):
return [replace_decimals(i) for i in obj]
elif isinstance(obj, dict):
return {k: replace_decimals(v) for k, v in obj.items()}
elif isinstance(obj, decimal.Decimal):
return int(obj) if obj % 1 == 0 else obj
return obj
ipmb
on 21 Nov 2020
Related issues
yannbriancon
路
3Comments
luistm
路
3Comments
nueverest
路
3Comments
hwine
路
3Comments
teran-mckinney
路
3Comments
Most helpful comment
FWIW, I use this little function to recurse into Python objects returned by the boto3 DynamoDB resource layer and convert any Decimal values to
intorfloat. It is by no means foolproof and doesn't in any way solve the problem of lack of precision in Python's float type but it solves my problem which is mainly to turn the data into something that can be returned to API Gateway via a Python Lambda function.