Mongoengine: Dict field can't handle python's Decimal

[ivan-kleshnin]

Attempt to save decimal.Decimal object as one of the values in DictField raises bson.errors.InvalidDocument: Cannot encode object: Decimal('20000')

[thedrow]

I'm not sure if bson can serialize Decimal objects.
We definitely need to solve this.
Care to provide a failing test case?

[lig]

Python Decimal should be saved via MongoEngine DecimalField. By the way DecimalField it self looks unusable see #637.

[thedrow]

@lig Is this a duplicate then?

[ivan-kleshnin]

It's not a duplicate. DictField should handle python Decimal's somehow as they are native Python objects. Arguing about DecimalField is an offtopic remark. I'm no longer interested in MongoEngine and Mongo (and Python)... just my final note here.

[lig]

@thedrow First of all it is very hard to understand DictField behavior basing on documentation. Too few words there: https://mongoengine-odm.readthedocs.org/en/latest/apireference.html#mongoengine.fields.DictField

In favor of 0.9 I would say that we should plan for next release:

1. Enhance DictField documentation in sense of "handle complex / varying types of data". What that types are, how are they handled.
2. Ensure DictField is able to handle any type that could be handled by scalar fields automatically.
3. Ensure that other free form fields like ListField, DynamicDocument, DynamicField, etc are able to do the same.

As for now I think we should record this as known issue as it is very specific case that could be easily worked around.

[ivan-kleshnin]

It can't be easily worked around because DictField with any nested field and DocumentField with nested DecimalField bring different semanthics and different limitations considering other existing issues with default values and so on which I wouldn't like to recall. Just remember it was a mess to workaround.

[lig]

Well you are always able to check deep into dict values before saving and
convert buggy values to something you will be able to restore them from on
load.

It could be done using custom Field around DictField. This is just
straight forward coding that is not hard enough to support by hands.

On Wed, Nov 26, 2014 at 2:39 PM, Ivan Kleshnin [email protected]
wrote:

It can't be easily worked around because DictField with any nested field
and DocumentField with nested DecimalField bring different semanthics and
different limitations considering other existing issues with default values
and so on which I wouldn't like to recall. Just remember it was a mess to
workaround.

—
Reply to this email directly or view it on GitHub
https://github.com/MongoEngine/mongoengine/issues/707#issuecomment-64582140
.

Serge Matveenko
mailto: [email protected]
github: http://lnkfy.com/1
linkedin: http://lnkfy.com/S

[DavidBord]

A test that reproduces it:

    class Doc(Document):
        keyval = DictField()
    Doc(keyval={"dec": Decimal(20000)}).save()

[DavidBord]

@thedrow, are you sure we want to solve this? indeed, bson shouldn't serialize Decimal objects so why should we coerce it?

[thedrow]

Because as a framework we should be able to deal with Python data types that are being saved to mongo.

[DavidBord]

Lets say we'll represent stuff like Decimal(3.14) as strings. What would we do with the aggregation framework?

[thedrow]

That's a very good question.
There are some suggestions on StackOverflow on how to represent decimals in MongoDB.

[thedrow]

Here's how eBay does it and you can't say they are not using decimals and aggregations.

[wojcikstefan]

Starting with MongoDB v3.4, you can use the Decimal BSON Type.

[carlos-jenkins]

For anyone looking for a quick answer:

from bson.decimal128 import Decimal128

for key in ['decimal_key1', 'decimal_key2']:
    row[key] = Decimal128(row[key])

You need to convert the Decimal to BSON's Decimal128 before inserting to the database. More info:

https://api.mongodb.com/python/current/api/bson/decimal128.html#module-bson.decimal128

You also require MongoDB 3.4+.