Mongoengine: Is there a "bulk update"?

[alexandre]

Hello,

Sometime ago I was looking for a way to insert many document per time and I found this:

pre_documents = [models.Document(attr_1=foo, attr_2=bar, attr_3=bleh), <another documents>]
models.Document.object.insert(pre_documents)

Cool, but now: is there a way to do it, but for updates? Well, I thought that update + upsert could do it, however I was wrong:

# My thought: the var foo is a list, so the ORM should create one document per element in this list
foo = ['red', 'blue', 'yellow', 'black']

result = models.Document.object(color__in=foo).update(
    set__bar='new colors',
    upsert=True,
    full_result=True)

# it (result) doesn't works, only one document was "upserted". =( 

I hope I have been clear, thanks for the effort and help. =]

[ ]'s (hugs)

[thedrow]

Can you provide more details or a failing test case (preferably in a PR)?
The fact that the rest of the objects were not updated sounds like a bug to me.

[MRigal]

We should anyway handle this better with PyMongo 3.0 and the update_many() function

[alexandre]

I'm using python3 and I'm still trying to run the tests (something with my local mongodb instance). But, I'll try send this PR [yet] today.

[ ]'s (hugs)

[alexandre]

Hey guys,

The version 3.0 of pymongo have an issue and I just discovered it today. =(
Well, I wrote a simple test ( #956 ) using something similar I've commented here initially. Please apologize and let me know if my test have some error or not is clear enough, I would be really happy to write it again.

[ ]'s (hugs)

[MRigal]

Hi @alexandre
It's not really clear from what you're writing. Have you ran your test previously with the PyMongo versions?
You mention at the beginning that you're using Python 3. Could you be more precise with the versions of Python, PyMongo and mongoengine you were using before and now?

Mongoengine is currently NOT supporting PyMongo3, but we are pretty close to it #946

Anyway, your tests make fully sense, you could really open a Pull Request, even if there are just tests for now. With Travis-CI here, we could easily see on which versions we have problems. For this I recommend to put your changes in a branch on your fork of mongoengine. Like this, any future commit on the branch would also automatically add to your PR :+1:

[alexandre]

@MRigal ,

I'm sorry, let me add some info:

• I discovered the issue in PyMongo3, so I did the downgrade to the version 2.7.2.
• Yep, I'm using Python3, the version 3.4.2.
• I'm using the last version of mongoengine available on PyPi - the version 0.9.0.

I did put my test on the master branch, but I'll open a specific branch to handle this "bulk update". How about? better? =]

[ ]'s (hugs)

[MRigal]

OK, thanks for your PR #956

It is now quite clear! I'm currently busy with some other issues, like PyMongo 3.X, but yourself or somebody else could work on a fix. =]

[MRigal]

I spent some time checking this here. mongoengine is doing all fine here. The problem is inside PyMongo 2.X.
The update method of pymongo/collection.py gets to following point:
It constructs docs as
SON([('q', {'title': {'$in': ['tittle1', 'tittle2', 'tittle3']}}), ('u', {'$set': {'tags': ['bulk update']}}), ('multi', True), ('upsert', True)])]
But then results coming from message._do_batched_write_command only gives:
[(0, {u'nModified': 0, u'ok': 1, u'upserted': [{u'index': 0, u'_id': ObjectId('554120d203245eabc80a2b12')}], u'n': 1})]

@rozza Do you want to test/report that for PyMongo 2.X versions? Else I would close the ticket as wontfix

As I said previously, it is clearly something PyMongo 3.X handles much better and something we can make work in the future.

[alexandre]

@MRigal ,

Thank man for the SitRep. =P

...waiting by a stable version of PyMongo 3.X to use all power with pypy...no C deps...

[ ]'s (hugs)

[hejiaji]

There is no way of MongoEngine with flask can do it?

[badjano]

this is 3 years old and no update? crucial feature

[lebenitza]

4 years!

[MacHu-GWU]

@alexandre I made a library called mongoengine_mate that support batch insert and batch update. It uses _id field to locate the object to update and smartly decide which one should be updated which one should be inserted. And it uses a strategy to bulk upsert those document to insert. allow you toggle on and off upsert.

Check this example, I use this for many years https://github.com/MacHu-GWU/mongoengine_mate-project#smart-update---batch-upsert

>>> class User(mongoengine_mate.ExtendedDocument):
....    _id = mongoengine.IntField()
....    name = mongoengine.StringField()
>>> User(id=2, name="Bob").save()

# update only
>>> User.smart_update([
    User(id=1, name="Alice"),
    User(id=2, name="Bruce"),
    User(id=3, name="Cathy"),
], upsert=False)
>>> User.objects.count()
1

# upsert
>>> User.smart_update([
    User(id=1, name="Alice"),
    User(id=2, name="Bruce"),
    User(id=3, name="Cathy"),
], upsert=True)
>>> User.objects.count()
3

[alexandre]

Hi, Sanhe,

I appreciate your support, but I stopped using mongoengine or any orm with MongoDB some years ago.

Best regards.

On Fri, Dec 27, 2019, at 14:52, Sanhe wrote:

@alexandre https://github.com/alexandre I made a library called mongoengine_mate that support batch insert and batch update. It uses _id field to locate the object to update and smartly decide which one should be updated which one should be inserted. And it uses a strategy to bulk upsert those document to insert. allow you toggle on and off upsert.

Check this example, I use this for many years https://github.com/MacHu-GWU/mongoengine_mate-project#smart-update---batch-upsert

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/MongoEngine/mongoengine/issues/933?email_source=notifications&email_token=ABICNOZHZW7BPOOYJR5PVZ3Q2Y6EXA5CNFSM4A7L3NNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHXRD7A#issuecomment-569315836, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABICNO7Z7OSKAUXCSITACPLQ2Y6EXANCNFSM4A7L3NNA.