Mongoengine: AttributeError at calling QuerySet.count() on mongomock

[daitakahashi]

After updating to mongoengine 0.21.0, I got a following error when counting the number of documents by QuerySet.count().

Traceback (most recent call last):
  File "mongoengine_mongomock_count.py", line 15, in <module>
    main()
  File "mongoengine_mongomock_count.py", line 12, in main
    print(Doc.objects().count())
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/mongoengine/queryset/queryset.py", line 144, in count
    return super().count(with_limit_and_skip)
  File "/Users/dtakahashi/Library/Python/3.8/lib/python/site-packages/mongoengine/queryset/base.py", line 430, in count
    filter=self._cursor._Cursor__spec,
AttributeError: 'Cursor' object has no attribute '_Cursor__spec'

As the message says, mongomock seems not to have the attribute Cursor._Cursor__spec.

The mongomock is not an intended backend for this library, so I understand there are possibly many incompatibilities. But workarounds are (if any) quite helpful because the mongomock is useful for CI tests.
Thank you very much in advance.

The code for reproducing the error is here.

import mongoengine


class Doc(mongoengine.Document):
    name = mongoengine.StringField()

def main():
    mongoengine.connect(host='mongomock://localhost/test')
    for i in range(3):
        Doc(name=f'i{i}').save()
    print(Doc.objects().count())

if __name__ == '__main__':
    main()

[bagerard]

Argh, indeed in the changes related to migrating from Cursor.count to Collection.count_documents, I had to use some internals of the pymongo cursor object which are obviously not present on mongomock's cursors.

I'm not very fond of adding code at the core of mongoengine to accommodate for the use of mongomock but I'll check if there is a workaround

[daitakahashi]

I deeply appreciate you.

[daitakahashi]

For the meantime, on CI, I monkeypatched mongomock to redirect Cursor._Cursor__spec to Cursor._spec as a workaround. I agree not to add this kind of "fix" to the mainline.

[pcorpet]

Hi, I'm the maintainer of mongomock. I see how we could patch mongomock to work for you, but I don't think this is a good thing to use internals of pymongo like that. Do you foresee a way where you would stop using the internals of pymongo? or maybe ask pymongo library to make this a public API of cursors?

[agarrido19]

@daitakahashi Could you explain how do you monkeypatched mongomock?

[daitakahashi]

We use mongomock only with pytest, so I abused pytest.monckypatch.

[agarrido19]

@daitakahashi Sorry bother you again, I was implementing your idea but I could not patch the attribute

monkeypatch.setattr(mongomock.collection.Cursor, '_Cursor__spec', mongomock.collection.Cursor._spec, raising=False)

The above patch throws this error:

AttributeError: type object 'Cursor' has no attribute '_spec'

[agarrido19]

@daitakahashi Nevermind, I figured out when printing what the heck was in _spec

Thanks

[dimon222]

@agarrido19 do you mind sharing the final solution? It's not clear.

[daitakahashi]

@bagerard Do you know how are mongoengine.queryset.QuerySet._query and pymongo.cursor.Cursor.__spec different? Those dicts look quite similar in my naive experiments (only 2 simple cases, though).

[bagerard]

Good catch @daitakahashi. I must admit I didn't consider ._query too much and rushed into using Cursor.__spec since that one was guaranteed to be reliable. I just checked and the 2 are strictly equal when I'm running the whole MongoEngine test suite which is a good sign. I'll dig more into it with some edge cases to see if this remain valid

[bagerard]

FYI I just released mongoengine 0.22.0 with the fix for this
I switched from 0.21.0 to 0.22.0 instead of 0.21.1 because I introduced f-strings in the codebase which breaks Py3.5 support