Mongoengine: How to fetch main document with matched embedded document instead all embedded documents
I'm fetching documents on the base of specific EmbeddedDocument, but I don't want to get all of the EmbeddedDocuments while retrieving, only the matched EmbeddedDocument with main Document.
Here's my code:
School Embedded Document
class School(EmbeddedDocument):
name = StringField(max_length=120)
User Document
class User(Document):
first_name = StringField(max_length=60, required=True)
last_name = StringField(max_length=60)
schools = EmbeddedDocumentListField(School)
Feeding documents
user = User.objects.create(first_name="Rohit", last_name="Khatri")
user.schools = [
School(name="Auden High School"),
School(name="Baldwin Boys High School"),
School(name="Baldwin Girls High School"),
School(name="Aukamm Elementary School"),
School(name="Mason-Rice Elementary")
]
user.save()
user = User.objects.create(first_name="ABC", last_name="DEF")
user.schools = [
School(name="Little Harbor Elementary School"),
School(name="Aukamm Elementary School"),
School(name="Mason-Rice Elementary")
]
user.save()
I'm using this code the retrieve the users who have a particular school in their schools field:
users = User.objects(schools__match={"name": "Aukamm Elementary School"})
I want to get only the selected school which is Aukamm Elementary School in the schools field.
Receiving
[
{
"first_name": "Rohit",
"last_name": "Khatri",
"schools": [
{
"name": "Auden High School"
},
{
"name": "Baldwin Boys High School"
},
{
"name": "Baldwin Girls High School"
},
{
"name": "Aukamm Elementary School"
},
{
"name": "Mason-Rice Elementary"
}
]
},
{
"first_name": "ABC",
"last_name": "DEF",
"schools": [
{
"name": "Little Harbor Elementary School"
},
{
"name": "Aukamm Elementary School"
},
{
"name": "Mason-Rice Elementary"
}
]
}
]
Required Output
[
{
"first_name": "Rohit",
"last_name": "Khatri",
"schools": [
{
"name": "Aukamm Elementary School"
}
]
},
{
"first_name": "ABC",
"last_name": "DEF",
"schools": [
{
"name": "Aukamm Elementary School"
}
]
}
]
Thanks
rohitkhatri
All 18 comments
Are you wanting some example code showing how to do this? Or, are you wanting a change to MongoEngine what would make such a thing automatic? You might want to say why you are wanting to do this; that will help out with the solution.
In general, since the 'schools' list will always contain what you are searching for, returning the single matched entry of that list does not really gain you anything. I'd recommend simply excluding the entry:
users = User.objects(schools__name="Aukamm Elementary School").exclude("schools")
which gave me:
[
{
"first_name": "Rohit",
"last_name": "Khatri",
"_id": {"$oid": "585ed98e85f4825594bc5585"}
},
{
"first_name": "ABC",
"last_name": "DEF",
"_id": {"$oid": "585ed98e85f4825594bc5586"}
}
]
Does this work for your purpose?
JohnAD
on 24 Dec 2016
@JohnAD I want to use the fields of matched School model, and I don't want to fetch all of the School models, because there will be a lot of schools added for a single user. and then I will have to find the matched school from the list of school models.
rohitkhatri
on 26 Dec 2016
@rohitkhatri unfortunately right now this isn't possible via MongoEngine, or at least it's not as pretty as you'd want it to be. Basically it boils down to the fact that MongoEngine doesn't support most of the projection operators at the moment and you need one of those (https://docs.mongodb.com/manual/reference/operator/projection/elemMatch/, to be specific).
We'll track support for remaining projection operators in #529, but here's a workaround for now:
In [14]: query = User.objects(schools__match={"name": "Aukamm Elementary School"})._query
In [15]: query
Out[15]: {'schools': {'$elemMatch': {'name': 'Aukamm Elementary School'}}}
In [16]: users = User._collection.find(query, query) # this works because the projection doc is the same as the query doc in this case
In [17]: users = [User._from_son(u) for u in users] # de-serialize pymongo docs
In [19]: users
Out[19]: [<User: User object>, <User: User object>]
In [20]: users[0].schools # See? Just one school retrieved!
Out[20]: [<School: School object>]
In [21]: users[1].schools # Same here!
Out[21]: [<School: School object>]
In [23]: users[1].schools[0].name
Out[23]: u'Aukamm Elementary School'
wojcikstefan
on 27 Dec 2016
Actually, there's an even easier, less hacky way to do it:
In [15]: name = 'Aukamm Elementary School'
In [16]: users = User.objects.filter(schools__name=name).fields(schools={'$elemMatch': {'name': name}})
In [17]: users
Out[17]: [<User: User object>, <User: User object>]
In [18]: users[0].schools
Out[18]: [<School: School object>]
In [19]: users[0].schools[0].name
Out[19]: u'Aukamm Elementary School'
In [20]: users[1].schools[0].name
Out[20]: u'Aukamm Elementary School'
In [21]: users[1].schools
Out[21]: [<School: School object>]
@wojcikstefan Thank you so much, this is what I was looking for, It solved my problem.
rohitkhatri
on 27 Dec 2016
@wojcikstefan I'm sorry to say, but It doesn't contain other fields now, can you tell me how to define other fields too, like
users = User.objects.filter(schools__name=name).fields('first_name', 'last_name', schools={'$elemMatch': {'name': name}})
rohitkhatri
on 27 Dec 2016
@rohitkhatri I improved the docstring for BaseQuerySet.fields, which hopefully makes it clearer. You have to use keyword arguments, not positional args. Try:
users = User.objects.filter(schools__name=name).fields(first_name=1, last_name=1, schools={'$elemMatch': {'name': name}})
@wojcikstefan When I tried the above example, I got this error
TypeError: unorderable types: int() < dict()
rohitkhatri
on 28 Dec 2016
@rohitkhatri are you on the latest mongoengine? It works for me:
In [3]: class School(EmbeddedDocument):
...: name = StringField(max_length=120)
...:
In [4]: class User(Document):
...: first_name = StringField(max_length=60, required=True)
...: last_name = StringField(max_length=60)
...: schools = EmbeddedDocumentListField(School)
...:
In [5]: User.drop_collection()
In [6]: user = User.objects.create(first_name="Rohit", last_name="Khatri")
...: user.schools = [
...: School(name="Auden High School"),
...: School(name="Baldwin Boys High School"),
...: School(name="Baldwin Girls High School"),
...: School(name="Aukamm Elementary School"),
...: School(name="Mason-Rice Elementary")
...: ]
...: user.save()
...:
...: user = User.objects.create(first_name="ABC", last_name="DEF")
...: user.schools = [
...: School(name="Little Harbor Elementary School"),
...: School(name="Aukamm Elementary School"),
...: School(name="Mason-Rice Elementary")
...: ]
...: user.save()
...:
Out[6]: <User: User object>
In [8]: name = 'Aukamm Elementary School'
In [9]: users = User.objects.filter(schools__name=name).fields(first_name=1, last_name=1, schools={'$elemMatch': {'name': name}})
...:
In [10]: users
Out[10]: [<User: User object>, <User: User object>]
In [11]: users[0].schools
Out[11]: [<School: School object>]
@wojcikstefan I'm doing the same and still getting the error TypeError: unorderable types: int() < dict()
Here's what versions I'm using:
Django==1.10.2
django-rest-framework-mongoengine==3.3.1
djangorestframework==3.4.7
mongoengine==0.11.0
pymongo==3.4.0
rohitkhatri
on 28 Dec 2016
Here's full error log:
Traceback (most recent call last):
File "/Volumes/Stuff/Projects/unengage/example.py", line 42, in <module>
users = User.objects.filter(schools__name=name).fields(first_name=1, last_name=1, schools={'$elemMatch': {'name': name}})
File "/Volumes/Stuff/Projects/unengage/lib/python3.5/site-packages/mongoengine/queryset/base.py", line 930, in fields
fields = sorted(cleaned_fields, key=operator.itemgetter(1))
TypeError: unorderable types: int() < dict()
And what's the exact document schema?
wojcikstefan
on 28 Dec 2016
Here's the exact schema:
import os
import django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "unengage.settings")
django.setup()
from mongoengine import *
class School(EmbeddedDocument):
name = StringField(max_length=120)
class User(Document):
first_name = StringField(max_length=60, required=True)
last_name = StringField(max_length=60)
schools = EmbeddedDocumentListField(School)
User.drop_collection()
user = User.objects.create(first_name="Rohit", last_name="Khatri")
user.schools = [
School(name="Auden High School"),
School(name="Baldwin Boys High School"),
School(name="Baldwin Girls High School"),
School(name="Aukamm Elementary School"),
School(name="Mason-Rice Elementary")
]
user.save()
user = User.objects.create(first_name="ABC", last_name="DEF")
user.schools = [
School(name="Little Harbor Elementary School"),
School(name="Aukamm Elementary School"),
School(name="Mason-Rice Elementary")
]
user.save()
name = 'Aukamm Elementary School'
users = users = User.objects.filter(schools__name=name).fields(
first_name=1, last_name=1, schools={'$elemMatch': {'name': name}}
)
print(users.to_json(indent=4))
Thanks @rohitkhatri! https://github.com/MongoEngine/mongoengine/pull/1452 should fix it. That being said, I think there's a separate bug where to_json doesn't include schools in the payload. If you change print(users.to_json(indent=4)) to print([user.schools for user in users]), it will work properly.
wojcikstefan
on 28 Dec 2016
@wojcikstefan I get this error even If I remove the print(users.to_json(indent=4)) statement.
rohitkhatri
on 29 Dec 2016
Hey there,
I stumbled across this earlier today, and I'm having issues implementing the solution given in https://github.com/MongoEngine/mongoengine/issues/1442#issuecomment-269256499.
So, I have the following data-set.
{
"data": [
{
"adducts": {
"negative": [
{
"accurate_mass": 1093.78043,
"type": "[M+Br]1-"
},
{
"accurate_mass": 1049.83094,
"type": "[M+Cl]1-"
}
]
},
"id" : "2131321312"
}
],
"has_more": false
}
I have the following mongodb query that returns [M+Cl]1- from the above example.
db.getCollection('metabolites').find({
"adducts.negative" : {
$elemMatch : {
"accurate_mass" : {$lte : 1050, $gte: 149}
}}}, {"adducts.negative.$" : 1})
However, when I attempt to implement it in my code - using your solution:
class AdductPpm(Operator):
op = "ppm"
def calculate_ppm(self, mz, ppm):
tolerance = abs(float(mz) * (float(ppm) * 0.0001))
return float(mz) - tolerance, float(mz) + tolerance
def generate_query(self, ionisation, l_mz, g_mz):
return {'adducts.'+ionisation: { '$elemMatch': { 'accurate_mass': {'$gte': l_mz, '$lte': g_mz}}}}
def apply(self, queryset, field, value, negate=False):
ionisation, mz, ppm = value.split(",")
l_mz, g_mz = self.calculate_ppm(mz, ppm)
found = documents.MetaboliteFull.objects(__raw__=(self.generate_query(ionisation, l_mz, g_mz)))
return found.fields(adduct__negative={'$elemMatch': {'accurate_mass': {'$gte': l_mz, '$lte': g_mz}}})
I get the following error.
pymongo.errors.OperationFailure
OperationFailure: Cannot use $elemMatch projection on a nested field.
Any help would be much appreciated, I find it hard to believe that implementing something as important as projection would be an impossible task in this excellent package!
Edit: I hadn't realised how long this question had been outstanding for, should I create a new issue @wojcikstefan ?
KeironO
on 13 Apr 2017
Hey there,
Anyone could retrieve a subnested embedded document using elemmatch?
Let's assume the same entry as the first example in this question and the Class model addition:
Class Embedded Document
class Class(EmbeddedDocument):
class_name = StringField(max_length=120)
School Embedded Document
class School(EmbeddedDocument):
name = StringField(max_length=120)
classes = EmbeddedDocumentListField(Class)
User Document
class User(Document):
first_name = StringField(max_length=60, required=True)
last_name = StringField(max_length=60)
schools = EmbeddedDocumentListField(School)
Feeding documents
user = User.objects.create(first_name="Rohit", last_name="Khatri")
user.schools = [
School(name="Auden High School"),
School(name="Baldwin Boys High School"),
School(name="Baldwin Girls High School"),
School(name="Aukamm Elementary School", classes=["math", "english", "science"]),
School(name="Mason-Rice Elementary")
]
user.save()
I'm trying to retrieve only the math classes.
I'm trying the following query:
users = User.objects.filter(schools__name=name).fields(schools__classes={'$elemMatch': {'class_name':'math'}})
But didn't work.
When I do the following query it returns all the classes, I want only math class.
users = User.objects.filter(schools__name=name).fields(schools={'$elemMatch': {'classes.class_name':'math'}})
The return:
[
{
"first_name": "Rohit",
"last_name": "Khatri",
"schools": [
{
"name": "Aukamm Elementary School",
"classes": [
{
"class_name": "math"
},
{
"class_name": "english",
},
{
"class_name": "science"
}
]
}
]
}
]
I want to retrieve only the math class:
[
{
"first_name": "Rohit",
"last_name": "Khatri",
"schools": [
{
"name": "Aukamm Elementary School",
"classes": [
{
"class_name": "math"
}
]
}
]
}
]
Anyone has face something like this before? Any help help will be welcome.
4ndr
on 28 May 2018
@4ndr i face same thing, but now i also want to now how to deal with it
cmdedj
on 8 Jul 2019
Related issues
lanfon72
路
3Comments
alon
路
4Comments
BurkovBA
路
4Comments
dmitry-saritasa
路
4Comments
BenCoDev
路
5Comments
Most helpful comment
Hey there,
Anyone could retrieve a subnested embedded document using elemmatch?
Let's assume the same entry as the first example in this question and the Class model addition:
Class Embedded Document
School Embedded Document
User Document
Feeding documents
I'm trying to retrieve only the math classes.
I'm trying the following query:
But didn't work.
When I do the following query it returns all the classes, I want only math class.
The return:
I want to retrieve only the math class:
Anyone has face something like this before? Any help help will be welcome.