Mongoengine: how to delete from filelist = ListField(FileField())

[francisliyy]

I have a listField for storing file list, for example, filelist = ListField(FileField())
how can I delete one file from this list? I try to get one file then use file.delete(), it really deletes the file from fs.files, but from filelist, you can still have two object id inside, like
"filelist" : [
ObjectId("5b6b45ab360e2eceefa39f18"),
ObjectId("5b6b5735360e2ed0cad8e9f7")
],

But if I try to use pull like
Classname.objects(id=pk).update_one(pull__filelist=file), it throws error "bson.errors.InvalidDocument: Cannot encode object: "

How can I deal with that? Please help, thank you.

[PaPablo]

OOC answer here, how do you create a save the FileFields and store them in the list?

I'm tyring to do the exact same thing and it didn't work

[francisliyy]

@PaPablo , you can do like this:
At first, declare the field as ListField(FileField()) in models.py, like "filelist = ListField(FileField())"
Then if you read an local file, do like this
afile = open(os.path.join(os.path.dirname(__file__),'test.csv'),'rb')
onefile = mongoengine.fields.GridFSProxy()
onefile.put(afile,content_type='csv',filename = 'test.csv')
obj.filelist.append(onefile)

[PaPablo]

@francisliyy I found a solution like that online and it worked.

Still a kinda obscure solution. But it works nonetheless

[bagerard]

Hi,
I also struggled to reproduce it... Thanks for clarifying it.
Using the __pull operator currently fails due to a bug but the following is working:

import io

class FileBundle(me.Document):
    files = me.ListField(me.FileField())

bundle = FileBundle()

file_content = io.BytesIO(b'blah blah blah')
grid_fs_proxy = me.fields.GridFSProxy()
grid_fs_proxy.put(file_content)

bundle.files.append(grid_fs_proxy)
bundle.save()

# instead of this (which raises an error):
# FileBundle.objects().update_one(pull__files=grid_fs_proxy)

# Do this to delete the file and remove it from the FileBundle.files list
print(bundle.files)            # [<GridFSProxy: 5b6de8b0f3ede836031f2dcc>]
bundle.files[0].delete()
print(bundle.files)            # [<GridFSProxy: None>]
bundle.files.pop(0)
print(bundle.files)            # []
bundle.save()

I've identified the issue with the "pull" operator on the FileField, I'll send a PR shortly

[francisliyy]

@bagerard Thank for your solution, I will try it later.
Just let you know, I am using a tricky way to do it now:
for file in filelist:
if hasattr(file,'name'):
print('file exist')
if the file was deleted, though it appeared in the filelist, it didn't have a name attribute.
But your way is the real solution.

[bagerard]

Ok good to know. Note that the 'grid_id' attribute of the GridFSProxy gets set to None once the file is deleted, its probably cleaner to do the following:

bundle.files = [f for f in bundle.files if f.grid_id is not None]

[PaPablo]

With the last comment from @bagerard I would guess that the original intent of the issue is solved.

Btw using the grid_id attribute can be used to avoid referencing inexistent attributes in FileField objects. For example the name attribute won't exist until you put a file and save the object

[francisliyy]

@PaPablo, I tried pop, like what you said, when you print(bundle.files), it shows [], but from MongoDB, you can still see the file.
Like what the picture shows, I have 3 files inside, after deleted one, from console log, you see only 2 left, but actually still have 3 when you use MongoDB client to search.

[bagerard]

Could it be that you didn't save your object after popping the GridFSProxy from the list? In fact see below, I'm printing the state of the object through pymongo (FileBundle._get_collection returns a pymongo.Collection instance) between each steps:

In [1]: import mongoengine as me
   ...: me.connect()
Out[1]: MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, read_preference=Primary())

In [2]: import io
   ...: 
   ...: class FileBundle(me.Document):
   ...:     files = me.ListField(me.FileField())
   ...: 
   ...: bundle = FileBundle()
   ...: 
   ...: file_content = io.BytesIO(b'blah blah blah')
   ...: grid_fs_proxy = me.fields.GridFSProxy()
   ...: grid_fs_proxy.put(file_content)
   ...: 
   ...: bundle.files.append(grid_fs_proxy)
   ...: bundle.save()
Out[2]: <FileBundle: FileBundle object>

In [3]: FileBundle._get_collection().find_one({'_id': bundle.id})
Out[3]: 
{'_id': ObjectId('5b7720284ec5dc1e504b9e6a'),
 'files': [ObjectId('5b7720284ec5dc1e504b9e68')]}

In [4]: print(bundle.files)            # [<GridFSProxy: 5b6de8b0f3ede836031f2dcc>]
   ...: bundle.files[0].delete()
   ...: 
   ...: 
[<GridFSProxy: 5b7720284ec5dc1e504b9e68>]

In [5]: FileBundle._get_collection().find_one({'_id': bundle.id})
Out[5]: 
{'_id': ObjectId('5b7720284ec5dc1e504b9e6a'),
 'files': [ObjectId('5b7720284ec5dc1e504b9e68')]}

In [6]: print(bundle.files)            # [<GridFSProxy: None>]
   ...: bundle.files.pop(0)
   ...: print(bundle.files)            # []
   ...: bundle.save()
   ...: 
   ...: 
[<GridFSProxy: None>]
[]
Out[6]: <FileBundle: FileBundle object>

In [7]: FileBundle._get_collection().find_one({'_id': bundle.id})
Out[7]: {'_id': ObjectId('5b7720284ec5dc1e504b9e6a')}

[francisliyy]

@bagerard , you right, I forgot to save.
Thank you very much.

[bagerard]

You are welcome, happy to help.
Let's keep this opened, because the following still need to be fixed even if there is a workaround:

FileBundle.objects().update_one(pull__files=grid_fs_proxy) # raises  "bson.errors.InvalidDocument: Cannot encode object: <GridFSProxy: 5b6b5735360e2ed0cad8e9f7>"

[francisliyy]

OK, Great

[erdenezul]

I believe this can be closed?

[francisliyy]

@erdenezul , bagerard would like to keep it open until solve the update problem.

[erdenezul]

added labels