Server: [Previews][Question/Request] Obsolete previews seem to be not cleaned regularly

[MichaIng]

In relation to the question of @JSoko in previewgenerator: https://github.com/rullzer/previewgenerator/issues/64

I recreated the issue here, because it's actually a question about the core preview system:

• Are the file previews regularly checked for obsolete/orphaned ones and cleaned like/together with the filecache database?
• If not, it would be helpful to have a command + background job the regularly do that.
• Normally previews will be removed automatically, if their related files are removed in common way, done by the watcher.php. But for process errors, manual file deletion (outside of nextcloud) and other reasons the deletion could be not tracked by nextcloud/watcher and the preview could remain and forgotten, consuming capacity.
• For the same reason there is the occ files:cleanup command and a related background job to clean the filecache/index. It seems just reasonable to have the same for previews or simply include previews into the existing jobs.
• Excuse me, if this already happens, but I didn't found it in the code so far and files:cleanup seems to just include tags and comments:

    public function run($argument) {
        $this->cleanSystemTags();
        $this->cleanUserTags();
        $this->cleanComments();
        $this->cleanCommentMarkers();
    }

[MorrisJobke]

@rullzer you recently fixed the preview deletion. I would also say, that the previews should be properly cleaned up now.

[rullzer]

They should (:tm:)
I'm still thinking a command that checks this might not be such a bad idea. But I must admit it is not high on the todo list

[tavinus]

Hi! +1 for the command

I was wondering how I can clean the previews safely at this point.
@rullzer 's app is nice for pre-generating and all, but I am creating a thumbnail monster that is trying to bite me back.

I had it processing for a full day and then I cancelled the processing to tune the config so I could have it a bit faster, but I still need big images to be shown. Because of that I think it is redoing everything again with different sizes, increasing even more the footprint of the thumbs.

At this point I have around 30Gb of thumbs and I think it is not even halfway with the processing. I am guessing that the 1TB of data they have could generate 100Gb of thumbs (10%) which is insane! It is still processing though, so we will see. BTW, they do have a lot of images there, but that is probably around 20% of their data at most.

So, first problem is to clean up this mess. I will probably want to delete this and reprocess (or not).
If I delete the NC_data/appdata_ochxsdyhi70d/preview folder, or its contents, I am guessing that the database will still have references to these files, right? How can I get rid of this safely? And also clear the database?

Second I have to find and stick to a config that does not create such a big footprint, but I still need big images to be shown as big as possible when clicked. I should probably ask about the config at the preview pregeneraton app. Maybe I shouldn't pre-generate at all :(

Edit: Can I clear the folder NC_data/appdata_ochzsdyhi70d/preview and use occ:files scan ?

Edit2: How about deleted files? are their previews removed? How about removed external storage devices?

[stweil]

I am running Nextcloud 13, have nearly 5 GB in data/appdata_*/preview and wonder whether I may just delete that directory. Or is there another solution? Searching for one I found old hints for running php ./occ gallery:delete-thumbnails --all (which does not work with Nextcloud 13) and the issue here.

Maybe the priority of this issue is higher than "low", because it obviously affects many people, and having a third of the whole disk space occupied by preview images does not seem a low priority issue for me.