Server: After deleting an (image) file, the preview files remain forever

[lukasb314]

Steps to reproduce

1. Upload any image file (i.e. via the web interface)
2. Open it / look at it, so that preview files are being created in appdata_appid/preview
3. Delete the file via the web interface
4. Empty Nextcloud's trash bin
5. Check the preview files on your HDD. You will still find the previews of the image you have just deleted.

I recorded my screen to show you what is going on.
In the video I'm going to upload a picture, then I'm going to look at the previews so that those are created and then I delete the file again. Simultaniously I keep an eye on the appdata_appid/preview folder. Watch the preview folder/file dates, then you'll see what I mean.
https://goo.gl/photos/5u1PCSPWU61we8EM8

Expected behaviour

All preview files of deleted images should be deleted as well.

Actual behaviour

Preview files are not being deleted.
Well, let's put it this way... It seems like they _are_ being deleted, but at the same time, some _new_ preview files are being created.

Server configuration

Operating system: Ubuntu 16.04

Web server: Apache

Database: MySQL

PHP version: 7.0

Nextcloud version: 12.0

Updated from an older Nextcloud/ownCloud or fresh install: Update from 11.0.3

Where did you install Nextcloud from: Manual installation via zipped files

[rullzer]

We do delete the previews when you remove the file. However it seems some are recreated when you view the file in the trashbin. And then when you empty that they are not deleted.

I'll look into this

[rullzer]

Aaah so it seems to happen because we don't emit the normal pre/post delete hooks for the trashbin... @icewind1991 any particular reason for that?

[lukasb314]

However it seems some are recreated when you view the file in the trashbin. And then when you empty that they are not deleted.

That is correct. That's exactly whats happening.

[lukasb314]

Meanwhile I have written a little PHP script, which deletes all the unused preview files. (In fact, as a first step, it does not delete the files but moves it into a different directory - safety reasons, as I was not sure if the script is doing what it's supposed to :) )

I want to share it with you, but i need to warn you!
My PHP or programming knowledges are very limited. I just wrote it very quickly for my personal use to get rid of the gigabytes of unused previews I have. So this little script is only there to fulfill its purpose. There was no focus on efficiency or security.

I run the script outside the web directory using the command line.
It only works with MySQL databases. Successfully tested it with NC 12.0.

<?php

$NC_system_path = "/var/www/html/NC/"; #needs to end with /
$NC_files_path = "/home/user/NC_data/"; #needs to end with /
$NC_appdata_path = "appdata_o9854jfqxnmj/"; #needs to end with /
$NC_newpath = "/home/user/NC_deleted_previews/"; #needs to end with /
#Instead of deleting the preview files, this script moves the preview folders/files to $NC_newpath.... you never know ;-)

#If not intended, there is nothing to edit below this line

$NC_full_appdata_path = $NC_files_path.$NC_appdata_path;

#To get DB-Information
include($NC_system_path.'config/config.php');

$pdo = new PDO('mysql:host=localhost;dbname='.$CONFIG['dbname'],$CONFIG['dbuser'],$CONFIG['dbpassword']);
$sql = "SELECT `fileid` FROM `oc_filecache`;";

foreach ($pdo->query($sql) as $row) {
    $files_in_db[]=$row['fileid'];
}


$files_in_dir = scandir($NC_full_appdata_path.'preview');

$unused_dirs = array_diff($files_in_dir, $files_in_db);

foreach ($unused_dirs as $row) {

    if ($row != "." and $row != "..") {
        rename($NC_full_appdata_path.'preview/'.$row,$NC_newpath.$row);
        echo "Moved ".$NC_full_appdata_path.'preview/'.$row."\n    to ".$NC_newpath.$row."\n";
    }
}


?>

I'd really appreciate if someone can take care of this bug

[MorrisJobke]

Aaah so it seems to happen because we don't emit the normal pre/post delete hooks for the trashbin... @icewind1991 any particular reason for that?

Ping

maybe also @schiessle

[rullzer]

I had a chat with @icewind1991
The main reason is it is outside the user files. which makes sense else we'd run a serious risk of running into loops etc.

Actually if we would use the Node API we'd have proper hooks to listen to.

So long story short. We should kill the view in files_trashbin (and all other places actually :P). But that is not a quick or easy fix.

[schiessle]

So long story short. We should kill the view in files_trashbin (and all other places actually :P).

Agree, but for now it is probably easier to just listen to the pre-/post-delete hook of the trashbin to clean up previews: https://github.com/nextcloud/server/blob/master/apps/files_trashbin/lib/Trashbin.php#L530

[rullzer]

Ah right. Mmmm might require me do do some nasty hacks. but I'll look into a quick fix.

[lukasb314]

Thank you for taking care of this.

[rullzer]

Fix in #5852

[alexandroskr77]

@lukasb314 Thank you for your script above!!! It just works!

[someonestolemyusername]

If you delete a folder with lots of images and PHP runs out of memory processing the delete hooks, is there a process to clean the orphaned previews up asynchronously?

[MorrisJobke]

If you delete a folder with lots of images and PHP runs out of memory processing the delete hooks, is there a process to clean the orphaned previews up asynchronously?

This is usually done in a background job. cc @rullzer

[rullzer]

We run a background job to clean up previews that do not have a corresponding entry in the filecache.