Core: Migrate preview generation to async job execution

[LukasReschke]

The preview generation should happen using cron (preferably only the system cron) instead of happening on demand. - Currently we have a lot of users complaining about the performance just because preview generation is taking ages.

\cc @georgehrke
@karlitschek FYI

[LukasReschke]

And yes, I know that this is not possible with encryption, but the cronjob could just check whether the instance is encrypted or not.

Just because there is one use-case where it doesn't work shouldn't prevent massive speed gains on other use-cases ;-)

[LukasReschke]

Actually, we should not remove the on-demand generation but add another cronjob that does the generation already before in some "common" sizes (i.e. retina and non-retina).
If the user visits the folder before the cronjob happened the on-demand generation would happen, otherwise the existing pre-generated thumbnails would be shown.

[karlitschek]

I'm not really sure how this would improve the user experience. A background job is not faster with generating the thumbnails neither is it done necessarily at the optimal time. I assume people keep on suggesting a background job because the old on demand generation was blocking the UI which is horrible. So perhaps we should wait if people are more happy with the behavior of 7.0.3

[georgehrke]

I agree with @karlitschek here. Let's wait for user-feedback from 7.0.3.
If people still experience performance issues with 7.0.3 this is definitely something we should consider for ownCloud 8.

a few things we should discuss when considering implementing this:

• breadth-first or depth-first. (I'd vote for breadth-first)
• sizes for caching: maybe also 1024x1024 for public sharing

[LukasReschke]

Setting to ownCloud 8 and triaging. - Let's close this if nobody complains, if not let's see what we can do.

[ronnicek]

@karlitschek it's easy.. set up Raspberry Pi /w Owncloud and load gallery with some photos :)

[budulinek]

Hi, first of all thanks a lot for considering the issue.

@karlitschek the problem is not the UI, but the preview generation.

I have just installed 7.0.3 and the UI itself is really quick. Good job. But the preview generation takes ages on slow machines (15 images take cca 2 minutes). I have a router running Openwrt (1200 MHz, 2GB RAM).

Background job is not faster. I agree. But once the process is over, all thumbnails are ready to be shown. And they do not have to be generated on-demand.

With the cron job, the user (server administrator) can decide when thumbnails are generated. Of course, it is difficult to choose optimal time. But it will be up to him to decide which time is optimal for the thumbnail pre-generation.

My suggestion:

• create new php file, do not use existing cron.php for thumbnail pre-generation
• the new php file would trigger preview generation for Gallery and for Files app
• server administrator can schedule the preview generation using crontab
• server administrator can set the priority of the thumbnail generation with nice
• If the user visits the folder before the cronjob happened the on-demand generation would happen, otherwise the existing pre-generated thumbnails would be shown

[oparoz]

• [x] Define a max size and expose it in the admin area perhaps https://github.com/owncloud/core/pull/13674
• [x] Suggest all apps use that setting for performance reasons.https://github.com/owncloud/core/issues/13607
• [ ] Add a batch processing button for users so that they can upload all the pics from their camera, press the button and come back later to check the result. Users don't know when the cron job will run and are less likely to get frustrated with the gallery loading pictures slowly if they know they simply have to wait 30 minutes for all previews to get generated

[DeepDiver1975]

As soon come to a conclusion regarding asnc file handling/job queing/etc thumbnail generation can make use of this as well. No specific implementation only for thumbnail operations. THX

[oparoz]

A first step could be to detect uploads via the desktop client and generate thumbnails then. All the heavy lifting would be done at that stage.

[cbxk1xg]

Please consider my posting:
https://github.com/interfasys/galleryplus/issues/88#issuecomment-78671040

[danimo]

Any update here? Thumbnail generation _is_ painfully slow, and the way I see it, there is only two ways to improve things:

• Make the client ask for all visible items only: Bad user experience during scrolling
• Pre-render thumbnails: Potential storage overhead

Worst case it doesn't help, I'd say to judge that we'd need a reference implementation first.

@DeepDiver1975 are any conclusion about the specifics you mentioned above yet?

[DeepDiver1975]

As of today thumbnails are stored after generation on the server.
The initial generation itself is done in a lazy manner - meaning as soon as a thumbnail is requested the first time it will be generated.

This initial lazy generation can cause issues - which might be solvable with the async job system we started to implement for 8.1 - we can use it for thumbnails in the dev cycle of 8.2

[mbazdell]

I'd just like to add that running OC8 on my Pi2 is great, and the only problem I have is with the thumbnail generation. I have about 400 images in my InstantUpload folder and while browsing it on my phone it literally takes about 5 minutes to generate thumbnails for 24 images. Then it stops generating until I scroll down and it needs to start working and generate more. All that to say I support the idea of a configurable way of thumbnail generation.

[jeffhoffman13]

Ok so I had deleted my perfectly working thumbnail directory pointed to an external shared directories mounted and linked locally now when I try to view images, I get the forever circling icon and no thumbnails?

Getting these errors:

[jeffhoffman13]

Error core OC_Image->data. Error getting image data. 2015-04-13T12:47:57+00:00
Error PHP imagepng() expects parameter 1 to be resource, boolean given at /var/www/owncloud/lib/private/image.php#292 2015-04-13T12:47:57+00:00
Error PHP Undefined index: REQUEST_URI at /var/www/owncloud/apps/calendar/appinfo/app.php#41 2015-04-13T12:45:05+00:00
Thoughts?

Ok so I have tried this again and multiple directories are being created in the data/user/thumbnails directory each of those containing a file call 200-200.png? So the question is where do the image thumbnails go?

Looking at the 200-200.png image it says could not load the png file? Fatal error loading the png file?

I tried the galley+ app and that generated Thumbnails but resolution is less and I don't believe there is a slide show capability.

[blackm]

I have update my RPi also to be version 2 and I enabled preview again. Since I have 4 core working on this the speed of generation is OK for and I can still do other things...much better that RPi B.

What is useful for me? Cron finding directory's that have been just added but no preview have been generated....and generate preview for this directories.

[oparoz]

This initial lazy generation can cause issues - which might be solvable with the async job system we started to implement for 8.1 - we can use it for thumbnails in the dev cycle of 8.2

@DeepDiver1975 - Still on track for 8.2?

[cmonteroluque]

I don't think this can make it for 8.2. Moving forward at least till 8.2.1

[karlitschek]

i think this is not a critical bugfix but more a feature. so most likely 9.0 and not in a .1 bugfix release

[DeepDiver1975]

so most likely 9.0 and not in a .1 bugfix release

yes

[budulinek]

Hi, I agree, this is not a "bugfix" since Owncloud was designed to generate previews in a lazy manner (on-demand generation). Cronjob for thumbnail generation is "just" another "new feature".

But without this "new feature", Owncloud is simply unusable (as a storage for pictures) for anyone with weaker system (Raspberry Pi, router with Openwrt etc.).

:-(

[oparoz]

@budulinek - You can use the new occ command if your files aren't encrypted:
https://github.com/owncloud/gallery/pull/387

And there should be a solution soonish which will create thumbnails at upload time.

[tobiasKaminsky]

The android app is requesting thumbnails in 128dp (density independent pixel), which is this:
ldpi @ 128.00dp = 96.00px
mdpi @ 128.00dp = 128.00px
hdpi @ 128.00dp = 192.00px
xhdpi @ 128.00dp = 256.00px

If the thumbnails are generated during upload, in which sizes are they created? It would not help the mobile users if they still have to wait as their pixel size is not pre-computed.

[elpraga]

Are there any thoughts of adding options to generate the thumbnails for the mobile devices to the occ command for generating thumbnails? It would mean a huge leap forward in usability on android clients if I could run a cron job to generate the desired thumbnails for newly added files at let's say 4 a.m.

When the client can download already existing thumbnails, they appear instantly in the app (together with the file list) - a behaviour users are used to from other apps when previewing cloud stored photos.

[oparoz]

If the thumbnails are generated during upload, in which sizes are they created?

@tobiasKaminsky - Nothing is generated at upload time, but in the future, the max preview, as defined in the config could be generated, speeding up further thumbnail generation.

Each resolution you've mentioned would be generated the first time it's encountered. After that they will be served from cache.

Are there any thoughts of adding options to generate the thumbnails for the mobile devices to the occ command for generating thumbnails?

@elpraga - Not at this time, since the first step would be for core to define thumbnail categories which would be used to serve the appropriate cached thumbnail

The Android app can already cheat if it wants to. It simply needs to always ask for the 256px thumbnails, avoiding having to generate other sizes, but I don't think it will improve performance by too much. It's the initial thumbnail creation which takes time.

[elpraga]

Do I understand it correctly that the idea is to generate 2048x1536 max preview thumbnail upon upload (size mentioned in https://github.com/owncloud/core/issues/12008 ) and then generate the rest of the thumbnails from this one upon request and thus spees up the process?

[oparoz]

@elpraga - This is what's happening already, yes.

[elpraga]

So, it is happening for every thumbnail request already, including the thumbnail requests from the Android App? Sorry to ask again, I just want to make sure. I just made a test. I uploaded an image to ownlcoud and opened the folder in Gallery app (OC 8.2), and indeed a thumbnail 2048-1516-max.png was created.

Then I opened the same folder in the Android Beta tab and small thumbnail appeared reasonably quickly (at least it did not seem a long time as the directory listing was refreshing as well). Then I clicked on it and it still took some 10-11 seconds for the 973-720-with-aspect.png thumbnail to appear.

That is too long :-/

[oparoz]

So, it is happening for every thumbnail request already, including the thumbnail requests from the Android App?

Yes

Then I clicked on it and it still took some 10-11 seconds for the 973-720-with-aspect.png thumbnail to appear.

• higher latency on mobile
• you're probably not using Redis for files locking
• your server might be underpowered

It's more difficult to debug what's going on on a mobile device, so unless you have experience in this field, you won't be able to see where all this time is wasted, but switching to Redis should halve the time it takes to generate thumbnails

[elpraga]

Thank You for clearing it out for me. Yes, my server is not very strong, but being a home server it is likely that it is a more common case. We are trying to find the best solution with the developers of the mobile app, because I can only see some inconveniences when dealing with the thumbnails, for the rest my server works like a charm.

That was also why I asked if it will be possible in the future to generate the desired resolution thumbnails using occ:gallery (I hope I used the correct name) command and a cron job.

As I mentioned before, once the thumbnails are generated, all works seamlessly.

Thank You again for Your answer.

[oparoz]

Yes, my server is not very strong, but being a home server it is likely that it is a more common case

I agree :)

That was also why I asked if it will be possible in the future to generate the desired resolution thumbnails using occ:gallery (I hope I used the correct name) command and a cron job.

You can always open an issue in Gallery to ask for the occ generate-thumbnails command to generate more than the max preview and depending on funding or a developer's free time, it may get implemented.

[elpraga]

You can always open an issue in Gallery to ask for the occ generate-thumbnails command to generate more than the max preview and depending on funding or a developer's free time, it may get implemented.

Ok! Thank You! As You mentioned here:

@elpraga - Not at this time, since the first step would be for core to define thumbnail categories which would be used to serve the appropriate cached thumbnail

It would be a good idea to wait until some kind of agreement on the thumbnail sizes is met, right?

[oparoz]

It would be a good idea to wait until some kind of agreement on the thumbnail sizes is met, right?

They're mostly undefined within the Web GUI, but @tobiasKaminsky has listed the ones used in Android and you can cheat.

So:

• 32 or 36 for rows in Files
• 400x200 for Gallery
• 200x200 for Gallery (lots will never be used)
• 256 for Android app. The app will still need to cheat and scale all those images or all the other resolutions will still need to be generated as well

[elpraga]

Then I clicked on it and it still took some 10-11 seconds for the 973-720-with-aspect.png thumbnail to appear.

And that is just my tablet, my nexus5 generated 1028xsomething. I am not sure if they have categories yet, I bet they do it by device now, but this might change in the future. The big thumbnails are available in beta now, but there are certainly more categories then @tobiasKaminsky mentioned before.

[oparoz]

Those are more previews than thumbnails and will require categories to be defined and/or all apps to be adjusted

[tobiasKaminsky]

currently the "resized image" in android app is defined by the screen size, which indeed can vary between different devices.
Of course we can force full hd resolution for this. But this has a drawback on devices with lower/higher screen resolution. Same is true for the thumbnails.

[elpraga]

Tobias, You are the one who understands the best the vast differences of Android devices. If I understand it correctly, creating categories would give us the possibility (after convincing the occ:gallery app developer to integrate the possibility of generating different previews :) ) to generate thumbnails and previews for a defined set of devices by those who need it. That would give the Android App huge usability jump on slower servers.

I really don't have enough experience and information to know if it would be a drawback to use several categories instead of tailored previews. But could it be possible to combine some of the larger previews (keeping in mind the bandwidth)? Using slightly bigger previews then the screen size would help us also when zooming.. Some of them really seem to me close engough..

-rw-r--r--   1 www-data www-data  94952 Nov  4 00:10 1080-799-with-aspect.png
-rw-r--r--   1 www-data www-data   3856 Nov  3 23:24 128-128.png
-rw-r--r--   1 www-data www-data 304924 Aug 28 11:35 2048-1516-max.png
-rw-r--r--   1 www-data www-data   9652 Oct 23 01:15 270-200-with-aspect.png
-rw-r--r--   1 www-data www-data    963 Oct 25 01:55 32-32.png
-rw-r--r--   1 www-data www-data  21195 Aug 28 11:35 384-384.png
-rw-r--r--   1 www-data www-data  58902 Nov  5 08:42 811-600-with-aspect.png
-rw-r--r--   1 www-data www-data  79458 Nov  4 00:08 973-720-with-aspect.png

Those are my real thumbnails and previews of one file. I am using Nexus5, Woxter IPS tablet and I opened the file in the web interface as well.

..but as I said, I might be wrong and missing something important..

[TheManOfTheMAn]

Well just to give Feedback on this problem.

It would be a good solution to use a crone script for tumbnail generation, but it should not remove the async. generator.

Example situation: Owncloud gets used by webinterface, desktop client, mobile client and webdav in our company. Now there is a external share also included what is available per smb and other protocols within the LAN.

Now how will the thumbnail generation takes advantage of the async. method, in all given possible variations? There will be only one circumstance: New uploaded files what get accessed per web or mobile client before a possible crone could generate thumbnails for these files.

In all other possible variations, when new files get stored in OwnClouds user or external space, there is no real advantage in waiting for the async. generator. A regular running crone job would have enough time to create thumbnails with each run. Because you normally will not always watch previously uploaded files (desktop client or webdav) in the webclient or in the mobile app. So the solution via crone each 10 or 15 minutes would be the better one in any circumstance except the already mentioned situation.

Also there might be a bug in the android App – preview images often will not be created if you open a once uploaded folder with or without sub folders with the App only. (Even if uploaded by desktop, web or dav). But if you visit a folder with files what are available for preview with the web interface until the async. generator has created them, also the App is displaying them immediately.

Now the Question is? Can you create a crone script, if it would be possible, to create little tumbs in about 100x100 or 200x200 at maximum only and big previous for detail preview on request. This would be the best solution between fast delivery of thumbs in file list and good detail preview of images if needed.

By the way, the App is useless in my case without thumbs for images in the file lists. Because we have hundreds of images just named by a auto number of the camera. So you would need to download each image before you might know on what you are looking at :P

[oparoz]

I'm not really sure how this would improve the user experience. A background job is not faster with generating the thumbnails neither is it done necessarily at the optimal time.

I agree. On demand (on/offline) is a better option.

Ideally, we would use a media processing queue which can process thumbnails 1-x at a time (depending on the number of cores) and a hook to send any file change to that queue. The same backend used for files locking would be used.

1. Something/one uploads a file via WebDAV (client sync, Android, etc.) or the GUI
2. Hook detects a change, assesses if it's a media file and if yes adds the file to the queue
3. Daemon processes queue and sends files to thumbnail generator
4. Generator checks if preview already exists (could have been generated via the GUI)
5. If there is no preview, it generates it

Now there would be a race condition between the thumbnail generator and any client asking for a preview. That could probably be solved by locking the file for reads if it doesn't completely break the clients (I think Gallery would die...)

The biggest problem is that there is no way (I think) to make it work that way, because there is no running daemon which could process the queue and if we don't use the queue, then the thumbnail generation would block the client and people would probably scream from the roof tops that ownCloud sync is slow.

@LukasReschke @MorrisJobke @georgehrke @icewind1991 Is this correct?

[pieter-groeneweg]

I am new at this and I am not a developer, but as I understand it, there are thumbnails created for each and every device that passes by?

• my android phone will create thumbs
• my wife's iPad creates thumbs
• my kids galaxy tab 2's creates thumbs
• my laptop does... and on.
• even on a desktop/laptop with a resized browser window, a new thumb or preview is created.

I just shared a bunch of images from a carnival parade with dozens of people... a lot of thumbs and previews will be created... ouch...

This process requires space and cpu time each time new thumbs and previews have to be made. The users having to wait.
Isn't there a way to restrict this to lets say just four sizes?

• thumb 36x36
• thumb 48x48
• preview 600x400
• preview 2000x1333
  and leave the rest to app/html/css styling?

this would do for practically all devices and save resources on the owncloud server.
it could be done on upload of the image when uploading through the web ui and by cron when uploaded or added through external means (ftp, external locations, other shares....)

Between the max preview of 2048x1365 and the 2000x1333 with aspect is not much difference in file size so now it takes twice as much space and uses cpu processing for solving an issue my browser or app could have done by just html/css styling.
IMO generic thumbs and previews would improve performance.

Thanks

[oparoz]

I am new at this and I am not a developer, but as I understand it, there are thumbnails created for each and every device that passes by?

Not exactly.

• 1 max size thumbnail, as defined by your configuration
• 1 thumbnail for the files app
• 1 photowall thumbnail
• As many thumbnails as there are screen resolutions used to load the image full screen, as long as they're smaller than the max size thumbnails. They're rounded to the nearest 100 pixels

Now, apart from the max size thumbnails, the other ones can also be duplicated depending on the pixel density of the screen. Per example a modern smartphone will have a screen resolution of 500x350, but with a pixel density of 4, so a thumbnail of 2000x1400 will be needed or the image would be blurry.

Isn't there a way to restrict this to lets say just four sizes?

Not really, because just with the example you've given, we would also need the same set with a pixel density of 2,3 and 4 and there are also the gallery thumbnails of 400x200 and 200x200.

Between the max preview of 2048x1365 and the 2000x1333 with aspect

This could be improved, yes. There should be a way to detect that we're near max size and use that.

[pieter-groeneweg]

The first.. where and how do I configure that?

The fourth..

"As many thumbnails as there are screen resolutions used to load the
image full screen, as long as they're smaller than the max size
thumbnails. They're rounded to the nearest 100 pixels"
That could just be done by the app or browser itself. This is the one I
am worried about.. as when I turn my mobile horizontally I already have
another screen resolution.. :(
Rounded by 100px means a possible 20! preview/thumb images per original
image... is there a way to bring this roundup down to lets say 500px?

Op 2-2-2016 om 22:33 schreef Olivier Paroz:

I am new at this and I am not a developer, but as I understand it,
there are thumbnails created for each and every device that passes by?

Not exactly.

• 1 max size thumbnail, as defined by your configuration
• 1 thumbnail for the files app
• 1 photowall thumbnail
• As many thumbnails as there are screen resolutions used to load
  the image full screen, as long as they're smaller than the max
  size thumbnails. They're rounded to the nearest 100 pixels

Now, apart from the max size thumbnails, the other ones can also be
duplicated depending on the pixel density of the screen. Per example a
modern smartphone will have a screen resolution of 500x350, but with a
pixel density of 4, so a thumbnail of 2000x1400 will be needed or the
image would be blurry.

Isn't there a way to restrict this to lets say just four sizes?

Not really, because just with the example you've given, we would also
need the same set with a pixel density of 2,3 and 4 and there are also
the gallery thumbnails of 400x200 and 200x200.

Between the max preview of 2048x1365 and the 2000x1333 with aspect

This could be improved, yes. There should be a way to detect that
we're near max size and use that.

—
Reply to this email directly or view it on GitHub
https://github.com/owncloud/core/issues/11685#issuecomment-178833805.

[oparoz]

The first.. where and how do I configure that?

Look at the sample config, the default is 2048x2048

That could just be done by the app or browser itself.

Except that this can lead to a lot of wasted bandwidth on mobile. Also, iOS devices have a limited amount of memory and will crash when images are too large.

as when I turn my mobile horizontally I already have another screen resolution

No, we define the size to request based on the longest edge.

Rounded by 100px means a possible 20! preview/thumb images per original
image... is there a way to bring this roundup down to lets say 500px?

You can change that in the code yourself or submit a PR to make this user selectable or sponsor that feature.

[elpraga]

Would my tablet's screen resolution help?

[PVince81]

We're past feature freeze, moving to 9.1

[pieter-groeneweg]

@oparoz, Thanks! This i valuable info.
So probably nearest to what I like would be setting these:

preview_max_scale_factor' => 1,
preview_max_filesize_image' => .25,

(can I set to .25 to have 250k? or can I only use Integers?)

[ditaW]

I know this might be kind of off-topic but is anybody interested in a (tiny and a littlebit ugly) greasemonkey-script i wrote to pre-render the thumbs? Currently it scans all dirs and requests the default-thumbnail used for desktop-web-access. It could be adjusted to request other scale-factors if necessary...
https://justpaste.it/rsh8 (hint: create new Greasemonkey-script and paste it unter the comment-section. Tested on FF only)

I know this is not a long-term-solution and i am looking forward to the cron-job but it might be helpful.

[Bugsbane]

Any word on if this is on track for either OC 9.1 or NC 10?

[PVince81]

It's scheduled for 9.2.

Note that the old mimetype repair code will not do anything at update time because it was version dependent. This means that upgrading to 9.1 will not have the slowness it used to have..

[Bugsbane]

upgrading to 9.1 will not have the slowness it used to have

Good news indeed!

[rokco]

would it be possible to calculate thumbnails per bash script on filesystem until this core feature is released?

[PVince81]

Backlog for now as it is tricky due to encryption and sharing.

Maybe the issue really is "previews generation can take a long time" and this async job is just one way of improving it.

CC @pmaier1 for considering this in future releases

[ladiko]

i tried to generate thumbnails on the filesystem by getting each jpgs fileid from the database. and adding the previews to $username/thumbnails/$fileid/1200x896-max.png, 268x200-with-aspect.png, 200x200.png and 32x32.png + scanning the thumbnails folder via occ files:scan --path ... but when I opened the gallery, it didnt use those thumbnails but regenerated them. So i was looking for another solution and it took me several hours of try'n'error but i finally managed to create a bash script which is slow, but works. It took 80 minutes to generate thumbnails for 2200 jpgs of 1200x896 pixels. the curl calls are very slow...

• issues:
  
  
  
  • for other database backends than sqlite one has to adjust the query command
  
  
  • this sqlite query is restricted to jpgs as you see in the command - adjust the query to your needs
  
  
  • as all my pictures are 1200x896, i use these values for the preview generation, no idea if it works for images with different resolutions
  
  

#!/bin/bash

OC_PATH='/srv/http/owncloud'
OC_USER='owncloud_username'
OC_PASSWORD='owncloud_password'
OC_FOLDER='' # f.e. OC_FOLDER='path/within/your/owncloud/webinterface'
HTTP_USER='www-data' # http on archlinux

# run as http / www-data user
[ "$(id -un)" = "${HTTP_USER}" ] || { sudo -u "${HTTP_USER}" "$0" "$@" ; exit ; }

# check if the selected folder exists
[ -e "${OC_PATH}/data/${OC_USER}/files/${OC_FOLDER}" ] || { echo "file not found: ${OC_PATH}/data/${OC_USER}/files/${OC_FOLDER}" ; exit 1; }

START_TIME=$(date +%s)
FILES=$(sqlite3 "${OC_PATH}/data/owncloud.db" "select fileid,path from oc_filecache where path like 'files/${OC_FOLDER}%.jpg'")
AMOUNT=$(wc -l <<< "${FILES}")
COUNT=1
while IFS='|' read -r OC_FILEID OC_FILE ; do
        CURRENT_TIME=$(date +%s)
        EPOCH_DIFF=$((CURRENT_TIME - START_TIME))
        TIME_DIFF=$(date -u -d @${EPOCH_DIFF} +%s)
        TIME_RUN=$(date -u -d @${EPOCH_DIFF} +%-H%-Mm%-Ss)
        ESTIMATED_DURATION=$(date -u -d @$(( AMOUNT * EPOCH_DIFF / COUNT )) +%-Hh%-Mm%-Ss)

        # print some stats about the progress
        echo "$((COUNT++))/${AMOUNT} ${TIME_RUN} of ${ESTIMATED_DURATION} - ${OC_PATH}/data/${OC_USER}/${OC_FILE}"
        curl -sS --user "${OC_USER}:${OC_PASS}" https://owncloud.cosmoproducts.de/index.php/apps/gallery/api/preview/${OC_FILEID}/1280/896 > /dev/null
        curl -sS --user "${OC_USER}:${OC_PASS}" https://owncloud.cosmoproducts.de/index.php/apps/gallery/api/preview/${OC_FILEID}/268/200 > /dev/null
done <<< "${FILES}"

[ronnicek]

Wohooo.. looks like solution is here https://twitter.com/Nextclouders/status/841246279279755264 :)

[hast0011]

Viewing pictures is still veeeeryyyyyyy slow. Even 100 pictures render the system unusable which is responsive in files, calendar or contacts. You can use bitnami, archlinux, debian and owncloud from 7.0 to 10.0 all the same over years.
Your can even try a ready to use stack in a virtual machine with 16 GB RAM.
How should this replace google, dropbox or onedrive?
Even the preview with windows explorer over network drive to the same data is 100 times faster.

Any preview generator is faster than now.

[hanzz]

My approach to workaround that while adding all the historical photos I have to owncloud was following:

1) Open some gallery which does not have previews generated.
2) Open Firefox web developer console (ctrl+shift+k).
3) Wait for XHDR request to /apps/gallery/thumbnails.
4) Right click on it and copy it to "curl" to get the full curl URL.
5) Copy paste that URL to the script below, reformat it to set "fileid" into "ids" GET attr, and run it.

from __future__ import division
import sys
import os
from multiprocessing import Pool
import MySQLdb

db = MySQLdb.connect("localhost","owncloud","pass","owncloud" )
cursor = db.cursor()

sql = "select fileid from oc_filecache where path like 'files/%.JPG'"
cursor.execute(sql)
results = cursor.fetchall()

def run_curl(row):
    fileid = row[0]
    os.system("curl -sS 'http://172.16.16.141/owncloud/index.php/apps/gallery/thumbnails?ids=" + str(fileid) + "&scale=1&square=0&requesttoken=....")

p = Pool(40)

for i, _ in enumerate(p.imap_unordered(run_curl, results), 1):
    sys.stderr.write('\rdone {0:%}'.format(i/len(results)))

You will need to tweak the script and probably also check other extensions, but for someone who can code it's easy to do that and it might save time to someone, so adding it here.

[tryan225]

Any update on this? Preview/thumbnail generation in large folders of photographs is enough to chew through all 1GB ram and knock over my server which otherwise runs ownCloud without a problem.

[ronnicek]

Migrate to Nextcloud :) - https://nextcloud.com/blog/setup-cron-or-systemd-timers-for-the-nextcloud-preview-generator/

[PVince81]

In the past, it was a design choice to not provide features if they cannot be applied to the whole platform and cannot work with some features. Such preview generator would not work with user-key encryption.

Now design goals have changed a bit, so it should be possible to provide such command / background job and add a disclaimer for user-key encryption. External storage will also be slow as it will need to download all the files so this could be added as a warning as well.

Note: the preview system will be reworked / rewritten in the future as the current way is suboptimal and outdated.

[nero120]

Note: the preview system will be reworked / rewritten in the future as the current way is suboptimal and outdated.

That is a very welcome statement @PVince81! Any idea on when we might be able to expect it?

[jonferreira]

Wondering if there's any updates on this subject? Preview is still painfully slow and sucks that thumbnails can't be generated "automagically" with a cronjob :(

[fungs]

@jonferreira see my suggestion #17916 from 2015, basically I was proposing to integrate another JPEG preview program which takes 3 sek instead of 3 min for my example digital photo on a weak CPU. Nothing has happened since :(

[nero120]

@PVince81 are you able to incorporate @fungs solution into your reworking of the preview system?

[jonferreira]

@fungs :(

[PVince81]

@nero120 the reason this was likely never incorporated back then is because the design goals of ownCloud was to be purely PHP to make sure that everyone can deploy this on their shared hosters, etc.

Now in 2019 I'm still unsure if / how to incorporate such external tool in a way to make it work from OC. Maybe it needs an OC app that is able to call it with shell_exec.

In any case, I don't think that's the right direction and we should stay in PHP-world to keep the easy of deployment for everyone.

Currently we're reworking the frontend and integrating both the video player and slideshow view into a single app, see https://github.com/owncloud/files_mediaviewer (release coming soon).

This app implements the previews with fixed scales instead of expecting previews with every possible pixel size. So the viewer would pick the preview with the closest size instead of requesting a preview with the exact size from the server.

I think this is a good prerequisite for a future background job for preview generation, because said job could then simply build previews for all these fixed supported sizes.

Now I'm not aware of any timeline to provide such background job.
On server with 100k users it is likely that such background job would take ages to finish and consume a lot of CPU on many servers. So this needs careful consideration.

@pmaier1

[elpraga]

Finally!! Set preview sizes!! Finally!

[nero120]

Thanks for your comments @PVince81.

On server with 100k users it is likely that such background job would take ages to finish and consume a lot of CPU on many servers.

Whilst that may or may not be true (depending on whether the users on the server upload many high quality images), surely it's for the server admin to determine how best to spend the server's resources and that will depend on the hosting scenario. If you are an admin intending to provide users with a place to upload and share their photos then without preview generation this is not really fit for purpose.

I appreciate your point, but the risk can easily be mitigated by providing a preview generation job with parameters to enable admins to control when and how previews are generated. But I have to be honest, a cpu intensive task such as preview generation must be done in the background otherwise the experience is severely degraded whilst the user is forced to wait long moments every time they want to look at an image...

[PVince81]

so instead of an OC bkg job this could be implemented as an occ command instead, which gives more freedom for admins to decide when to run them.

[tribut]

See also https://apps.nextcloud.com/apps/previewgenerator, which adds an occ command for this (Nextcloud).

[TNCS-git]

After 5-years, still a milestone of 'maybe someday' goal... I sincerely think the target of 'ownCloud' audience is loosing its plot along the way. I believe ownCloud is best suit target at the audience of people who choose not their data over public cloud offering, whether it Dropbox, LiveDrive, Sharepoint, etc. The reason can simply be, but not limited to:

1. Privacy reason
2. Decision over it should be opex (Cloud based) or capex (self host solution)
3. People who like more flexibility and control

I have so many self scenario that a simple OCC command line for admins to do background thumbnail generating at their own freewill will work out better than ondemand generation. For one, copy over the images manually (without OC), another sync mass images without the OC due to performance factors. Made a decision of sync over my Android phone using Folder sync, after 8000+ images, realize it was impossible to use now as ownCloud app can never finish generating the Thumbnail.

Regardless of which situation, this type of limit (and few others), really make me think the ownCloud target audience isn't for what aforementioned, but instead smaller Cloud reseller. What even puzzling me is, how NextCloud has implemented this nearly two years ago, and ownCloud is still considering to do this or not...

[nagelp]

so instead of an OC bkg job this could be implemented as an occ command instead, which gives more freedom for admins to decide when to run them.

Any update on this? I think an occ command would be perfect. My small home server takes really long to generate the thumbnails, so browsing images through the web interface is really painful.

[hast0011]

I removed Nextcloud from my document workflow in the meantime.

Patrick Nagel [e-mail] schrieb am Di., 18. Aug. 2020,
19:52:

so instead of an OC bkg job this could be implemented as an occ command
instead, which gives more freedom for admins to decide when to run them.

Any update on this? I think an occ command would be perfect. My small home
server takes really long to generate the thumbnails, so browsing images
through the web interface is really painful.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/owncloud/core/issues/11685#issuecomment-675623922,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH47X5QHETFMIRPDFZ2KZQTSBK5UHANCNFSM4AWERQEQ
.