October: Slow performance in Media Manager with a large number of images

[lethnh]

I user OctoberCMS and use MediaLibrary and i has 98k images in MediaLibrary and when i login and click media in menubar
media plugin lag and load two minutes, please help me,
i think a few solutions:
+pagination MediaLibrary
+config MediaLibrary
please help me how to pagination MediaLibrary thank all !

[bennothommo]

@lethnh Another suggestion is that you could also perhaps split the images off into separate folders, so you're not loading all 98k images in the root folder.

[lethnh]

do you have another solution? @bennothommo
like pagination

[Samuell1]

Pagination isnt possible because its still needs to read all files in filesystem. Your only solution is to cache list of them somehow. Or like @bennothommo said its best solution

[bennothommo]

@Samuell1 is right - when it reads a folder, it reads all files within. Whilst we could potentially look to lazy-loading the list of files, it will still be slow reading that amount of files regardless.

[Samuell1]

@bennothommo i think best option to make media library perfect is to make it to store to database

[drmzio]

This issue should be re-opened as it could be a problem in the near future. It's possible that October renders ALL files at once, when it should only render what's in-view of the window. This library could fix this issue https://clusterize.js.org. They have a good demo. This could fix October's performance issue with media manager.

The main idea is not to pollute DOM with all used tags. Instead of that - it splits the list to clusters, then shows elements for current scroll position and adds extra rows to top and bottom of the list to emulate full height of table so that browser shows scrollbar as for full list

[Samuell1]

@drmzio this isnt frontend issue...

[LukeTowers]

There have also been suggestions to use glob functions instead of the current listAllDirectories() approaches on the storage facade.

[bennothommo]

@Samuell1 It could be both a front-end and back-end issue - I'm sure the DOM rendering 90k rows is likely going to be slow. @drmzio's suggestion looks interesting.

[akshaysethia]

@lethnh What you can do is make an index of all the images in the database, with the names of the image as the key and their location as the value. [ Using the indexing method of DBMS] [Do the mapping during the insertion of the image] [Try making migration for this]. SO when you open the page or request for the page, your program should fetch this index file and load one page of data on the front-end part of the website. SO once the user moves using the pagination, the files can be viewed accordingly without any lag or much of buffering on the frontend and also does not require the server to take so much stress. So using this indexing method will help you to resolve the loading or the lagging problem.

[drmzio]

@Samuell1 This is a combination of both the backend and frontend. The backend finds and returns 98K records and the frontend has to render 98K records. Each record, assuming it's an image, has to send a request (98K HTTP requests in this case). If we only load the records in the window view, only a few requests can be made rather than all 98K requests.

[bennothommo]

I wonder if it is worth forcing some form of pagination as soon as the number of files/images in a folder reaches a certain threshold. I would imagine a folder with 200 files will display just fine with no slowdown, but once you hit say 500, it's likely performance will decrease.

[github-actions[bot]]

This issue will be closed and archived in 3 days, as there has been no activity in the last 30 days. If this issue is still relevant or you would like to see action on it, please respond and we will get the ball rolling.

[github-actions[bot]]

This issue will be closed and archived in 3 days, as there has been no activity in the last 30 days. If this issue is still relevant or you would like to see it actioned, please respond and we will re-open this issue.

[github-actions[bot]]

This issue will be closed and archived in 3 days, as there has been no activity in the last 30 days.
If this issue is still relevant or you would like to see it actioned, please respond and we will re-open this issue.
If this issue is critical to your business, consider joining the Premium Support Program where a Service Level Agreement is offered.

[github-actions[bot]]

This issue will be closed and archived in 3 days, as there has been no activity in the last 30 days.

• If this issue is still relevant or you would like to see it actioned, please respond and we will re-open this issue. - If this issue is critical to your business, consider joining the Premium Support Program where a Service Level Agreement is offered.

[github-actions[bot]]

This issue will be closed and archived in 3 days, as there has been no activity in the last 30 days.
If this issue is still relevant or you would like to see it actioned, please respond and we will re-open this issue.
If this issue is critical to your business, consider joining the Premium Support Program where a Service Level Agreement is offered.

[mabehiry]

I've the same issue, how can I limit the number of rendered files to only 30 files?

[duycong2509]

I hope this issue will be resolve. It could be a necessary problem in the future.

[LukeTowers]

It will be resolved when someone has the time to deal with it themselves to submit a PR, or when / if someone sponsors the issue if they don't have the time or skill required to fix it themselves.

[bennothommo]

@duycong2509 Separate and store your files in folders. I can't think of many cases where it would be necessary to have 10,000+ files in the one folder that you want accessible through the Media Manager.

[github-actions[bot]]

This issue will be closed and archived in 3 days, as there has been no activity in the last 60 days.
If this issue is still relevant or you would like to see it actioned, please respond and we will re-open this issue.
If this issue is critical to your business, consider joining the Premium Support Program where a Service Level Agreement is offered.

[bennothommo]

I've made an attempt to improve the performance of the Media Library with a large number of images (10k+), however, the main performance decrease that occurs may unfortunately be unsolvable at this point.

The Flysystem library that we use to provide the ability to store files locally, or on remote sources such as FTP, AWS, etc, does not have the capability to limit or paginate the contents list via the listContents method they provide on their API. As such, calling this method will retrieve the entire folder's contents. This is the main source of slow performance - while API-driven file sources like AWS can generally handle this pretty well, reading paths locally or via FTP will always slow down as more files are read.

Even if we were to implement a way to limit the folder content list, however, there's no guarantee that this will work with the filtering and ordering functionality that the Media Manager provides - while the API-driven filesystems usually provide ways to do this through their API, this is simply not the case with local files and FTP, in which we have to be able to read the entire content list before being able to filter or sort the results.

We have tried to mitigate this with caching, which does work after the initial load, but as soon as the cache is flushed, is disabled or expires, or if you navigate to another folder with a large number of images, the slowdown will occur again.

At best, all we can do is limit the amount of results that displayed on screen at one time, to at least mitigate browser-based slowdown from the amount of HTML code being loaded and/or manipulated - this is only a very small fraction of the slowdown experienced with a large number of files.

[LukeTowers]

@bennothommo I think it's those reasons which is why the System\Models\File model uses folder sharding and WP uses date based upload folders. A plugin / config option could probably be made to implement the date based sharding of uploaded flies, but I can't think of anything off the top of my head that would easily fix it for existing files beyond updating all the places the existing files are referenced in.

[github-actions[bot]]

This issue will be closed and archived in 3 days, as there has been no activity in the last 60 days.
If this issue is still relevant or you would like to see it actioned, please respond and we will re-open this issue.
If this issue is critical to your business, consider joining the Premium Support Program where a Service Level Agreement is offered.