Cms: Craft 3: eating up a lot of system resources while generating dashboard thumbnails

[michalochman]

Description

It seems that when you view the thumbnail list associated with Asset Field there are a lot of requests being made by Craft to generate the thumbnail files.

These requests eat up system resources very fast making the whole site less responsive.

I'm not sure if it's a good idea for the dashboard thumbnails to be cached alongside user defined image transforms, but they should be cached somehow.

Steps to reproduce

1. Create an Asset Field
2. Create an Entry with the above Asset Field assigned
3. Click the "Add an asset" button
4. See the popup with thumbnails
5. Observe logs

Additional info

• Craft version: Craft CMS 3.0.0-beta.26
• PHP version: 7.1.9
• Database driver & version: MySQL 5.7.12
• Plugins & versions:
  
  
  
  • Amazon S3 1.0.6
  
  

Generated Thumbs HTML:

I can see a lot of log entries like these in my access.log:
1.2.3.4 - - [13/Sep/2017:19:32:50 +0000] "GET /admin/resources/resized/42/60?d=1497534046 HTTP/1.1" 200 763 "https://example.com/admin/entries/somehandle1/2137-somehandle2" "User Agent string"

Example average latency spike I have encountered:

[brandonkelly]

They do get cached, but they’re stored in the storage/runtime/ folder, so they are still served over PHP, which isn’t very efficient.

We have experimented with having them be actual transforms, there couple a few disadvantages to going that route:

• takes even longer to generate the transforms the first time, especially if the images live on a remote volume like S3.
• thumbnails won’t work if the volume URL setting is incorrect (this is perhaps minor, but I do kindof like how thumbnails always work in the CP, regardless of whether you got the URL right).

We do have an option with Craft 3 that we didn’t have before though – maybe we can start storing them in that cpresources folder where other CP images/CSS/JS resources get published. That way if the image already exists, Craft can set the img src attribute to a web-accessible URL, rather than having to serve the images over PHP.

We could also consider pregenerating the thumbnails on upload, to help with the upfront processing the first time you go to the Assets section.

[michalochman]

Ok, they are indeed stored in storage/runtime/ folder which allows to serve them directly by web server using some not-so-clever rewrites. I'll get back here with my nginx solution when I have some time soon.

[michalochman]

This seems to work with my nginx configuration:

# try to serve resized dashboard assets directly
location ~ /resources/resized/([0-9]+)/([0-9]+) {
    root /app;
    try_files /storage/runtime/assets/cache/resized/$2/$1.png
              /storage/runtime/assets/cache/resized/$2/$1.jpg
              /storage/runtime/assets/cache/resized/$2/$1.jpeg
              @rules
  ;
}

If the files are already cached, they are served directly from disk. If they are not cached then they are served through Craft. The only problem I can see is that the cache is never revalidated this way. I'm not replacing the assets so I don't care. This would be possible by comparing the ?d= query string parameter with the actual modification time of cached file.

And this is my @rules related part if anyone is curious:

location / {
    try_files $uri $uri/ @rules;
}

location @rules {
    rewrite ^(.*) /index.php?p=$1 last;
}

[brandonkelly]

Yeah we obviously don't want to force everyone to add that redirect. Working on a better solution…

[yellowmamba]

@brandonkelly This is becoming quite a blocker for us in Craft 2 as well. When I go to Assets section in admin, browsing the list of an asset folder, some requests to /admin/resources/ take forever and eventually cause the nginx to give up (504 timeout error). The php container has also quickly spiked the memory at 100%. What's the best workaround? I assume this thumbnail is not the same thing as a transform right?

[brandonkelly]

@yellowmamba We fixed this for Craft 3 with a new feature in Yii 2 that isn’t available in Craft 2. So if this is a serious issue, you’ll have to look into upgrading your site to Craft 3. (It’s in the Release Candidate stage now, so pretty stable.)

[proposifymax]

Hi There @brandonkelly. Can you elaborate on the fix you mentioned here? We just updated from craft 2 to 3 and are still having the issue of an almost unusable asset library that sometimes kills the server when generating asset thumbnails from S3 volumes. In other words, unlike @yellowmamba 's problem, we're not just 504ing the CP requests but the entire site for all visitors as well.

We're running a load-balanced Elastic Beanstalk AWS environment with an m5.large powering the CMS and nothing else.

Scrolling through or searching in the asset library (~500-1000 assets per volume) very often 504s the entire box. We're currently investigating adding the storage/runtime/assets/thumbs folder to our deployment package to ELB but that seems like a band-aid at best. Is there some way to use a console command or something in craft to tell it to regenerate the thumbnails in an off thread?

I looked into https://github.com/ostark/craft-async-queue but just installing it doesn't seem to alleviate the resource consumption issue.

EDIT:
Some more digging showed that the Control Panel kicks off a AJAX request to fetch each thumbnail individually. So even if a scroll loads 20 or a search returns 5 images, it would then fire an additional 50 or 5 AJAX requests to grab each thumbnail. Handling all these concurrent requests in combination with some of the time having to create the thumbnail is what drives the serve CPU % up and 504s the site. Is it not possible to batch the thumbnail calls with the return of the scroll / pagination loading and background processing for thumbnails that update in the UI via a websocket?

AWS EC2 -> M5.large (2 vCPU, 8 GB Ram)

PHP version | 7.2.8
MySQL 5.6.41
Imagick 3.4.3 (ImageMagick 6.9.6-2)
Craft Pro 3.0.27.1
Yii 2.0.15.1
Twig 2.5.0
Guzzle 6.3.3
Imagine 0.7-dev

[brandonkelly]

@proposifymax Craft should only be loading up to 3 thumbs at once (see ElementThumbLoader.js). So that shouldn’t be more than the server can handle.

You might try setting up an EFS mount for your storage/ folder, and telling Craft to use it by defining the CRAFT_STORAGE_PATH PHP constant in index.php. That way Craft won’t have to re-fetch your images from S3 each time you redeploy or EC2 spins up a new server.

[proposifymax]

@brandonkelly we added the EFS mount and set the ENV VAR to point to the folder but craft isn't using it and keeps writing to storage/ instead. EFS mounts default to 777 so it's not a write permission issue. Any advice?

[brandonkelly]

@proposifymax CRAFT_STORAGE_PATH isn’t an environment variable; it’s a PHP constant. You set it from index.php (before the require line).

You could have its value set by an environment variable, though.

if ($storagePath = getenv('STORAGE_PATH')) {
    define('CRAFT_STORAGE_PATH', $storagePath);
}

If you’re using .env to set environment variables, make sure to add that _after_ this code:

if (file_exists(CRAFT_BASE_PATH.'/.env')) {
    (new Dotenv\Dotenv(CRAFT_BASE_PATH))->load();
}

[mtwalsh]

@proposifymax We're suffering from a similar issue, did moving the storage folder to EFS resolve it for you?

[proposifymax]

@mtwalsh It did to a certain degree, but there were quite a few gotchas:

• You'll have to do some testing on your .elbextensions script to make sure the mount works/comes back on instance reboots or new deploys
• We ended up only symlinking part of the craft storage folder to the efs volume rather than the entire storage folder as suggested above. The reason for that was that was that in a load-balanced environment with multiple instances various craft auto-generated files such as storage/runtime/compiled** caused server errors when it was used by multiple instaces in the same auto-scaling group
• we also ended up using the blitz-cache plugin to take a load of the server as the thumbnail generation still hammered the server unreasonably much which means we also use the efs volume to serve the static cache files
• When running multiple instances you'll have to add session storage support via Memcache or redis (we used memcache) and DO NOT use session stickyness in the AWS load balancer (as that would mean one users thumbnail requests would still hammer one instance rather than 2+)

A tip for sym-linking if you're going that route:

• in the .ebextensions/my-efs.config file the symlinking has to be run under container_commands rather than just commands where you'd be mounting the efs volume
• in order for the directories to be correctly readable/writeable by php it's important to do the linking as the webapp user ELB uses. So the commands have to be used like this:

container_commands: 01_storage: command: sudo -u webapp mkdir ./storage/runtime test: '[ ! -d ./storage/runtime ]' 02_folders: command: sudo -u webapp ln -sf /efs/assets ./storage/runtime/assets

Overall it's not a great solution but works for now. We have two C5 instances running that both spike to 80% CPU for about 5 seconds whenever a user scrolls through the asset library. At least the asset library doesn't 502 the site anymore.

[mtwalsh]

@proposifymax Wow, thanks for such a detailed response that's really helpful.

I've used EFS before and just baked it into the AMI via the /etc/fstab file so it's always there on reboot or a new instance launch.

Can you recall what sort of errors you were getting when shifting the entire storage folder off to EFS? I've found with EFS performance is pretty lousy, especially when the amount of data is small (which is likely the case here). So swapping to the provisioned throughput mode even just at 10 MiB/s can make a difference for not that much additional cost.

We've had good success with using Varnish Cache with Craft, I haven't tried the blitz-cache plugin.

Yes, we're already sharing sessions via AWS Elasticache (Redis).

@brandonkelly Are you aware of other Craft sites that are successfully running this setup?:

You might try setting up an EFS mount for your storage/ folder, and telling Craft to use it by defining the CRAFT_STORAGE_PATH PHP constant in index.php.

[brandonkelly]

@mtwalsh We are for craftcms.com and id.craftcms.com. @angrybrad set it all up, maybe he can offer some guidance.

[mtwalsh]

@brandonkelly That's a pretty good vote of confidence then! I'm contemplating running some tests to see how Craft performs running entirely from EFS (provisioned throughput mode) as it would streamline a lot in terms of deployment, launching new sites and autoscaling. Yes, any further guidance @angrybrad can offer would be most welcome.

[angrybrad]

@mtwalsh we're putting the whole storage folder on EFS without any issues using CRAFT_STORAGE_PATH. The only thing we have symlinked to EFS is the cpresources folder. Overall, it's been a pretty solid setup for us.

[mtwalsh]

@angrybrad Thanks, I will be giving this a try very soon.

[mtwalsh]

@angrybrad Sorry to dig up an old issue but moving the storage folder to EFS has (predictably) hit performance. So I'm wondering if there's any room for compromise? I've been able to symlink the cpresources folder over to EFS so perhaps doing the same with the storage/runtime/assets folder? But leave the remainder of the storage folder on the instance? I can use the "Temp Uploads Location" setting to avoid any problems caused by storage/runtime/temp not being shared. Is this possible or will it just create other problems?

Out of interest are you guys still sharing your entire storage folder via EFS? Is that with throughput mode burstable or provisioned?

@proposifymax Sounds like you are running with this setup (just symlinking the storage/runtime/assets folder)? Is that still working OK for you?

[brandonkelly]

@mtwalsh We still use an EFS mount for storage/ yes (not sure about the throughput mode), but it’s not necessary anymore thanks to that new Temp Uploads Location setting as you noted.

Everything else in storage/runtime/ can be easily regenerated as needed.

Even cpresources/ files can be republished on demand as of 3.0.3, so no need to symlink that. (If on-demand republishing isn’t working for you, check your server config. We’ve seen Nginx configs that don’t point 404ing file requests over to index.php like other web traffic.)

[proposifymax]

@mtwalsh yeah, we're symlinking a bunch of other cache folders as well but it is working nicely and reliably. For instance we use the imager and blitz plugins and their respective storage folders are also symlinked to the EFS volume.

[mtwalsh]

@brandonkelly Ah, this is interesting! Hopefully I can abandon EFS altogether then. I'll report back (and hopefully not in 9 months time!).

@proposifymax Thanks for getting back to me, I've just tested scrolling through assets at speed and Craft (3.3.3) is only requesting a maximum of 3 asset thumbnails at a time so the original issue is no more. So perhaps you can also drop EFS (for the storage and cpresources folders).

[proposifymax]

@mtwalsh There is another benefit to leaving it on the EFS though and that is persistence. Any new ELB deploy will nuke the existing thumbnails which means that when the control panel requests them they have to be regenerated on the fly. Keeping them on the EFS volume means the cpanel just requests existing resources as opposed to requesting and making them. The benefit here of course depends on how often you deploy and how many thumbnails/assets you have in craft :)

[mtwalsh]

@proposifymax Yes, that's a good point.

[mtwalsh]

I ended up switching everything away from EFS, the performance hit was too much (when symlinking the whole storage folder) and more importantly we hit this issue, which actually took a site down entirely for a short period:

https://github.com/craftcms/cms/issues/3475#issuecomment-531703581