Cms: CP slow with DevMode = 1

[davidhellmann]

Description

Can't figure out whats going wrong. But on just one project the CP is totally slow when the DevMode is enabled. If DevMode disabled it's fast. Don't know whats the problem here but I didn't know any setting whats related to this.

It's so slow. Each request takes over 1 minute

Additional info

• Craft version: 3.3.16.4
• PHP version: 7.2.20
• Database driver & version: 8

[Mosnar]

@davidhellmann Is it still slow after the first load of each CP page? Craft is typically quite slow on the first load due to it needing to regenerate cpresources when things get modified or they get deleted.

[davidhellmann]

@Mosnar sadly on every page load. it's a bit pain currently and not usable :(

[lukeyouell]

I suffer with this when working locally.

Having dev mode enabled _drastically_ increases the time it takes to load a page.

[narration-sd]

I'm a little reticent to add in here, as this may be a red herring -- but on the other hand if it's the real situation, reporting could give a means to excite and test. Or another avenue as to what the problem really is

I've seen such long delays literally for years. Not all the time of course, and never on a web-published site. I see them locally, on the usual sort of Homestead/Vagrant VM, on latest-everything Windows 10.

• When occurring, this happens with a vagrant reload, which re-copies from the dev environment to the VM nginx server. Usually done because vendor is different after a composer update etc.
• The site was I think always or near so fast as expected, on a first vagrant up, and if I reboot the laptop and bring Vagrant up again entirely. Not so sure it works often if I just vagrant halt; vagrant up.
• What's seen here has been over years, Craft version doesn't matter. So if it's the actual problem and is in Craft, it's a long-standing problem.
• After an amount of attention, I've put this off to being some sort of Vagrant weirdness. There is often some of that in other areas, which in due course they fix.
• I've had some clues in recent months that when it occurs, it may actually be a DNS problem.
• I set DNS for the VMs via lines in the usual Win10 c:windowssystem32driversetchosts, so DNS should be instant for the Craft page request, but there are other accesses potentially involved.
• I’m more than a little fuzzy on what I saw as far as a request that went outside and hung until a long timeout, unfortunately. And it’s always possible this was something on the page to be accessed, like an image or Vue parts. There was some reason at the time I couldn't track this down.

Ok, this is what I can say. Maybe it will give some ideas.

[narration-sd]

as usual it seems, edited the post usefully after it hit your mail, more facts put in

And a p.s. -- I almost always have been running with devMode set. Of course will try turning it off if/when I see this again...

[narration-sd]

well, a few more thoughts that could be helpful

• I've always used rsync at vagrant up scripting to load the dev state into the VM, so my issues aren't about inter-file system slowness Vagrant can have. But could be for any of you, for 'local' if on Vagrant.

• I've remembered more about some'DNS-seeming' cases.
  
  
  
  • The most recent one at the least, wasn't DNS - the timeout was too long.
  
  
  • this was delay on previews, not on the main Craft CP
  
  
  • I resolved it was because a) I often configure image asset links pointing to the live server for the dev on vm, because this is much more convenient with dev/build states of Vue or React components b) in this case, the Craft server then actually being addressed was having the browser time out on asset requests, rather than returning 404 etc: something to do with caches likely/Andris's magic transform creation.
  
  
  • Fixing the server then brought back all speed.
  
  
  • there's still something that can cause long delays for the CP and any access, and if that's _not_ the presumed Vagrant weirdness, it's still to find
  
  

[brandonkelly]

Not able to reproduce, but if either of you have a Blackfire account, you could try profiling the same CP request with and without Dev Mode enabled, and then compare them. That should help identify the bottleneck(s).

[narration-sd]

Yes, that's a great idea, @brandonkelly.

Blackfire is easy and gives very informative graphic/numeric answers.

If I'd thought of delays as performance...

On Tue, Dec 3, 2019, 12:13 Brandon Kelly [e-mail] wrote:

Not able to reproduce, but if either of you have a Blackfire
https://blackfire.io/ account, you could try profiling the same CP
request with and without Dev Mode enabled, and then compare them. That
should help identify the bottleneck(s).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/craftcms/cms/issues/5298?email_source=notifications&email_token=AAB4RCJWTODPVV2LBU72H7TQW24XZA5CNFSM4JUS6A7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF2VJAQ#issuecomment-561337474,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAB4RCMDH5DFXDSEVJOVJQLQW24XZANCNFSM4JUS6A7A
.

[davidhellmann]

I've no Blackfire Account :|
@brandonkelly are there some other settings beside the devmode setting that can affect the CP speed?

When I open the Debug Bar within the CP I have this error:

Strange :(

[ghost]

Don't know if it's related to this, but i see something similar to the first screenshot (missing icons) when messing up site urls, e.g. with this setting in .env (both pointing to the same target):

# DEFAULT_SITE_URL="http://starter.local"
DEFAULT_SITE_URL="http://192.168.178.35:8081"

when accessing the cp with http://starter.local/admin.

Doesn't slow anything down on a WAMP installation, but maybe in other environments?

[narration-sd]

@davidhellmann David,, you have always been able to use Blackfire for free, at a very useful level. It just costs if you want full enterprise/collab features: https://support.blackfire.io/en/articles/888235-can-i-try-blackfire-for-free

However, see what I say to Werner. I tracked this down using just the Network trace in the Chrome DevTools.

@wsydney76 Werner, by your case, you are helping me remember what happened when I tracked down this long delay, where a remote server was involved. I'm fuzzy on it because I was actually deep in another issue at the time, and this sort of thing appeared to be that occasional side annoyance which a local fresh restart had always been able to send away.

I believe what happened was something like this, which may hint to David, possibly @lukeyouell :

• I had the environmentals for the Craft server mis-set due to some inproper copying of .env.
• I think the two in question were DEFAULT_SITE_URL as you mention, but also the likely critical one ASSET_BASE_URL
• if I try to piece together what had happened, the ASSET_BASE_URL was fully pathed, and pointed to the wrong server, not the one where the assets would actually be provided.
• the result then might be that Craft hung on trying to generate/provide the asset, rather than returning a 404 or 500.
• this hang led to the full browser access timeout, which is in the one minute range

Conclusion: have a look at what you are or are not setting for ASSET_BASE_URL.

I still think there have been quirky issues with Vagrant or the rsync reload of Craft parts which could cause a similar delay. These may actually be gone by now, as I don't recall seeing that occasional hang in recent months, and were pretty clearly not involved here.

If this purely local VM variant of the problem does happen for me again (will try to excite it), will be sure to try states of devMode...

Viel Glück, natürlich

[davidhellmann]

Hm still slow. Someone else an other idea?

[narration-sd]

@davidhellmann , I wonder if you noticed that in your screenshot here, its showing that 2.7 seconds of your 3.3 sec response time is database queries?

This is of course way out of line, but what may be a reason why? A thought or two, but you have to find it:

• Craft (Yii) devMode logging goes into RAM memory. before in the longer term it's flushed. So is it possible you haven't got enough available memory on the system where the problem occurs, maybe just marginal enougy so that devMode is enough added use to get into problems?

Even if you've set PHP limits high enough, if there's not enough actual memory available, the operating system will go into swapping, which would then be likely to load and run the database separately from php-fpm, and then back...lots of delay for each query. This could easily happen on an underprovisioned droplet server, or similarly on a local VM running for example on one of those short-of-memory Mac laptops, or...

• well, I thought about DNS/path environmentals again, but this seems not likely now, even though path to the DB could be involved, because even if there were a delay, it would be resolved once. Still, if resolution were long....but again, this is not so likely, and shouldn't occur on more than an initial Craft activity.

Tracking this down will require actually measuring something that relates to the delay. Brandon's suggestion of Blackfire would be very good to isolate the area, but your Yii Debug Toolbar may have done that.

You can for the memory question run top or htop to watch consumption during these long responses, setting refresh to 1 sec for instance.

Once the database issue is tamed, the remaining 0.5 sec response would be in the reasonable range for a not-too-powerful Craft server and not much caching.

[jasonmccallister]

@davidhellmann I am seeing a similar issue with another installation. Can you open the database queries tab in debug bar (like @narration-sd mentioned) and identify the query that is taking a long time?

The installation that I am seeing similar behavior on is having an issue querying the sessions table, which is taking 90 percent of the request time. Can you confirm which database query is taking the most time?

[davidhellmann]

@jasonmccallister can test it tomorrow. on my mac at home is no problem.

[davidhellmann]

@jasonmccallister delay, sorry.

There are a lot queries like this:

[davidhellmann]

        'db' => function() {
            $config = craft\helpers\App::dbConfig();
            $config['enableSchemaCache'] = true;
            $config['schemaCacheDuration'] = 60; // 1 day
            return Craft::createObject($config);
        },

added this to the app.php and it's faaaaaaaaaaaaaaasssst now!

[jasonmccallister]

Thank you for the update @davidhellmann! Can you by chance post a similar screenshot as your original post with a timeline? I'm very curious about the new response time :)

[brandonkelly]

Ahh, I’m guessing this is related to #4037. Yii’s upsert() command in particular can take a very long time if you have a lot of databases installed on the same MySQL server, which is why we started enabling schema caching in the first place (but only when dev mode is disabled by default).

Glad you got it sorted!

[davidhellmann]

@jasonmccallister sure:

It's so fast now! Awesome, we've added it to all projects. Speak something against this @brandonkelly ?

[ccsantos23]

@davidhellmann I've gotten the grindingly-slow CP when I was trying an ubuntu/xenial64 box for my vagrant vm. Even setting the schemaCacheDuration barely helped. When I switched back today to a bento/ubuntu-16.04 box, then CP was blazing fast again (without any changes to app.php).
[edit: ok, "blazing" might be relative.]
[edit2: When I reverted to bento/ubuntu-16.04 version 201812.27.0, it did become blazing fast. Newer versions of the box slowed it down.]

[jamesmacwhite]

If there is a performance drop between vagrant boxes, could it be perhaps related to the patches for Intel CPUs related to Meltdown and Spectre? I don't know the exact timelines of when these might have been applied, but a performance drop could be tied to it. It has been known to hit Intel CPU quite hard on performance, so you could be suffering from that? I'd imagine old Vagrant boxes will use an older kernel that might not have the Intel patches, where as the newer boxes with a newer kernel would.

[narration-sd]

Just an anecdote or two...

• As far as my some years with Vagrant running on VirtualBox, later versions of the _Homestead_ 'box' have exhibited the slow-starting or sometimes slow-running issue less and less. I did happen to see it in a minor way yesterday, though, after a long session with several boxes running, and reloading a number of times. So I kind of doubt the Meltdown/Spectre patches have to do with it, and don't see any evidence either on an always-snappy if small DigitalOcean box that's always up-to-date with them.
• A full stop-start rather than reload, then, will often reset the kinds of delays I see; I believe a reboot of the offending laptop will aways do so
• I just learned 'something' in the last week or so which may or may not be related. I tend to use Chromes as browsers for Craft/ES6 platforms/Vagrant work, especially Canary. I found an annoying delay on Win10 which has seemed to get worse and worse on those alone can be fixed. Go to the Settings | Proxy page (via search) and turn off everything there -- in particular 'automatically detect settings' per this of many articles. Then any Chrome goes back to being fast-er.
• in relation, I've tried to troubleshoot a few times seriously when Vagrant/Craft got slow. This was inconclusive at best, but what I seemed to find, as have mentioned here I think before, is that what seemed to happen is that Windows lost track of the hosts file's definiton of my local servers, and went out for some kind of long-winded external DNS. The result may have been the problem, in retrospect -- there's a kind of 'non-definitive' answer DNS will give when it actually fails, a best guess at what 'might' be usable. Using this bogus or at least non-HTTP-answering IP address, the browser will hang until it times out, which is in the length spectrum of the seeming Vagrant-related blockings when they happen. What Vagrant itself may have to do with this, since the browser's on the Win10 host, is not a cloud I've penetrated...

Best fortune, and while it's gone down to a not-so-interrupting and only very occasional annoyance here, it would be great to find a true fix, one that works as well as turning of the Win10 proxy indetermination does for Chrom(ium) itself.

[narration-sd]

...late-added to first point, for those reading only by email: 'So I kind of doubt the Meltdown/Spectre patches have to do with it, and don't see any evidence either on an always-snappy if small DigitalOcean box that's always up-to-date with them.'

And bonus here for those via git -- meant to mention an article for the Chrome/Win10 proxy hunting thing, so here is one which weaves around unnecessarily through Chrome itself, but gets to the point: https://techdows.com/2018/08/fix-chrome-downloading-proxy-script-issue.html

[peimansh]

@davidhellmann I am experiencing the same issue but with some differences.
In my case, it's not tied to devMode but exactly the same symptoms.
It is suuper fast in my local development but it is slooow when running in production and exactly the same issues. (icons loading veery slow and high response times for /cpresources/*) and I am not using Vagrant, Vbox etc.. I'm using docker-compose.

I noticed that changing @web alias, makes the same situation in my local development and they become slow, So my guess is there could be a connection.

Btw, I tried using your solution but I am getting an error:

Can you send your app.php ?

This is mine and is not working:
return [ 'modules' => [ 'my-module' => \modules\Module::class, ], //'bootstrap' => ['my-module'], 'db' => function() { $config = craft\helpers\App::dbConfig(); $config['enableSchemaCache'] = true; $config['schemaCacheDuration'] = 60; // 1 day return Craft::createObject($config); }, ];

[jasonmccallister]

This knowledge base article does a great job of describing what devMode does. Specifically this note:

As you can probably tell, Dev Mode is definitely not suited for production environments

Local Development
If you are experiencing these slowdowns locally and are mounting sites with something like Vagrant, Docker, or Nitro v2 (Docker based) there will be some delays with file loading and syncing. Those are getting better but kind of expected for local development.

Staging or Production
If you are experiencing this in a production or staging environment, I would highly recommend setting up Blackfire and profiling those slow requests to determine what specifically is causing the delays. I would also make sure any custom modules are not performing expensive setups during the init due to how modules are loaded in Yii meaning if the module is bootstrapped it will be initialized on every request and can cause slow loads.