Valet-plus: Some operations cause Valet+ to hang completely

[SanderAtom]

It's hard to pinpoint what is going on, but after certain actions involving Ajax calls (I think), Valet+ becomes completely unresponsive. There is minor php-fpm usage when I open a page, but very short then nothing and eventually it times out with a 504 Gateway Timeout.
Very rarely it started working again after a few minutes without doing anything. Mostly I have to restart php and it works again. Sometimes it hangs again on the same Ajax call immediately after, but mostly the Ajax call is handled quickly and without problems after restarting php. Until I open some other pages, save something and try the Ajax call again, it breaks. This happens with a Drupal 8 project.

~/.valet/Log/nginx-error.log:

2017/08/22 10:15:50 [error] 5708#0: *5648 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://unix:/Users/sander/.valet/valet.sock", host: "project.dev", referrer: "http://project.dev/"

Nothing in ~/.valet/Log/php.log

I see this in /var/log/system.log, but I think it's just a warning because php is always running:

Aug 22 10:07:14 nm2d-02345 com.apple.xpc.launchd[1] (homebrew.mxcl.php70): This service is defined to be constantly running and is inherently inefficient.

Does anyone have any idea which php/nginx settings I can try to solve this?

[SanderAtom]

This happens now every time Drupal has to create new images based on image styles. When I change the image style and load the page, php hangs when trying to create the new images. I have to go to the image link directly to get the image to be created. After they are created, the page loads find and quick. Not sure if this is because these scripts are executed multiple times at the same time.

[Tristan-N]

@SanderAtom This also occurs on Magento 2 installations, so it's not a Drupal only issue.

[SanderAtom]

Ye but less often than certain drupal actions. Also only restarting php resolves this issue, restarting nginx has no effect.

[SanderAtom]

@Tristan-N do you have anymore information about what kind of actions cause this complete crash for you in Magento2?
When this happens it won't even get to server.php so might have something to do with the valet.sock? Not sure though how that works.

[SanderAtom]

I see this in /usr/local/var/log/php-fpm.log:

[04-Oct-2017 13:40:17] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
....
[04-Oct-2017 13:50:58] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 5 total children
[04-Oct-2017 13:50:59] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 6 total children
[04-Oct-2017 13:51:00] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 7 total children
[04-Oct-2017 13:51:01] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 8 total children

etc, etc.. Whatever I keep raising, the other settings will max out again. So if I raise pm.max_children it'll warn about being busy. When I then up the min/max and start_server numbers, eventually I'll run into pm.max_children problems again.

It seems specifically bad when there are a ton of requests, like unmerged css/js + image files etc when those calls might have to generate files as well.

I kept raising the numbers, now it reads:

pm.max_children = 50
pm.start_servers = 20
pm.min_spare_servers = 20
pm.max_spare_servers = 40

I get rid of the warnings in php-fpm.log but it'll still break at some point without any error and give 504's until php is restarted.

[Tristan-N]

@SanderAtom I don't have real examples. The problem is I am not sure what is happening. When I, for example, want to create a backup from the backend (instead of CLI) it takes up to like 20 seconds before getting the 504 time-out. However, when I just keep waiting suddenly a backup-file is created in the folder it should be. So it seems the processes are still running but 504 is shown in browser.

Funny enough, I don't have anything in the errlogs either.

[SanderAtom]

Ok, but when you get this 504, can you still open other pages? Or you have to restart php before you can do anything on the website?

[Tristan-N]

I do have to restart php to be able to open pages again (I use valet restart for no particular reason, other then lazyness)

[SanderAtom]

Using higher numbers for children and start/min/max servers does lower the rate at which php crashes, but I keep getting warnings to increase them. Are there related settings we can try, something like buffer size or something?

[SanderAtom]

I see this in php.log, not sure if it's related..

[24-Oct-2017 09:16:16 Europe/Amsterdam] PHP Warning:  apcu_add(): Unable to allocate memory for pool. in /Users/sander/.composer/vendor/weprovide/valet-plus/server.php on line 61
[24-Oct-2017 09:16:16 Europe/Amsterdam] PHP Stack trace:
[24-Oct-2017 09:16:16 Europe/Amsterdam] PHP   1. {main}() /Users/sander/.composer/vendor/weprovide/valet-plus/server.php:0
[24-Oct-2017 09:16:16 Europe/Amsterdam] PHP   2. apcu_add() /Users/sander/.composer/vendor/weprovide/valet-plus/server.php:61

[Neodork]

@SanderAtom and all. Noticed same behaviour (504 gateway timeouts, on pages doing a lot of requests) when installing weprovide/valet-plus. laravel/valet v2.0 or higher does not cause this behaviour (if you install it without PHP installed). Spend a lot of time getting to the root of the problem here, but eventually a college of mine cracked the problem. He has no Github account so I will post on his behalf, credits to him. But the likes are mine!

Cause: Opcache conflicts with APC(U) which causes PHP-FPM children to stay open after a request. Which eventually causes your machine to run out of resources, one way or another.

In /usr/local/etc/php/{php-version}/conf.d/z-performance.ini

Change:

apc.enabled=1

To:

apc.enabled=0

End Result:

...

[apcu]
apc.enabled=0
apc.shm_size=512M
apc.ttl=7200
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.enable_cli=1

...

Then restart your PHP service. But to be sure, just restart everything: valet restart

NOTE: Fully uninstalling the APC PHP extension breaks Valet so no need to try that. Instead disable APC following the instructions above.

[SanderAtom]

@Neodork thanks for that information, I'll look into it. A quick scan reveals this thread about PHP-FPM crashing or processes lingering due to APCu turned on, so that seems promising. Although I do remember turning off features before, including APCu, without any direct improvements but I can't say for sure. I might setup a test situation where this crashing occures, without a full webshop system and all the caching layers around it, and see if simply turning off APCu solves the problem.

[SanderAtom]

I'm fairly certain this is the bug we are experiencing. Since turning off APCu 2 weeks ago, I have not had any php-fpm/socket etc hanging like it would before.

[Neodork]

@SanderAtom Thanks for getting back to this. Since you're confirming this solves the problem I will continue to PR this today. Thanks!

[SanderAtom]

Well just turning it off isn't really a fix for the problem. Do you have more information about this conflict between OPcache and APCu? Is it something other people are working on and we can add the fix for it later?
On the other hand, what's the benefit of having APCu and is it important enough to wait for the fix or resolve the conflict? Otherwise we just need to completely remove APCu.. or OPcache .. ?
Turning it off for now is ok to at least prevent new people from experiencing this conflict, but I feel we need a better solution long term.

[Neodork]

I agree that we should look deeper into the problem. After searching I did not come up with any obvious conflict reports.

I can see the need for APCu and OPCache if the two can operate in harmony. Since APC is deprecated in PHP 5.6+ and replaced with APCu and OPCache. As far as the difference go:

• APC (PHP 5.5<) opcode cache and data store
• APCu data store only
• OPCode opcode cache only

Although I haven't done a lot of research I did find an article mentioning that the v4 of APCu is not compatible with PHP 7 or higher. Doing a quick search in the code however reveals that APCu is installed according to the PHP version so it shouldn't be a version conflict either. However I haven't actually checked if the correct package is linked when switching PHP versions.

if($this->brew->installed($currentVersion.'-'.$extension)) {
// Intl breaks often when switching versions
    if($extension === 'intl') {
        $this->cli->runAsUser('brew upgrade '. $currentVersion . '-' . $extension);
    }

    $this->cli->runAsUser('brew link '. $currentVersion . '-' . $extension);
    info('['.$currentVersion.'] '.$extension.' already installed');
} else {
    $this->brew->ensureInstalled($currentVersion.'-'.$extension, [], $this->taps);
}

The above code will never refresh links when switching between PHP versions of already installed php versions since APCu is "installed" already.Which is true but might still contain links for an old APCu version. Then again I will need to sink more time into this.

Edited: Code does run links if installed.

[matt-bailey]

I don't know if this is related, but I'm also struggling with Magento 2 and valet+ (and the original valet for that matter) hanging completely.

When I visit a site for the first time some of the static content (css, js, images etc) will start to load, but then everything will start 504ing. If I then try to reload the page I just get a 504 gateway timeout message on the page.

I can also see that even though the page has stopped loading the php-fpm process is still running at well over 200%. It's not until I restart php that the cpu usage drops. I can then reload the page again, but at some point the static content starts 504ing, and we're back where we started...

I tried the fix by @Neodork to disable APC, but it doesn't seem to have had any effect.

[matt-bailey]

Looking into this a bit more I discovered there is an additional file that takes precedence over the settings in z-performance.ini.

So if you set apc.enabled=0 in /usr/local/etc/php/{php-version}/conf.d/ext-apcu.ini instead, then it will work.

That issue solved, I now have to figure out some other stuff (now all my static assets are 502ing rather than 504ing) - it doesn't matter how easy/simple tools like valet purport to be, nothing is ever straightforward... ;)

[rglepper]

Hi, I was getting errors with APCu after switching the PHP version from 7.0 to 7.1 with valet use following this article seems to have fixed the issue. Probably not a solution, but thought I'd mention it.

[D19cannon]

Or just fucking delete the entire z-performance.ini from php folder... would've save me 3h of my life... it not more...

Thanks for the solutions any way. Deleting the file worked for me 🥇 👍

[EHLOVader]

This appears to still be causing issues with normal use.
Not sure if it is still disabled but after restarting i get undefined apcu_fetch errors and had to resort to adding this into my server.php to prevent them from occurring and bypass the apcu caches.

if(!function_exists('apcu_fetch')){
 function apcu_fetch() {
        return false;
 }
}

if(!function_exists('apcu_add')){
 function apcu_add() {
 }
}

Was this supposed to have a resolution? if not why are there still apcu fetch and add calls in the server.php if it isn't enabled?

[Neodork]

Dear Contributor,

With the release of Valet+ 2.0.0 the Valet+ development team and contributors have implemented the contribution guidelines. The contribution guidelines aim to improve the support and workflow of Valet+. For further information why this was implemented please check https://github.com/weprovide/valet-plus/issues/393.

We're closing this ticket because it does not follow the correct format. If you feel your ticket is still relevant we encourage you to re-open your ticket using the correct format.

Kind regards,
The Valet+ team

[Leland]

@Neodork this issue is still in use, see the link on https://github.com/weprovide/valet-plus/wiki/Drivers