Cms: Deleting stale template caches unable to complete

[lukebailey]

Description

On a clean build of latest Craft 3 RC (13) the 'Deleting stale template caches' job cannot run and gets stuck. Running ./craft queue/run throws the following exception:

PHP Warning 'yiibase\ErrorException' with message 'unlink(/home/vagrant/code/craft/storage/runtime/mutex/d2cd3eeb43126a9da9ebb748dc3680d4.lock): Text file busy'

Steps to reproduce

1. Clean Craft installation (running on homestead)
2. Deleting stale template caches job fails to run, from admin and console

Additional info

• Craft version: Craft 3 RC 13
• PHP version: 7.2.2
• Database driver & version: mysql
• Plugins & versions: none

[lukebailey]

As a follow up this seems to always hang on the first job Craft runs after install

[angrybrad]

My guess is you're using the default homestead filesystem, which is pretty sluggish, and that's causing a deadlock/race condition. Try swapping it over to NFS, which is faster: http://tech.osteel.me/posts/2016/01/19/how-to-enable-nfs-on-laravel-homestead.html

[webkp]

I had the same problem yesterday and did a little more research. I assume @lukebailey also uses Vagrant on Windows with Virtualbox as provider. Virtualbox shares behave like the Windows filesystem, but the Guest OS thinks it is a Unix filesystem. The problem should therefore occur for all those who run Craft3 in a Virtualbox container and use Virtualbox shares, whether Vagrant, Docker, etc.

The problem occurs because in the file vendor/yiisoft/yii2/mutex/FileMutex.php on line 161 an unlink() is made, although the file is still opened by the system and is only closed on line 163.
https://github.com/yiisoft/yii2/blob/959cb985e3ab9f72e2aaf7fcf4a170ef7591a608/framework/mutex/FileMutex.php#L152-L164

While this is not a problem with Unix, it does not work with Windows. However, the check on line 152 obviously does not notice that there is a Windows host at the other end of the share. If you copy the Windows block (lines 155-157) down (lines 161-163), the problem no longer occurs.

If you want to unlink() first under Unix, this would be a possible alternative. (First unlink() with suppression of a possible error message, then check if file still exists (always the case with Windows) and in this case try unlink() again which should work as the file is closed now.)

        if (DIRECTORY_SEPARATOR === '\\') {
            // Under windows it's not possible to delete a file opened via fopen (either by own or other process).
            // That's why we must first unlock and close the handle and then *try* to delete the lock file.
            flock($this->_files[$name], LOCK_UN);
            fclose($this->_files[$name]);
            @unlink($this->getLockFilePath($name));
        } else {
            // Under unix it's possible to delete a file opened via fopen (either by own or other process).
            // That's why we must unlink (the currently locked) lock file first and then unlock and close the handle.
            @unlink($this->getLockFilePath($name));
            flock($this->_files[$name], LOCK_UN);
            fclose($this->_files[$name]);
            // still there on Win-Shares
            if(is_file($this->getLockFilePath($name))) {
                unlink($this->getLockFilePath($name));
            }
        }

Is there an easy way to override this behavior without tinkering with the yii files or would you adjust the function even via Craft's cms/src/mutex/FileMutex.php? Thank you for considering.

[brandonkelly]

@webkp Do you have any idea if there’s a way for PHP to detect that it’s a Windows host in this context?

[webkp]

@brandonkelly Not natively as far as I know. I saw some do exec('grep ... against mount points but find that somewhat bulky.

However, I can tell you that this call has exactly the opposite effect depending on the underlying filesystem.

$f = fopen('/var/www/test/storage/runtime/temp/test.tmp','w+');
var_dump(@unlink('/var/www/test/storage/runtime/temp/test.tmp'));
fclose($f);
var_dump(@unlink('/var/www/test/storage/runtime/temp/test.tmp'));

Result for Win-based FS will be
bool(false) // still open, would throw E_WARNING about "text file busy" without @
bool(true) // all set

and for Unix
bool(true) // good to go; doesn't matter if file is not closed
bool(false) //already deleted, would throw E_WARNING "No such file or directory" without @

This would need to be done in a Craft directory like /storage/runtime/temp/ as it's not about the host but the filesystem of current mount point the website is running on. So for /tmp all is good anyway even on Windows host because /tmp is not mounted to Windows.

[adrienne]

@brandonkelly - I also have this issue with new entries in a Vagrant-on-Windows environment, so this is a big deal. Wonder if you could just add a possible variable to .env that would let us override the behavior? Just WINDOWS_FS="true" or something?

[brandonkelly]

I just submitted an issue to Yii about this: yiisoft/yii2#16603

And a PR that fixes it: yiisoft/yii2#16604

[brandonkelly]

The PR has been merged in, so this will be fixed in Yii 2.0.16. At that point you could create a WINDOWS_FS=1 variable in your .env file on the environment(s) that are affected by this issue, and use it to configure Craft’s mutex component in config/app.php:

return [
    'components' => [
        'mutex' => [
            'isWindows' => (bool)getenv('WINDOWS_FS') ?: null,
        ],
    ],
];

[zauni]

Until Yii 2.0.16 is out, a workaround is to use a path, which is not shared with the windows host.

config/app.php:

return [
    'components' => [
        'mutex' => function() {
            $config = \craft\helpers\App::mutexConfig();
            $config['mutexPath'] = '/tmp';

            return \Craft::createObject($config);
        },
    ]
];

[brandonkelly]

If you are using MySQL, you could use the MysqlMutex class instead.

return [
    'components' => [
        'mutex' => \yii\mutex\MysqlMutex::class,
    ],
];

There’s also a PgsqlMutex class, but it doesn’t support timeouts.

[zauni]

Are there plans to update Yii to the now released 2.0.16 version? (https://github.com/yiisoft/yii2/releases/tag/2.0.16)

[brandonkelly]

@zauni Yep, on my list for the next release.

[jamesmacwhite]

@brandonkelly I'm experiencing various mutex issues on Windows and looking through this thread, the issue should be fixed, however when applying the Windows environment variable added to Yii, I get this error:

The configuration for the "mutex" component must contain a "class" element.

The example provided suggests you shouldn't need to specify anything else, but do you now have to explicitly define a mutex class value?

Using the MySQL mutex class fixed some of the issues, but now I'm finding issues with other areas like downloading an asset throws this:

```
2019-03-09 16:12:49 [-][43903][43d4c7df2408cf1349774162ac80c33f][error][yiibase\ErrorException:2] yiibase\ErrorException: unlink(/app/storage/runtime/temp/asset016453e014c206.43823587.jpg): Text file busy in /app/vendor/yiisoft/yii2/helpers/BaseFileHelper.php:407
Stack trace:

0 /app/vendor/yiisoft/yii2/helpers/BaseFileHelper.php(407): ::unlink()

1 /app/vendor/craftcms/cms/src/controllers/AssetsController.php(812): yii\helpers\BaseFileHelper::unlink()

2 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): craft\controllers\AssetsController->actionDownloadAsset()

3 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): ::call_user_func_array:{/app/vendor/yiisoft/yii2/base/InlineAction.php:57}()

4 /app/vendor/yiisoft/yii2/base/Controller.php(157): yiibase\InlineAction->runWithParams()

5 /app/vendor/craftcms/cms/src/web/Controller.php(109): craft\controllers\AssetsController->runAction()

6 /app/vendor/yiisoft/yii2/base/Module.php(528): craft\controllers\AssetsController->runAction()

7 /app/vendor/craftcms/cms/src/web/Application.php(297): craft\web\Application->runAction()

8 /app/vendor/craftcms/cms/src/web/Application.php(561): craft\web\Application->runAction()

9 /app/vendor/craftcms/cms/src/web/Application.php(281): craft\web\Application->_processActionRequest()

10 /app/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest()

11 /app/web/index.php(21): craft\web\Application->run()

12 {main}```

[brandonkelly]

The example provided suggests you shouldn't need to specify anything else, but do you now have to explicitly define a mutex class value?

@jamesmacwhite If you are referring to https://github.com/craftcms/cms/issues/2526#issuecomment-414674762, that should still be all that’s required to use MysqlMutex.

Can you post your config code?

[jamesmacwhite]

@brandonkelly This is what I tried first, but the I get an error about no class being specified, so I have to define one within the mutex array.

return [
    // Workaround for initial Craft 3 DB migration on Windows (dead lock mutex issue)
    // https://github.com/craftcms/cms/issues/2526
    'components' => [
        'mutex' => [
            'isWindows' => filter_var(getenv('WINDOWS_FS'), FILTER_VALIDATE_BOOLEAN),
        ],
    ]
];

An Error occurred while handling another error:
yii\base\InvalidConfigException: Unknown component ID: request in /app/vendor/yiisoft/yii2/di/ServiceLocator.php:139
Stack trace:
#0 /app/vendor/yiisoft/yii2/base/Module.php(742): yii\di\ServiceLocator->get('request', true)
#1 /app/vendor/craftcms/cms/src/web/Application.php(348): yii\base\Module->get('request', true)
#2 /app/vendor/yiisoft/yii2/web/Application.php(160): craft\web\Application->get('request')
#3 /app/vendor/yiisoft/yii2/base/Component.php(139): yii\web\Application->getRequest()
#4 /app/vendor/yiisoft/yii2/di/ServiceLocator.php(77): yii\base\Component->__get('request')
#5 /app/vendor/yiisoft/yii2/web/ErrorHandler.php(502): yii\di\ServiceLocator->__get('request')
#6 /app/vendor/yiisoft/yii2/web/ErrorHandler.php(115): yii\web\ErrorHandler->shouldRenderSimpleHtml()
#7 /app/vendor/craftcms/cms/src/web/ErrorHandler.php(132): yii\web\ErrorHandler->renderException(Object(yii\base\InvalidConfigException))
#8 /app/vendor/yiisoft/yii2/base/ErrorHandler.php(111): craft\web\ErrorHandler->renderException(Object(yii\base\InvalidConfigException))
#9 /app/vendor/craftcms/cms/src/web/ErrorHandler.php(63): yii\base\ErrorHandler->handleException(Object(yii\base\InvalidConfigException))
#10 [internal function]: craft\web\ErrorHandler->handleException(Object(yii\base\InvalidConfigException))
#11 {main}
Previous exception:
yii\base\InvalidConfigException: The configuration for the "mutex" component must contain a "class" element. in /app/vendor/yiisoft/yii2/di/ServiceLocator.php:205
Stack trace:
#0 /app/vendor/yiisoft/yii2/di/ServiceLocator.php(261): yii\di\ServiceLocator->set('mutex', Array)
#1 /app/vendor/yiisoft/yii2/base/Component.php(180): yii\di\ServiceLocator->setComponents(Array)
#2 /app/vendor/yiisoft/yii2/BaseYii.php(546): yii\base\Component->__set('components', Array)
#3 /app/vendor/yiisoft/yii2/base/BaseObject.php(107): yii\BaseYii::configure(Object(craft\web\Application), Array)
#4 /app/vendor/yiisoft/yii2/base/Application.php(206): yii\base\BaseObject->__construct(Array)
#5 /app/vendor/craftcms/cms/src/web/Application.php(100): yii\base\Application->__construct(Array)
#6 [internal function]: craft\web\Application->__construct(Array)
#7 /app/vendor/yiisoft/yii2/di/Container.php(384): ReflectionClass->newInstanceArgs(Array)
#8 /app/vendor/yiisoft/yii2/di/Container.php(156): yii\di\Container->build('craft\\web\\Appli...', Array, Array)
#9 /app/vendor/yiisoft/yii2/BaseYii.php(349): yii\di\Container->get('craft\\web\\Appli...', Array, Array)
#10 /app/vendor/craftcms/cms/bootstrap/bootstrap.php(255): yii\BaseYii::createObject(Array)
#11 /app/vendor/craftcms/cms/bootstrap/web.php(42): require('/app/vendor/cra...')
#12 /app/web/index.php(20): require('/app/vendor/cra...')
#13 {main}

Originally I defined the MySQL Mutex class directly as without this on Windows, the Craft 2 to 3 DB migration won't start, because of file locking. However I still seem to be getting "text file is busy" messages for certain actions still, so it doesn't full resolve the issue, so then I looked at the isWindows parameter, but now that doesn't seem to work without a specific class being specified now. Is that the intended behaviour?

[brandonkelly]

@jamesmacwhite If you just want to customize the default mutex component (which uses yii\mutex\FileMutex, then you can do so like this:

'mutex' => function() {
    $config = craft\helpers\App::mutexConfig();

    // make changes to the $config array here...

    return Craft::createObject($config);
},

If you want to use a different mutex class, then you must supply a complete component config array, which would at the least require defining the 'class' key.

'mutex' => yii\mutex\MysqlMutex::class,

// or:

'mutex' => [
    'class' => yii\mutex\MysqlMutex::class,
    // ...
];

[jamesmacwhite]

@brandonkelly Thanks for clarifying. In my case I had originally switched the mutex to MySQL entirely. However I now want to test the WINDOWS_FS environment override to see if that helps with other file locking issues happening on my local dev on Windows.

Thanks for providing a couple of examples related to the mutex config. I now understand the process better!