Magento2: QUESTION: Strange static-content:deploy error on 2.1.3

[avra911]

Preconditions

1. Install composer 1.3.0 or higher on a server which has ssh access.
2. Install and check out Magento 2.1.3.
3. (optional) Install capistrano-magento2 5.9.

Steps to reproduce

1. run ssh [email protected] /usr/bin/env php -f /path_to_site/bin/magento -- setup:static-content:deploy en_US | stdbuf -o0 tr -d . from a local computer or from a different server OR configure capistrano deploy and run cap production deploy.

Expected result

1. Command/Deploy runs fine.

Actual result

1. Part of the log ( I added -vv for debugging purpose to static content generation command, so the output has more details, not only errors count )

Processing file 'requirejs/requirejs' for area 'frontend', theme 'NAMESPACE/nec', locale 'en_US'
    Deploying the file to 'frontend/NAMESPACE/nec/en_US/requirejs/requireminjs'
Processing file 'requirejs/domReadyjs' for area 'frontend', theme 'NAMESPACE/nec', locale 'en_US'
    Deploying the file to 'frontend/NAMESPACE/nec/en_US/requirejs/domReadyminjs'
Processing file 'js-translationjson' for area 'frontend', theme 'NAMESPACE/nec', locale 'en_US'
    Deploying the file to 'frontend/NAMESPACE/nec/en_US/js-translationjson'
#0 /var/www/magento/releases/20170221151910/vendor/colinmollenhour/cache-backend-redis/Cm/Cache/Backend/Redisphp(313): Credis_Client->__call('hget', Array)
#1 /var/www/magento/releases/20170221151910/vendor/magento/zendframework1/library/Zend/Cache/Corephp(390): Cm_Cache_Backend_Redis->save('a:105:{i:123;s:', '537_TRANSLATE_E', Array, false)
#2 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Corephp(74): Zend_Cache_Core->save('a:105:{i:123;s:', '537_TRANSLATE_E', Array, false, 8)
#3 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Adapter/Zendphp(47): Magento\Framework\Cache\Core->save('a:105:{i:123;s:', 'TRANSLATE_EN_US', Array, false)
#4 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/Barephp(75): Magento\Framework\Cache\Frontend\Adapter\Zend->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
#5 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/TagScopephp(49): Magento\Framework\Cache\Frontend\Decorator\Bare->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
#6 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/Barephp(75): Magento\Framework\Cache\Frontend\Decorator\TagScope->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
#7 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/Barephp(75): Magento\Framework\Cache\Frontend\Decorator\Bare->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
#8 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Cache/Type/AccessProxyphp(85): Magento\Framework\Cache\Frontend\Decorator\Bare->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
#9 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/Barephp(75): Magento\Framework\App\Cache\Type\AccessProxy->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
#10 /var/www/magento/releases/20170221151910/vendor/magento/framework/Cache/Frontend/Decorator/TagScopephp(49): Magento\Framework\Cache\Frontend\Decorator\Bare->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
#11 /var/www/magento/releases/20170221151910/vendor/magento/framework/Translatephp(489): Magento\Framework\Cache\Frontend\Decorator\TagScope->save('a:105:{i:123;s:', 'translate_en_US', Array, false)
#12 /var/www/magento/releases/20170221151910/vendor/magento/framework/Translatephp(185): Magento\Framework\Translate->_saveCache()
#13 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Areaphp(240): Magento\Framework\Translate->loadData(NULL, false)
#14 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Areaphp(211): Magento\Framework\App\Area->_initTranslate()
#15 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Areaphp(138): Magento\Framework\App\Area->_loadPart('translate')
#16 /var/www/magento/releases/20170221151910/vendor/magento/module-translation/Model/Json/PreProcessorphp(84): Magento\Framework\App\Area->load('translate')
#17 /var/www/magento/releases/20170221151910/vendor/magento/framework/View/Asset/PreProcessor/Poolphp(74): Magento\Translation\Model\Json\PreProcessor->process(Object(Magento\Framework\View\Asset\PreProcessor\Chain))
#18 /var/www/magento/releases/20170221151910/vendor/magento/framework/View/Asset/Sourcephp(152): Magento\Framework\View\Asset\PreProcessor\Pool->process(Object(Magento\Framework\View\Asset\PreProcessor\Chain))
#19 /var/www/magento/releases/20170221151910/vendor/magento/framework/View/Asset/Sourcephp(105): Magento\Framework\View\Asset\Source->preProcess(Object(Magento\Framework\View\Asset\File))
#20 /var/www/magento/releases/20170221151910/vendor/magento/framework/View/Asset/Filephp(150): Magento\Framework\View\Asset\Source->getFile(Object(Magento\Framework\View\Asset\File))
#21 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/View/Asset/Publisherphp(73): Magento\Framework\View\Asset\File->getSourceFile()
#22 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/View/Asset/Publisherphp(61): Magento\Framework\App\View\Asset\Publisher->publishAsset(Object(Magento\Framework\View\Asset\File))
#23 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/Deploy/LocaleDeployphp(398): Magento\Framework\App\View\Asset\Publisher->publish(Object(Magento\Framework\View\Asset\File))
#24 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/Deploy/LocaleDeployphp(235): Magento\Deploy\Model\Deploy\LocaleDeploy->deployFile('js-translation', 'frontend', 'Magento/blank', 'en_US', NULL)
#25 [internal function]: Magento\Deploy\Model\Deploy\LocaleDeploy->deploy('frontend', 'Magento/blank', 'en_US')
#26 /var/www/magento/releases/20170221151910/vendor/magento/framework/App/Statephp(171): call_user_func_array(Array, Array)
#27 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/DeployManagerphp(175): Magento\Framework\App\State->emulateAreaCode('frontend', Array, Array)
#28 [internal function]: Magento\Deploy\Model\DeployManager->Magento\Deploy\Model\{closure}(Object(Magento\Deploy\Model\Process))
#29 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/Processphp(53): call_user_func(Object(Closure), Object(Magento\Deploy\Model\Process))
#30 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/ProcessManagerphp(55): Magento\Deploy\Model\Process->run()
#31 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/ProcessQueueManagerphp(152): Magento\Deploy\Model\ProcessManager->fork(Object(Closure))
#32 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/ProcessQueueManagerphp(130): Magento\Deploy\Model\ProcessQueueManager->fork(Object(Magento\Deploy\Model\ProcessTask))
#33 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/ProcessQueueManagerphp(88): Magento\Deploy\Model\ProcessQueueManager->internalQueueProcess(Array, Array)
#34 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/DeployManagerphp(187): Magento\Deploy\Model\ProcessQueueManager->process()
#35 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Model/DeployManagerphp(122): Magento\Deploy\Model\DeployManager->runInParallel(Object(Magento\Deploy\Model\DeployStrategyProvider))
#36 /var/www/magento/releases/20170221151910/vendor/magento/module-deploy/Console/Command/DeployStaticContentCommandphp(386): Magento\Deploy\Model\DeployManager->deploy()
#37 /var/www/magento/releases/20170221151910/vendor/symfony/console/Symfony/Component/Console/Command/Commandphp(257): Magento\Deploy\Console\Command\DeployStaticContentCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#38 /var/www/magento/releases/20170221151910/vendor/symfony/console/Symfony/Component/Console/Applicationphp(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#39 /var/www/magento/releases/20170221151910/vendor/symfony/console/Symfony/Component/Console/Applicationphp(195): Symfony\Component\Console\Application->doRunCommand(Object(Magento\Deploy\Console\Command\DeployStaticContentCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#40 /var/www/magento/releases/20170221151910/vendor/magento/framework/Console/Cliphp(96): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#41 /var/www/magento/releases/20170221151910/vendor/symfony/console/Symfony/Component/Console/Applicationphp(126): Magento\Framework\Console\Cli->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#42 /var/www/magento/releases/20170221151910/bin/magento(23): Symfony\Component\Console\Application->run()
#43 {main}

Successful: 2249 files; errors: 1

What could possibly be wrong, because while running setup:static-content:deploy from magento itself, it works, with no errors?! It is happening only when running the command remotely. At first look seems like an out of memory, but not sure yet! Please advice!

I hope the issue wont be closed because of any guidelines.

cc @Vinai :)

Thanks in advanced! Any help will be appreciated!

PS: this might be more related to magento, not capistrano, because is reproducible without deploy tool, using ssh [email protected] /usr/bin/env php -f /path_to_site/bin/magento -- setup:static-content:deploy en_US | stdbuf -o0 tr -d ., see: https://github.com/davidalger/capistrano-magento2/issues/61#issuecomment-281460939

EDITED: If I put magento cache to files instead of redis, the error is not reproducible anymore.

[magexo]

We see similar behaviour when running static deploy on 2.1.4 over ant in production mode. Seems like redis client is not thead-safe in some conditions.

[avra911]

sorry @magexo but why would it be thead-safe when running static-content:deploy from same server and not over ssh or ant, seems very weird, after all is the same command

[magexo]

Static deploy is running in more threads...

We have confirmed random issues with redis even without ant, only from shell.

Running with -j 1 will always end with success.

[avra911]

That is a great workaround @magexo (if that so), really appreciate that!

[magexo]

It's nasty workaround, because deploy takes ages...

[avra911]

yes, I can understand, but maybe is better ages than never :)

[avra911]

strange that I never reproduced the issue running the command in the server, glad to hear that from you. I reopened he issue now.

[lingwooc]

Just hit this in 2.1.7. I am using redis for a backend cache and session cache (issue appeared with the former). This was running static-content:deploy from the command line.

Fortunately my normal deployment script only deploys the static content to a staging instance which doesn't use redis. rsync does the _actual_ static deployment to production.

[avra911]

@lingwook, did you tried decresing jobs to 1? Is available in capistrano-magento2 gem. Frankly I never tried it, but should solve the problem, with the downside being a bit slower. I think is a redis issue.

[lingwooc]

Sorry I should have said, yeah that seams to have helped. Just wanted to update the ticket to show it still affects the latest version.

From a quick look about: "You might not know it, but Redis is actually single-threaded, which is how every command is guaranteed to be atomic. While one command is executing, no other command will run." https://github.com/karlseguin/the-little-redis-book

Which makes me think its a magento2 bug.

[hostep]

It's a magento2 bug, there is some more detail over in this issue: https://github.com/magento/magento2/issues/9287. An internal MAGETWO ticket was assigned, whichs means they probably confirmed the issue and are looking into resolving it.

[magento-engcom-team]

@avra911, thank you for your report.
The issue is already fixed in 2.1.9

[dlkinsey85]

I'm on 2.1.9 and am still having this issue...

[domeglic]

I am also having this issue on 2.2. Before updating from 2.1 I didn't experience this at all.