Magento2: After upgrade 2.1.2 to 2.1.7, Redis cache ends up in bad state intermittently, then all requests return 503

Created on 21 Jun 2017 · 22Comments · Source: magento/magento2

Preconditions

Magento CE 2.1.7
OS: RHEL 7.2
Apache 2.4.6-45
PHP 7.0.19-1 (IUS)
Redis 3.2.4-1 (IUS)
Percona MySQL 5.7.15-9
2 identical servers behind an F5 load balancer, plus a separate admin server.
Media is shared between the servers via NFS.

Steps to reproduce

Run production traffic for hours to days until the 503s begin.
We have been unable to reproduce on staging, even replaying captured production traffic. However, capturing an rdb dump from the Redis server while in the error state and transplanting it to a test server has allowed us to at least reproduce the error state.

Expected result

Home page should load without issue

Actual result

User receives error processing request page with error number.
var/report contains file with error number, exception contained therein is one of the following, but during a particular error state the same exception occurs on every request:

a:4:{i:0;s:134:"Warning: array_merge(): Argument #1 is not an array in /data/www/magento2/vendor/magento/framework/Image/AdapterFactory.php on line 37";i:1;s:7726:"#0 [internal function]: Magento\Framework\App\ErrorHandler->handler(2, 'array_merge(): ...', '/data/www/magen...', 37, Array)
#1 /data/www/magento2/vendor/magento/framework/Image/AdapterFactory.php(37): array_merge(NULL, Array)
#2 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(93): Magento\Framework\Image\AdapterFactory->__construct(Object(Magento\Framework\ObjectManager\ObjectManager), Object(Magento\Framework\Image\Adapter\Config), Array)
#3 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(89): Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject('Magento\\Framewo...', Array)
#4 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(71): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('Magento\\Framewo...')
#5 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(126): Magento\Framework\ObjectManager\ObjectManager->get('Magento\\Framewo...')
#6 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(53): Magento\Framework\ObjectManager\Factory\AbstractFactory->resolveArgument(Array, 'Magento\\Framewo...', NULL, 'adapterFactory', 'Magento\\Framewo...')
#7 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(82): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->_resolveArguments('Magento\\Framewo...', Array, Array)
#8 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(71): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('Magento\\Framewo...')
#9 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(126): Magento\Framework\ObjectManager\ObjectManager->get('Magento\\Framewo...')
#10 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(53): Magento\Framework\ObjectManager\Factory\AbstractFactory->resolveArgument(Array, 'Magento\\Framewo...', NULL, 'imageFactory', 'MageArray\\Blog\\...')
#11 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(82): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->_resolveArguments('MageArray\\Blog\\...', Array, Array)
#12 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(71): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('MageArray\\Blog\\...')
#13 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(126): Magento\Framework\ObjectManager\ObjectManager->get('MageArray\\Blog\\...')
#14 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(53): Magento\Framework\ObjectManager\Factory\AbstractFactory->resolveArgument(Array, 'MageArray\\Blog\\...', NULL, 'dataHelper', 'MageArray\\Blog\\...')
#15 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(82): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->_resolveArguments('MageArray\\Blog\\...', Array, Array)
#16 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(57): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('MageArray\\Blog\\...', Array)
#17 /data/www/magento2/var/generation/MageArray/Blog/Model/BlogpostFactory.php(43): Magento\Framework\ObjectManager\ObjectManager->create('MageArray\\Blog\\...', Array)
#18 /data/www/magento2/app/code/MageArray/Blog/Controller/Router.php(154): MageArray\Blog\Model\BlogpostFactory->create()
#19 /data/www/magento2/vendor/magento/framework/App/FrontController.php(50): MageArray\Blog\Controller\Router->match(Object(Magento\Framework\App\Request\Http))
#20 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(74): Magento\Framework\App\FrontController->dispatch(Object(Magento\Framework\App\Request\Http))
#21 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(70): Magento\Framework\App\FrontController\Interceptor->___callParent('dispatch', Array)
#22 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'requestPreproce...')
#23 /data/www/magento2/vendor/magento/module-store/App/FrontController/Plugin/RequestPreprocessor.php(94): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#24 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\FrontController\Plugin\RequestPreprocessor->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#25 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'install')
#26 /data/www/magento2/vendor/magento/framework/Module/Plugin/DbStatusValidator.php(69): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#27 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Framework\Module\Plugin\DbStatusValidator->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#28 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#29 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/VarnishPlugin.php(55): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#30 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\PageCache\Model\App\FrontController\VarnishPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#31 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#32 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/BuiltinPlugin.php(68): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#33 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(142): Magento\PageCache\Model\App\FrontController\BuiltinPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#34 /data/www/magento2/var/generation/Magento/Framework/App/FrontController/Interceptor.php(26): Magento\Framework\App\FrontController\Interceptor->___callPlugins('dispatch', Array, Array)
#35 /data/www/magento2/vendor/magento/framework/App/Http.php(135): Magento\Framework\App\FrontController\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#36 /data/www/magento2/vendor/magento/framework/App/Bootstrap.php(258): Magento\Framework\App\Http->launch()
#37 /data/www/magento2/pub/index.php(37): Magento\Framework\App\Bootstrap->run(Object(Magento\Framework\App\Http))

a:4:{i:0;s:60:"Time to live is a mandatory parameter for set public headers";i:1;s:9650:"#0 /data/www/magento2/var/generation/Magento/Framework/App/Response/Http/Interceptor.php(50): Magento\Framework\App\Response\Http->setPublicHeaders(NULL)
#1 /data/www/magento2/vendor/magento/module-page-cache/Model/Layout/LayoutPlugin.php(48): Magento\Framework\App\Response\Http\Interceptor->setPublicHeaders(NULL)
#2 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(152): Magento\PageCache\Model\Layout\LayoutPlugin->afterGenerateXml(Object(Magento\Framework\View\Layout\Interceptor), Object(Magento\Framework\View\Layout\Interceptor))
#3 /data/www/magento2/var/generation/Magento/Framework/View/Layout/Interceptor.php(78): Magento\Framework\View\Layout\Interceptor->___callPlugins('generateXml', Array, Array)
#4 /data/www/magento2/vendor/magento/framework/View/Layout/Builder.php(104): Magento\Framework\View\Layout\Interceptor->generateXml()
#5 /data/www/magento2/vendor/magento/framework/View/Layout/Builder.php(64): Magento\Framework\View\Layout\Builder->generateLayoutXml()
#6 /data/www/magento2/vendor/magento/framework/View/Layout.php(244): Magento\Framework\View\Layout\Builder->build()
#7 /data/www/magento2/vendor/magento/framework/View/Layout.php(859): Magento\Framework\View\Layout->build()
#8 /data/www/magento2/var/generation/Magento/Framework/View/Layout/Interceptor.php(414): Magento\Framework\View\Layout->getBlock('page_content_he...')
#9 /data/www/magento2/vendor/magento/module-cms/Helper/Page.php(171): Magento\Framework\View\Layout\Interceptor->getBlock('page_content_he...')
#10 /data/www/magento2/vendor/magento/module-cms/Controller/Index/Index.php(43): Magento\Cms\Helper\Page->prepareResultPage(Object(Magento\Cms\Controller\Index\Index\Interceptor), 'home')
#11 /data/www/magento2/var/generation/Magento/Cms/Controller/Index/Index/Interceptor.php(24): Magento\Cms\Controller\Index\Index->execute(NULL)
#12 /data/www/magento2/vendor/magento/framework/App/Action/Action.php(102): Magento\Cms\Controller\Index\Index\Interceptor->execute()
#13 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(74): Magento\Framework\App\Action\Action->dispatch(Object(Magento\Framework\App\Request\Http))
#14 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(70): Magento\Cms\Controller\Index\Index\Interceptor->___callParent('dispatch', Array)
#15 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'contextPlugin')
#16 /data/www/magento2/vendor/magento/module-store/App/Action/Plugin/Context.php(106): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#17 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\Action\Plugin\Context->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#18 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'customer-app-ac...')
#19 /data/www/magento2/vendor/magento/module-customer/Model/App/Action/ContextPlugin.php(61): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#20 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Customer\Model\App\Action\ContextPlugin->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#21 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'storeCheck')
#22 /data/www/magento2/vendor/magento/module-store/App/Action/Plugin/StoreCheck.php(44): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#23 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\Action\Plugin\StoreCheck->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#24 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'tax-app-action-...')
#25 /data/www/magento2/vendor/magento/module-tax/Model/App/Action/ContextPlugin.php(91): Magento\Cms\Controller\Index\Index\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#26 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(142): Magento\Tax\Model\App\Action\ContextPlugin->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#27 /data/www/magento2/var/generation/Magento/Cms/Controller/Index/Index/Interceptor.php(39): Magento\Cms\Controller\Index\Index\Interceptor->___callPlugins('dispatch', Array, Array)
#28 /data/www/magento2/vendor/magento/framework/App/FrontController.php(55): Magento\Cms\Controller\Index\Index\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#29 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(74): Magento\Framework\App\FrontController->dispatch(Object(Magento\Framework\App\Request\Http))
#30 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(70): Magento\Framework\App\FrontController\Interceptor->___callParent('dispatch', Array)
#31 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'requestPreproce...')
#32 /data/www/magento2/vendor/magento/module-store/App/FrontController/Plugin/RequestPreprocessor.php(94): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#33 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\FrontController\Plugin\RequestPreprocessor->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#34 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'install')
#35 /data/www/magento2/vendor/magento/framework/Module/Plugin/DbStatusValidator.php(69): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#36 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Framework\Module\Plugin\DbStatusValidator->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#37 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#38 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/VarnishPlugin.php(55): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#39 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\PageCache\Model\App\FrontController\VarnishPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#40 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#41 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/BuiltinPlugin.php(68): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#42 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(142): Magento\PageCache\Model\App\FrontController\BuiltinPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#43 /data/www/magento2/var/generation/Magento/Framework/App/FrontController/Interceptor.php(26): Magento\Framework\App\FrontController\Interceptor->___callPlugins('dispatch', Array, Array)
#44 /data/www/magento2/vendor/magento/framework/App/Http.php(135): Magento\Framework\App\FrontController\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#45 /data/www/magento2/vendor/magento/framework/App/Bootstrap.php(258): Magento\Framework\App\Http->launch()
#46 /data/www/magento2/pub/index.php(37): Magento\Framework\App\Bootstrap->run(Object(Magento\Framework\App\Http))
#47 {main}";s:3:"url";s:1:"/";s:11:"script_name";s:10:"/index.php";}

Additional Information

What seems to happen is something corrupts the Redis cache, and from that point forward all requests start to return an error page with a 503. Deleting the bd5_SYSTEM_CACHE_EXISTS key from Redis, or issuing a FLUSHDB command, clears the error. The corrupted cache state even prevents the bin/magento CLI from functioning correctly.

FrameworCache Fixed in 2.1.x Clear Description Confirmed Format is valid bug report

Source

apicht

Most helpful comment

This is exactly an issue, which we have faced on our project after upgrading to 2.1.6. The root cause of the issue is in the class "\Magento\Config\App\ConfigType\System": there is a code, which checks, are the config records present in Cache storage, by checking the availability of another record ("SYSTEM_CACHE_EXISTS") in Cache storage. :-D
As the result: on Production, which has a lot of concurrent requests, some records may be deleted from Redis by tag, but some other threads may renew some records in cache. As the result: you may get some config records absent in Redis, however SYSTEM_CACHE_EXISTS record may still be present and have positive value. As the result: Magento does not renew the config records from Disk and your application stops working. With our traffic we got such errors approximately every 1-2 hours.
The solution is: the availability of a record in Cache storage can be 100% determined by only reading this record from cache. You cannot 100% be sure that all other records are still present in Cache by reading the value of another record. We have fixed this issue a little bit tricky by overriding the entire class (because a lot of methods in this class are PRIVATE!) "\Magento\Config\App\ConfigType\System" and making changes in several methods. However I believe Magento core team will understand my comment and will make better fix for Core. In 2.1.7 there is the same code in that class as in 2.1.6.

Note: on Production we use Redis for regular cache and Varnish for FullPageCache. However the described issue was applicable for our projects too.

amoskvenk on 20 Jul 2017

❤2 👍2

All 22 comments

How is your env.php setup?

Here's mine - i've replaced GZIP with LZF since it is faster:

'cache' =>
array (
'frontend' =>
array (
'default' =>
array (
'backend' => 'Cm_Cache_Backend_Redis',
'backend_options' =>
array (
'server' => '/run/redis/redis6380.sock',
'port' => '',
'persistent' => '',
'database' => '0',
'force_standalone' => '0',
'connect_retries' => '1',
'read_timeout' => '10',
'automatic_cleaning_factor' => '0',
'compress_data' => '1',
'compress_tags' => '0',
'compress_threshold' => '20480',
'compression_lib' => 'lzf',
),
),
),
),
'session' =>
array (
'save' => 'redis',
'redis' =>
array (
'host' => '/run/redis/redis6379.sock',
'port' => '',
'password' => '',
'timeout' => '2.5',
'persistent_identifier' => '',
'database' => '0',
'compress_data' => 1,
'compress_tags' => 1,
'compression_threshold' => '2048',
'compression_library' => 'lzf',
'log_level' => '1',
'max_concurrency' => '32',
'break_after_frontend' => '5',
'break_after_adminhtml' => '30',
'first_lifetime' => '6000',
'bot_first_lifetime' => '60',
'bot_lifetime' => '7200',
'disable_locking' => '0',
'min_lifetime' => '60',
'max_lifetime' => '2592000',
),
),

southerncomputer on 21 Jun 2017

env.php:

<?php
return array (
  'backend' => 
  array (
    'frontName' => 'redacted',
  ),
  'install' => 
  array (
    'date' => 'Mon, 11 Apr 2016 22:01:25 +0000',
  ),
  'crypt' => 
  array (
    'key' => 'redacted',
  ),
  'session' => 
  array (
    'save' => 'redis',
    'redis' => 
    array (
      'host' => 'redis-master.redacted.com',
      'port' => '6379',
      'password' => 'redacted',
      'timeout' => '2.5',
      'persistent_identifier' => '',
      'database' => '2',
      'compression_threshold' => '2048',
      'compression_library' => 'gzip',
      'log_level' => '1',
      'max_concurrency' => '6',
      'break_after_frontend' => '5',
      'break_after_adminhtml' => '30',
      'first_lifetime' => '600',
      'bot_first_lifetime' => '60',
      'bot_lifetime' => '7200',
      'disable_locking' => '0',
      'min_lifetime' => '60',
      'max_lifetime' => '2592000',
    ),
  ),
  'db' => 
  array (
    'table_prefix' => '',
    'connection' => 
    array (
      'default' => 
      array (
        'host' => 'mysql-master.redacted.com',
        'dbname' => 'magento2',
        'username' => 'prod',
        'password' => 'redacted',
        'active' => '1',
      ),
    ),
  ),
  'resource' => 
  array (
    'default_setup' => 
    array (
      'connection' => 'default',
    ),
  ),
  'x-frame-options' => 'SAMEORIGIN',
  'MAGE_MODE' => 'production',
  'cache_types' => 
  array (
    'config' => 1,
    'layout' => 1,
    'block_html' => 1,
    'collections' => 1,
    'reflection' => 1,
    'db_ddl' => 1,
    'eav' => 1,
    'config_integration' => 1,
    'config_integration_api' => 1,
    'full_page' => 1,
    'translate' => 1,
    'config_webservice' => 1,
    'customer_notification' => 1,
  ),
  'cache' => 
  array (
    'frontend' => 
    array (
      'default' => 
      array (
        'backend' => 'Cm_Cache_Backend_Redis',
        'backend_options' => 
        array (
          'server' => 'redis-master.redacted.com',
          'port' => '6379',
          'password' => 'redacted',
          'database' => '0',
        ),
      ),
      'page_cache' => 
      array (
        'backend' => 'Cm_Cache_Backend_Redis',
        'backend_options' => 
        array (
          'server' => 'redis-master.redacted.com',
          'port' => '6379',
          'password' => 'redacted',
          'database' => '1',
          'compress_data' => '0',
        ),
      ),
    ),
  ),
);

@southerncomputer - Thanks for the tip on lzf, I'll have to try it once this is resolved.

apicht on 21 Jun 2017

Wait you have your full page_cache using Redis? I don't think anything but Varnish is supported in production!

If you had varnish installed you'd also have:
'http_cache_hosts' =>
array (
0 =>
array (
'host' => '127.0.0.1',
'port' => '80',
),
),

I would consider installing varnish 4 and removing page_cache section from your setup!

Btw, Since redis is only single-core non-threaded. It is better to run multiple instances of redis if you have spare cores to keep things separated such as back-end cache and session cache! for performance reasons!

southerncomputer on 21 Jun 2017

I can't seem to find anything suggesting that Redis is not supported in production. We've been successfully using Redis on Magento sites since 1.x and this would come as a surprise to me.

The TTL exception above led me to believe this was somehow related to the FPC, but flushing the Redis FPC database doesn't clear the exception.

I'm aware of the single-thread issue with Redis, but this particular site doesn't get enough traffic to tax 1 core (~98% idle).

apicht on 21 Jun 2017

http://devdocs.magento.com/guides/v2.1/config-guide/varnish/config-varnish.html Strongly recommends varnish caching for production.. I just assumed everyone took that as meaning you should use varnish or-else. M2 is definitely a bit more complicated to deploy than M1!

maybe you are being affected by this issue:
https://github.com/magento/magento2/issues/8554

I've patched my 2.1.2-2.1.7 against this but i don't use a CMS on the main page! good luck!

southerncomputer on 21 Jun 2017

Note: on Production we use Redis for regular cache and Varnish for FullPageCache. However the described issue was applicable for our projects too.

amoskvenk on 20 Jul 2017

❤2 👍2

@amoskvenk Is there a possibility that you share your rewritten file in a gist until the Magento team fixes it?

elzekool on 20 Jul 2017

Sorry, but the code I wrote belongs to my company and I cannot share it by NDA. I can only give some tips how to implement it: need to make changes only in the method "readFromCache". If a config record is not loaded from cache, then it is: broken or just never exist in a config tree. Therefore after reloading need to check: if your $pathParts[0] and $pathParts[1] keys are present in the loaded config array. If keys are present, then the config was broken in Cache and now it is repaired and renewed. If keys are not present, then such path is never present in a config and need to push serialized empty array into Cache storage. Otherwise your config cache will be reloaded on each request from disk and your server will just stop working.

amoskvenk on 20 Jul 2017

@southerncomputer FWIW, we just finally got Varnish through staging, QA, and onto production. The same thing keeps happening with Varnish in place.

apicht on 21 Jul 2017

@amoskvenk Thank you for your help. With your pointers I managed to create a patch myself. I can understand the NDA restrictions. If all works correctly I will share my patched file after the weekend

elzekool on 21 Jul 2017

👍2 ❤1

As promised, this is a patched version for System.php that fixes the Redis issue. Use this patched version as a temporary fix, I strongly advise you not to change core Magento files! Patch it in your own extension!

https://gist.github.com/elzekool/b3e77a48ff3010d082c94f98c3e3f0ea

elzekool on 24 Jul 2017

🎉2

@elzekool could you also provide a unified diff patch file? So I could see the actual changes and apply the patch to my own existing System.php file?

7ochem on 24 Jul 2017

@elzekool Nice! Your code is almost like mine. Good job!

amoskvenk on 25 Jul 2017

Aha. Sorry. Have found one issue in your code. You add empty array to the entire config array ($config). Then you save the entire config into Cache. This will lead to full cache reload problem every time, because each config record, which is not present neither in the DB, nor in a default.xml file, will delete empty arrays of other such config records from the entire config object. So on the next request another config record will reload the config again and will just add empty array into the $config array, but other empty arrays will be cleared. Therefore correct solution is: to add this empty array into a Separate cache record into own cache key. Therefore after reloading of the config cache from disk you will not delete empty arrays of other non-existing config records in cache, which will not trigger the cache reloading again.

amoskvenk on 25 Jul 2017

This seems to have been resolved by MAGETWO-69840 on the branch 2.1.8-preview. The class has been completely refactored.
https://github.com/magento/magento2/commits/2.1.8-preview/app/code/Magento/Config/App/Config/Type/System.php

mrIntegrator on 28 Jul 2017

👍1

so can you just apply the changed files for 2.1.8 to 2.1.7..2 ?

southerncomputer on 28 Jul 2017

I'll let you know after I've tried it!

mrIntegrator on 28 Jul 2017

The MAGETWO-69840 commits can be applied to 2.1.7. So far, the bug has not reoccured for us. Don't take our word for it though; always test your application before applying the patch to production.

mrIntegrator on 28 Jul 2017

Alright i've patched it into my dev - patches clean! when is 2.1.8 coming out btw?

southerncomputer on 28 Jul 2017

I'm guessing that at the latest it would be in September when they launch 2.2, but I really don't know for sure. I wish they would release at regular intervals (every 6 or 8 weeks) instead. It would be easier to schedule maintenance that way to. At least they make the future releases visible here.

mrIntegrator on 28 Jul 2017

@apicht, thank you for your report.
The issue is already fixed in 2.1.10