Magento2: Cron not Reindexing Index

[ghost]

The indexes are not being reindexed with the standard cron:run cron job. This is particularly noticeable when making a change to a category and it not being updated on the front end. I can add a static block or description to a category and save it in the backend and the updates will not appear on the frontend until the indexers have been manually reindexed. This could possibly be related to #2855.

Preconditions

1. Magento 2.1 CE
2. CentOS6
3. PHP 5.6.24
4. MySQL 5.6
5. Shared hosting with Plesk control panel
6. Using flat categories and products

Steps to reproduce

1. Make a change to a categories description or add a static content block
2. Save changes
3. Go to category on frontend to see result

Expected result

1. Category should be updated with new content

Actual result

1. Category is not updated with new content

I have noticed that the debug.log has been filling up quite fast with lines like the following:

[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_1"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_2"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_3"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_4"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_1"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_2"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_3"},"is_exception":false} []
[2016-07-27 11:33:01] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_design_config_grid_flat_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_customer_grid_flat_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_decimal_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_decimal_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_decimal_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_decimal_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_text_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_text_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_text_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_text_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_varchar_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_varchar_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_varchar_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_varchar_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_value_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_value_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_value_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_int_tmp_indexer_value_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_datetime_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_datetime_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_datetime_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_entity_datetime_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_tmp_indexer_4"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_drop_indexer_1"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_drop_indexer_2"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_drop_indexer_3"},"is_exception":false} []
[2016-07-27 11:33:02] main.DEBUG: cache_invalidate:  {"method":"GET","url":"http:/","invalidateInfo":{"identifier":"DB_PDO_MYSQL_DDL_catalog_product_flat_1_drop_indexer_4"},"is_exception":false} []

I have setup a cron job just for reindexing as a work around for now and it seems to be working. My cron jobs are setup as follows:

* * * * * /opt/plesk/php/5.6/bin/php -d memory_limit=1G -c /opt/plesk/php/5.6/etc/php.ini /magento2/bin/magento cron:run >> /magento2/var/log/magento.cron.log
* * * * * /opt/plesk/php/5.6/bin/php -d memory_limit=1G -c /opt/plesk/php/5.6/etc/php.ini /magento2/update/cron.php >> /magento2/var/log/update.cron.log
* * * * * /opt/plesk/php/5.6/bin/php -d memory_limit=1G -c /opt/plesk/php/5.6/etc/php.ini /magento2/bin/magento setup:cron:run >> /magento2/var/log/setup.cron.log
* * * * * /opt/plesk/php/5.6/bin/php -d memory_limit=1G -c /opt/plesk/php/5.6/etc/php.ini /magento2/bin/magento indexer:reindex

[hostep]

We also see this using CE 2.1.0 - PHP 7.0.8 - Mysql 5.7.13

When adding products, they don't appear on the frontend, until we manually run a bin/magento indexer:reindex every time.

Cronjobs are setup correctly and indexers are set to 'Update by Schedule'.

Sounds like a pretty important bug.

[grasdaggel]

We are experiencing the same problem on lower Mage2 versions (2.0.4 and 2.0.6). So far we haven't found a solution for that.

[nmallepally]

Yes, it is a important bug. I am experiencing the same issue as well, Magento 2.1 v, The cron is setup properly, but does not show products until it is re indexed.

[boldhedgehog]

Works for me if I set 'Use Separate Process' to 'No' for the index CRON group.

[hostep]

@boldhedgehog: tnx, but unfortunately it doesn't seem to fix our issue :(

[ghost]

Setting 'Use Separate Process' to 'No' for the index CRON group does not work for me either.

[boldhedgehog]

@hostep I had to wait a little and after 5-10 runs of cron:run Magento had added indexer tasks to cron_schedule:

| schedule_id | job_code | status | messages | created_at | scheduled_at | executed_at | finished_at |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1106 | indexer_update_all_views | success | NULL | 2016-07-29 15:34:45 | 2016-07-29 15:34:00 | 2016-07-29 15:34:47 | 2016-07-29 15:34:47 |
| 1105 | indexer_reindex_all_invalid | success | NULL | 2016-07-29 15:34:45 | 2016-07-29 15:34:00 | 2016-07-29 15:34:47 | 2016-07-29 15:34:47 |

[hostep]

@boldhedgehog: yes, I see them too, even with 'Use separate process' on 'Yes'. So this means the indexers are correctly executed through the cron, but something in the execution itself goes wrong I guess...

If I find some time in the coming days, I'll debug and try figure out what exactly goes wrong.

Tnx for the hints!

[ghost]

This is the same behaviour for me as well. The cron_schedule table says success for the status but the frontend does not update.

[boldhedgehog]

It has rebuilt my Elasticsearch index. This can be an index/caching issue, not a CRON/index issue you have now, in addition.

[ojhaujjwal]

I have the same problem in 2.1.0. I have setup the cron job nicely. I even tried to run cron:run manually more than once but the indexing was not done. However, running indexer:reindex did all the reindexing. So, I think that means a bug in cron:run which does not do the job of indexing.

[ojhaujjwal]

This seems a critical issue. Is there any workaround for fixing this until an official fix is done?

[ghost]

@ojhaujjwal I setup an additional cron job to run the indexer:reindex command for now. It fills up the debug.log file quite quickly when it's run though.

[ojhaujjwal]

@crtsl I noticed that if I add magento indexer:reindex to cron, the cache is invalidated everytime the command is run and I have to flush the page cache. This creates even more problems.

[Ctucker9233]

I have noticed this behavior as well. I have to run indexer:reindex and it makes my debug.log incredibly large. It's a pain but I clear out information that is more than a month old. Magento 1 had an automated function to clear the logs. Wish they would bring that back.

[ojhaujjwal]

Can anybody tell me the status of this issue? Is there any internal-ticket for the issue? Is there any commits that may fix the issue?

[gman-1986]

Has anyone found a solution to this problem yet?

[ghost]

@gman-1986 running php bin/magento indexer:reindex works to reindex but that's a manual approach, far from a fix.

I don't see an internal ticket reference for this issue though which is concerning considering it is an important bug.

[patrickpeng2091]

I have the same problem in 2.0.7 ,PHP7.0.8 ,mysql 5.6.30

running bin/magento indexer:reindex ,
Customer Grid index has been rebuilt successfully in 00:00:00
Category Products index has been rebuilt successfully in 00:00:00
Product Categories index has been rebuilt successfully in 00:00:00
Product Price index has been rebuilt successfully in 00:02:17
SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction, query was: DELETE FROM catalog_product_index_eav_idx
Stock index has been rebuilt successfully in 00:00:30
Catalog Rule Product index has been rebuilt successfully in 00:00:00
Catalog Product Rule index has been rebuilt successfully in 00:00:00
Catalog Search index has been rebuilt successfully in 00:00:00
Cleaned cache types:

[gman-1986]

Does anyone know if this issue is fixed in 2.1.1.

[ghost]

The issue still appears to be there after upgrading to 2.1.1.

[wildekin]

+1 this, so annoying having to manually reindex via ssh every time I make a change to the site!

[ghost]

@oserediuk is there any update to this issue? Is there an internet ticket?

[mbrehme]

+1 This issue is critical for us

[serzhik]

I use temporary solution.
Create pub/reindex.php:

<?php
header('Content-Encoding: none;');

set_time_limit(0);

$handle = popen("php /var/www/html/bin/magento indexer:reindex", "r");

if (ob_get_level() == 0)
    ob_start();

while(!feof($handle)) {

    $buffer = fgets($handle);
    $buffer = trim(htmlspecialchars($buffer));

    echo $buffer . "<br />";
    echo str_pad('', 4096);

    ob_flush();
    flush();
    sleep(1);
}

pclose($handle);
ob_end_flush();

Run http://example.com/pub/reindex.php to reindex.

Still waiting for solving of this issue.

[pitangaweb]

I have the same issue with Magento 2.1.1 on cPanel.
Cron jobs run everytime without errors
Nothing in log files or php error log
indexer_reindex_all_invalid shows in cron_schedule table

But new products won't show if I do not manually reindex - php bin/magento indexer:reindex

[gman-1986]

Does anyone know if the latest version 2.1.2 has fixed the problem?

[ghost]

@gman-1986 I just upgraded from 2.1.1 to 2.1.2 and this issue is not fixed.

[PieterCappelle]

Can confirm this is still an issue in 2.1.2. Could this please be added to the Magento Internal Bugtracker. This is critical...

[Grumag]

Same problem. 2.1.2. Cron jobs running, no errors. Have to manually reindex. Is anyone working on this?

[T-minus]

same issue going on over here for a while now. new to magento so Im not much help, sorry!

[PierAlex]

same issue on 2.1 and 2.1.2.

It appears that indexers are not invalidated after adding a new product. The cron job is correctly running. as boldhedgehog wrotes this issue may be not related to the cron job.

[Tjitse-E]

Same issue here on Magento 2.1, only the catalog_product_category index seems affected though.

[pantaoran]

I have the same problem. I can add new products but they don't appear in the frontend. Magento CE 2.1.

It seems that it's not invalidating the index correctly, since "indexer:status" shows all indexes as "ready", which means that the cron job will skip them.

When I force a reindexing of every index with "indexer:reindex" the new product appears on the frontend.

[ghost]

It appears that the following indexes are not getting invalidated/reindexed when creating/adding a new product:

catalog_category_flat - Category Flat Data
catalog_category_product - Category Products

I am able to get the new products to show by going to Products -> Categories and clicking Save on the default category(making no changes to the default category).

[pantaoran]

I was excited to read that, but it doesn't work for me. Saving the default category doesn't change anything, only manual (CLI) indexing will make the new product appear.

I also note that I don't have flat data.

Edit: after enabling both flat tables, the described workaround works! Saving the default category made the new product appear.

Another edit: I tried the above on a different M2 instance and it doesn't seem to work there. Not sure what the difference is exactly...

[ghost]

I also have multi-store enabled due to single-store mode causing other issues.

[sysradium]

Any news regarding that issue? After installation Magento says that indexes need to be reindexed even after running a cron job.

I could only change that manually running index:reindex job. I could add it as a cron job but it seems to be kind of weird and maybe expensive.

[nmallepally]

any work around for reindexing error or temporary solution ?
it is a painful process for us, indexing every hour to make configurable products attributes display for customers.
every hour or so, the config attributes disappears and displays again aft

er manually reindexing.
attached the one with indexing error and after indexing manually.

[kunzi]

We have this problem too. Crons are set up and running in Magento 2.1.2, but still new products are not visible in frontend and it also seems that extensions that rely on product indexing are not working properly.

[sijad]

same problem here with 2.1.2 and PHP 7.0.12

[petterkj]

EDIT:
I can reproduce this error in 2.1.5.

[gman-1986]

Anyone tried the new release 2.1.3 to see if the issue is fixed? It includes 90 functional fixes.

[Grumag]

Well upgraded to 2.1.3 and it doesn't seem that this bug was fixed. Still manually re-indexing via ssh. On the upside, everything is quite a bit faster.

[ghost]

I have found that the indexer was not reindexing due to duplicate SKUs. Magento for some reason duplicated a few products.

When I exported all products and looked at the export file, the second column (store_view_code) is blank for almost all my items. Some lines had "default" in the store_view_code column. These items were duplicates of other items, the line right above it was the actual item with all the info. Both had the same SKU.

I noticed most of these duplicate items were products that I had chosen the option in the admin to "Save and Duplicate". It appears for some reason it was also duplicating the item it was saving. I did not look into it further to see why. Not all were from the Save and Duplicate option but I noticed a lot of them were.

I tried deleting those SKUs so there were no duplicates and see if the indexer worked properly afterwards but it did not. I then deleted all products from Magento and imported them all back in without any duplicate SKUs. I cleared/flushed all caches and after that I was able to add new products and they would appear on the front end fine. The indexer was working again.

Here is an image showing the duplicate SKUs I seen on the product export.

I would suggest checking for duplicate SKUs to see if that's the issue anyone else is having.

If that doesn't work you could just make a php file to invalidate the indexes so they get reindexed and run it on a cron schedule.

<?php
    $servername = "SERVER_NAME";
    $dbname = "DATABASE_NAME";
    $username = "DATABASE_USER";
    $password = "DATABASE_PASSWORD";
    $sql = "UPDATE indexer_state SET status='invalid' WHERE status='valid'";    
    try {
        $conn = new PDO("mysql:host=$servername;dbname=$dbname", $username, $password);
        $conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
        $stmt = $conn->prepare($sql);
        $stmt->execute();
    }
    catch(PDOException $e) {
        echo "Error: " . $e->getMessage();
    }
    $conn = null;
?>

[AirmanAJK]

Spent a lot of time today tracing this issue. When I add/remove a category to a product, I only see the update when i manually reindex on the CLI. This all appears to be due to the indexer not being set to the invalid state when I save the product.

The following file: \vendor\magento\module-indexer\Model\Processor.php at Line 57 shows why this is an issue:

public function reindexAllInvalid()
    {
        $sharedIndexesComplete = [];
        foreach (array_keys($this->config->getIndexers()) as $indexerId) {
            /** @var Indexer $indexer */
            $indexer = $this->indexerFactory->create();
            $indexer->load($indexerId);
            $indexerConfig = $this->config->getIndexer($indexerId);
            if ($indexer->isInvalid()) {

The function only continues to reindex if it has the invalid state. If you actually manually update the database table indexer_state to have state = 'invalid' for the catalog_category_product or catalog_product_category indexer_id rows, the next cron:run call will update the index as expected.

I haven't yet dived into why changing the category doesn't invalidate the index, and at this point, I don't know if I'm going to bother, butI hope this info leads someone down the right track to squash this bug once and for all.

Curiously, if I change a product price, my indexers are all still valid, but after a minute when the cron jobs runs, my prices get updated in the index... weird.

[jvreeken]

This is still a problem in v2.1.3 and the ticket has been open since Jul 27, 2016.... I am never using magento for another project ever again...

[ssikyou]

Same problem in v2.1.3, why not anyone fix it ?

[romeof1980]

hi guys!
same behaviour for us: indexer_reindex does not work
when running "SELECT * FROM cron_schedule;" we see the other tasks be accomplished correctly ("success"), the scheduled tasks correctly reporting "pending" and the running tasks correctly reporting "running" as well. THE ERROR "missing" in indexer_reindex give also a hint: "Too late for the schedule"
Looks for us like a very similar issue we had with Magento 1.8 and 1.9 installations
That was caused by this cron.php issue: "cron.php will start a new process every time it’s being called which could easily result in performance problems or race conditions for tasks operating on the same datanot". (Credits: https://www.webguys.de/magento-1/tuerchen-08-magento-cron-demystified look at the section cron.php vs cron.sh)

apparently we have got the same problem here in Magento 2.0.4.

and @boldhedgehog solution points us straight in this direction: I reckon that saying that he can solve "setting 'Use Separate Process' to 'No' for the index CRON group" is like setting cron.php in Magento 1.8 and 1.9: it works, but it hits resources and fails if you're not on a dedicated. Would be interesting to know if other guys experiencing the issue are on shared solutions and could possibly suffer of performing issues.
I am by the way (a well-known Magento 2 partner that has worked like charm like no other) but I will test it immediately to check if we're slightly hitting performances.
Hence, ready to tackle a performance issue and knowing a bit of how we tackled it in Mage1, the solution came VERY EASY:

1. we've got the cron set every 2 minutes (very good for us in order to not stress too much the poor hardware)
2. we went to Store->Configuration->Advanced->System->Cron(scheduled tasks)->Cron configuration options for group:index and we set as following:
   Generate schedules every: 2 minutes (no need to generate every minute if the cron is set every 2)
   Missed if not run within: 4 minutes (for obvious reason..)

and that's it! Our Nexcess SIP100 is once again perfectly fit to handle it: he just needed a bit of relief.... the poor guy hosts 30 websites after all.

a hint: sometimes very old cron-tasks are present and still running (see the entry in cron_schedule): that keep the system under stress and could happen that the newly generated cron-tasks fail with "Too late for the schedule" (that has happened to us too)

[lgoudriaan]

Any update on this?

[gman-1986]

@lgoudriaan still not fixed in 2.1.5 or 2.2 dev

[romeof1980]

hi @lgoudriaan , hi @gman-1986 ,
would you mind to post the result of "SELECT * FROM cron_schedule;" and give us some details about your server env (is it shared? is it a VPS, Dedicated or a Magento-tailored solution like the Nexcess-SIP for example)?
As detailed in the comment above we've found that the reason of the issue isn't a bug but resides often (always for us) in servers hitting resources: after tuning the cron jobs (if you're on a shared env or on a not robust VPS) works fine for us (see pic).

cheers

[lgoudriaan]

Hello @romeof1980,

I'm using a shared hosting. At this moment i have 190 rows in that table, which looks wrong to me? You can find my dump here: http://pastebin.com/LgzKBGXP

[romeof1980]

hi @lgoudriaan ,
look i.e. at the very first row:
|27138|sales_send_order_emails|running|NULL|2017-02-10 21:18:05|2017-02-10 21:18:00|NULL|NULL
that means to me that this cron was scheduled to run at 2017-02-10 21:18:00 AND IS STILL RUNNING RIGHT NOW!!! (it says "running")

the second also:
|32518|indexer_update_all_views|running|NULL|2017-02-12 12:49:03|2017-02-12 12:49:00|2017-02-12 12:50:10|NULL
that means to me that this cron was scheduled to run at 2017-02-12 12:49:00 , it was run at 2017-02-12 12:50:10 AND IS STILL RUNNING RIGHT NOW!!! (it says "running").

the "NULL" in the fourth and last column means that it didn't finish to run.
Below you see another screenshot of my cron where you see the column's legend too:

I'd advise to run "top" from you public_html (from SSH) to see how much memory the running crons (there are many if you look closely at your file) are eating.
Then delete all the entry of cron_schedule (from SSH or from phpmyadmin) and check top and the cron_schedule again.
I see your cron running smoothly since 2017-03-06 16:40:06 (as of row 32 of your attachment) so I reckon it is now configured correctly but you have to to clear the processes still running.

cheers!

[lgoudriaan]

Thanks @romeof1980! it seems it's updating correctly now, there where no cronjobs running so i just cleared the cron_schedule table.

[romeof1980]

hi @lgoudriaan , sounds very good!
if the cron jobs should go wrong again, clear the cron_schedule but also check my first post above about tuning the cron jobs in order to avoid hitting server resources.
cheers!

[gman-1986]

Hi @romeof1980
I removed all cron jobs
in phpmyadmin ran the following TRUNCATE cron_schedule;
changed the group:index as follows
Generate schedules every: 2 minutes
Missed if not run within: 4 minutes

applied the cron jobs again to run every 2 minutes but still no luck.

I am on shared hosting
testing on 2.1.4
cron.txt

really appreaciate your help
thankyou

[romeof1980]

hi @gman-1986 ,
could you please provide an explanation about the file cron.txt you attached? it seems to me you're trying to run a dtb query like:
INSERT INTOcron_schedule(schedule_id,job_code,status,messages,created_at,scheduled_at,executed_at,finished_at) VALUES (1, 'catalog_product_outdated_price_values_cleanup', 'success', NULL, '2017-03-06 21:22:06', '2017-03-06 21:22:00', '2017-03-06 21:24:03', '2017-03-06 21:24:03'),

please clarify to me what you're trying to accomplish. I reckon that it should be the cron that is supposed to populate the cron_schedule, not you via a SQL query (by the way you're also inserting the 'status=success'... looks weird to me).
Thus, if you were so kind to give us some more details and background info I'm sure we can work it out: please clarify your issue the best you can and provide me the steps to reproduce it.
I'm pretty confident it's not a bug, we'll get it fixed.
cheers!

[Grumag]

So this issue is still happening for me. I am on Nexcess SIPR server, latest version of Magento 2. Anything I can do / tell Nexcess support to do?

[romeof1980]

hi @Grumag , would you mind to explain preconditions and step to reproduce?
Nexcess works just very fine (I can confirm it because my own experience) you just need to set up and tune the crons correctly. Please post the cron jobs you set up on your Siteworx and the result of "SELECT * FROM cron_schedule;"
It's not a bug, it's a set up process.
cheers!

[gman-1986]

Hi @romeof1980
It is working now after following your setup process carefully, that cron.txt was just an import of what it showed in cron_schedule through phpmyadmin.
There is one thing I did not change which is what @boldhedgehog said about 'Use Separate Process' to 'No' for the index CRON group.
Would you suggest I set it to No or Yes?
Thanks again for your help

[romeof1980]

hi @gman-1986 ,
thanks for clarifying it and for giving us a feedback: great to know the solution works for your env too.
About boldhedgehog suggestion, I would advise to NOT set 'Use Separate Process' to 'No' for the index CRON group: let it set to YES.
As I pointed out, I reckon that _setting 'Use Separate Process' to 'No' for the index CRON group" is like setting cron.php (instead of cron.sh) in Magento 1.8 and 1.9: it works, but often it hits resources and fails if you're not on a dedicated_.
Hence, set it on yes only as a last resort, keeping in mind that in most cases that fixes the problem only using much more resources that needed with a fine-tuned cron configuration.

cheers!

[PierAlex]

hi @romeof1980 ,
maybe there are different problems on this issue. I'm on a robust VPS plesk server; sheduled cron jobs every 2 minutes, truncated the cron table and applied admin system cron index configuration every 2 - 4 minutes as described by you but nothing happens.
One question, after you update a product or add a new product, do you have some index invalidated (or do you get that message) until the cron job runs?

Thanks,
Pierale

[PierAlex]

For a workaround I solved creating two triggers on the table catalog_product_entity, after insert and after delete that updates the field status of the table indexer_status from valid to invalid for the indexer_id = catalog_product_category.
When cron jobs runs, correctly every minutes, indexes are rebuilt and new products appears on the catalog, also deleted products disappears.
Page cache is not invalidated as it happens when running magento indexer:reindex.

[lgoudriaan]

Status update:
The cronjob runs smoothly and updates product properties in front-end.
Changing the category of an existing product doesn't update the front-end. For now i used the workaround of PierAlex.

[romeof1980]

Hi @pieralex ,
it's difficult for me to help without seeing exactly how the cron_schedule table looks like (without the triggers you added).
Feel free to post here a screenshot if you'd like me to take a look. I'll let you know about your previous question too asap.

cheers!

---- On Fri, 10 Mar 2017 15:52:22 +0100 [email protected] wrote ----

For a workaround I solved creating two triggers on the table catalog_product_entity, after insert and after delete that updates the field status of the table indexer_status from valid to invalid for the indexer_id = catalog_product_category.
When cron jobs runs, correctly every minutes, indexes are rebuilt and new products appears on the catalog, also deleted products disappears.
Page cache is not invalidated as it happens when running magento indexer:reindex.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[PierAlex]

hi @romeof1980 ,
here you can see my cron_shedule table that is not affected by my triggers.
Previous records you can see, all operation are with status = success.

[romeof1980]

hi @PierAlex ,
answering your question "One question, after you update a product or add a new product, do you have some index invalidated (or do you get that message) until the cron job runs?"
No, I don't get any invalidated index nor any message.
Here is what I think about this functionality in Magento 2: for what I've seen in my own experience in Magento 2, you're NOT going to see an invalidated index when editing a category or a product: it already says it will update on schedule. I also don't agree with the many technicians that say an index get invalidated when "saving" a product in Magento 1.9: the cache get invalidated, not the index: hence you shouldn't expect to see an invalidating index when saving/editing products or categories.
Anyway the Dev_docs confirms what they say and what you pointed out: http://devdocs.magento.com/guides/v2.0/extension-dev-guide/indexing.html
looking at the workflow it seems that the indexes should get invalidated when "update on schedule" is set and when you add or modify products.

I can not reproduce the issue in 2.1.2 though: when I add a new product or add a product to an existing category it shows up the first time the cron runs (as it has always done with other releases).
Sometimes after a couple of times if the server is temporarily overloaded.

About the "robust VPS plesk server": sorry buddy but I can judge without seeing the server features. "Robust" is too generic and subjective. Hence I won't exclude a server-side issue.
The cron on your cron_schedule looks fine though.
Please confirm that the problem happens only when adding a product to an existing category and not when editing a product or when adding a product to a category created "on-the-fly"

cheers!

[PierAlex]

Hi @romeof1980,
the server is a ten core Xeon 2,4 GHz with 50GB RAM and SSD with only this installation running.

Relating to my question, what about adding a new product (to the catalog)? Do you get the message that some index are invalid adding a new product to the catalog, not to a category?

The problem happens when adding a new product to the catalog.

The new product does not appear in the catalog until a reindex is done.

Sorry if I wasn't clear before.

thanks!

Pierale

[romeof1980]

hi @pieralex ,
thanks for clarifying it.
I updated my last post, though: you're right, looking closer at the DevDocs workflow it seems it should invalidate the index (when adding a new product) if you set it as "update on schedule": to be honest, I've never seen an invalidated index when inserting new products, categories etc (I tested it again today with the same result). I reckon that the index get invalidated only if certain modifications get done: I'm trying to take a look at the code but this is going to take some time.
Difficult for me to advise you for other tests. If you like me to take a look at your server and Magento, feel free to contact me directly. Just to make it clear: I'm not part of Magento itself but a member of the community like you.

cheers!

[romeof1980]

hi @PierAlex ,
please review the server's features you gave above: 50GB RAM looks like you're mistaken... I've never seen a VPS with such features... usually on a VPS 50GB is the starting storage....

[PierAlex]

Hi @romeof1980,
Thank you very much for your time and support. I'll wait for a solution by the magento team.

This is my server's memory:

cheers!
Pierale

[paulpartington]

I've been having this issue as well. I found that setting the the catalogsearch_fulltext indexing to be 'Update on Save' and keeping the others to be updated on schedule meant that changes were shown straight away.

[pmathbliss]

I more information that might help someone else here and lead to a solution. (I've experienced the same indexing issue on php 5.6.x,7.0.x, MySQL and MariaDB. It appears that using shell_exec to call the php bin/magento indexer:reindex fails to reindex properly.

I switched to the exec function and echoed the line to standard in of sh.

I tested on 5 different Magento 2 installs with varying php versions and MySQLs

_You mileage may vary for your php binary location._

<?php 

//location of the bin folder relative to the script mine was in /folder/script.php
$bin = dirname(__DIR__) . "/bin";
file_put_contents("status","Magento 2 Reindexing");
exec('echo "php \"'.$bin.'/magento\" indexer:reindex" | sh', $out);
echo implode(PHP_EOL,$out);
//caching
file_put_contents("status","Magento 2 Cleaning Cache");
exec('echo "php \"'.$bin.'/magento\" cache:clean" | sh',$out);
echo implode(PHP_EOL,$out);

?>

[lgoudriaan]

I just installed an other magento shop on the same server, same version and cronjobs and it's updating correctly. I still don't know why my main site doesn't

[spyrule]

@PierAlex Can you post the exact triggers you setup to invalidate those two indexes?

[PierAlex]

Hi @spyrule,
here they are:

CREATE TRIGGER trg_x_indexer_invalidator_product_delete AFTER DELETE ON catalog_product_entity
FOR EACH ROW BEGIN
UPDATE indexer_state
SET indexer_state.status = 'invalid'
WHERE indexer_state.indexer_id = 'catalog_product_category';
END

CREATE TRIGGER trg_x_indexer_invalidator_product_insert AFTER INSERT ON catalog_product_entity
FOR EACH ROW BEGIN
UPDATE indexer_state
SET indexer_state.status = 'invalid'
WHERE indexer_state.indexer_id = 'catalog_product_category';
END

Pierale

[riprod]

Is there a solution to this ? I am running 2.1.5 and have to reindex after every inventory, product or price change to get it t take

[srokatonie]

@PierAlex Your solution seems to be pretty good workaround, when I manually changed indexer_state.status to invalid it worked.
However when I'm trying to set triggers on the database as described above I get

#1235 - This version of MySQL doesn't yet support 'multiple triggers with the same action time and event for one table'

Unfortunately...

[matin73]

same issue on 2 different server with magento 2.1.5 - does anybody know how to fix it?

[matin73]

maybe similar - but cron is working, not setting invalid to indexing after adding a new product! no, i think is not a cron problem! what is solution?

[tobik999]

I have found that the indexer was not reindexing due to duplicate SKUs. Magento for some reason duplicated a few products.

I also noted that this happened after I duplicated a product in backend, thanks to you @crtsl I made an export and also found those duplicate skus. This bug drove me crazy for the whole weekend, even though it is a bug of the duplicate product function the indexer should give us an error message if things like this happen if somehow possible

[matin73]

@tobik999 i can confirm that it happens when duplicate product in backend and also having sku twice. If I change sku to have it only once, reindexing does not work. realy bad bug!

[tobik999]

@matin73 have you tried to delete the whole catalog and reimport it clean like crtsl suggested? Did you also use the Save and duplicate product function?

[matin73]

@tobik999 no i did not - does this solve the problem? yes, we used function save and duplicate prodcut

[ghost]

@matin73 That solved it for me. I haven't used the duplicate product feature since and it's been working okay.

[tobik999]

@matin73 do you have duplicate skus in your export csv? I will try it out this night, since I am using multichanneling on this shop, and will let you know afterwards if it is working for me. But for me it sounds like a reasonable solution if you have duplicated skus in your db.

[matin73]

yes i have duplicate sku if export csv. there is 1 product i do not see in admin but if i export products i can see this product which has duplicate sku

[matin73]

does somebody knwo where this duplicate entry comes from? i can not see the product in admin and i can not see it in database table catalog_product_entity?

[matin73]

so, i found problem and solution for me! problem was that 1 product has duplicate entry for sku. i do not know how this happens, but exporting products to csv show duplicate sku. in admin i could only see 1 product - the original, but not the double one. so i deleted the product with duplicate entry of sku in admin. then i exported csv and imported same csv again - and voila - indexing is working correct. saving new product is shown in frontend. thanks to tobik999 and crtsl!!

[PierAlex]

@srokatonie
my installation is running on mysql 5.7.17.
The code i posted before was autogenerated and i didn't test it.
Executing that code i get an error due to the delimiter, (;) and not the same error you get.
for the delimiter you can try this (tested):

DELIMITER $$
CREATE TRIGGER trg_x_indexer_invalidator_product_delete AFTER DELETE ON catalog_product_entity FOR EACH ROW BEGIN
UPDATE indexer_state
SET indexer_state.status = 'invalid'
WHERE indexer_state.indexer_id = 'catalog_product_category';
END
$$
DELIMITER ;
DELIMITER $$
CREATE TRIGGER trg_x_indexer_invalidator_product_insert AFTER INSERT ON catalog_product_entity FOR EACH ROW BEGIN
UPDATE indexer_state
SET indexer_state.status = 'invalid'
WHERE indexer_state.indexer_id = 'catalog_product_category';
END
$$
DELIMITER ;

[PierAlex]

@matin73, @crtsl:
do you have a MultiLanguage installation?

[matin73]

no, i do not have multilanguage. i have default magento installation with luma template ...

[tobik999]

although setting up a new product catalogue seems to fix the problem, it is not really applicable for me and I guess many more. I searched the db all over and over again and found no duplicates. The reason that there are a duplicate rows in the export is another bug https://github.com/magento/magento2/issues/4531 if you export without the description the export file looks fine.

Note to me and maybe others, if you want to go through the pain of setting up a new product catalogue:

• apply patch from https://github.com/magento/magento2/issues/4531 (or cleanup file manually) to get rid of duplicate rows in export.
• if you have configurable products you also need to apply this patch https://github.com/magento/magento2/issues/6938 (look at stackexchange links) to be able to ex-/import configurables with multiple variations.
• remap products to multichannel listings.
• do not use "Save and duplicate" until this is solved ( I am almost sure it started after using this)

OR without new product catalogue

• use the catalogsearch_fulltext indexing to be 'Update on Save' (dos not work in my case)

OR

• set in the db indexer_state.status to invalidthrough script

[matin73]

so - problem still exists!!! if you duplicate a product the indexer does not get invalidated!!! please fix this bug!!!!

[tobik999]

Maybe helpful for some of you who do not want to mess arround with the shell command: If I add one or multiple product and assign it (them) to category A and it does not get displayed, but afterwards I go to the category A and simply hit save it will get shown in the frontend. Maybe it could also be good enough to just hit save for the parent category, but I did not try yet.

[mironsoft75]

I think that works with all cronjob and all have correctly corrected. How to determine what we've all changed last time that leads to the error.

If I change everything to 'invalid' in the data index table indexer_state, the cronjob will reindex all. That is, when a new product is created or old is changed, the indexer_state table is to be changed to invalid and this does not happen. Why?
When will catalog_product_category be changed to 'invalid' in 'indexer_state', so the cronjob will notice this?

I have the database of test environment ported in live environment. I would have to adjust the DEFINER for the trigger. Just like here:

https://www.ask-sheldon.com/magento-error-cause-by-definer-in-mysql-dumps/

Reindex in the test environment still works in the live but not. In both, however, I duplicated the products.

[toddbc]

I'm sure there can be many issues for indexes not running, but for some, it may be caused by missing triggers.

You can recreate the triggers in Magento 2 with this command:

magento indexer:set-mode realtime && magento indexer:set-mode schedule

If you continue to experience indexer issues afterward, your problem is likely not related to triggers. But many of the comments above sound like missing triggers, and this method is the simplest way to rule it out.

[matin73]

thanks tddbc. but problem is NOT that reindexing is not running. problem is, that products are not set to invalid index after saving. this is really a BAD BAD BAD bug and I am confused why magento team is not doing anything against this bug - bug is knwons for more than 1 year! I have already lost 2 customers. And at the moment I can not recommend magento at all because of this bug.

[korostii]

Hi @matin73, you seem to be mixing up the reindexing by cron (schedule mode) with the "update on save". As far I understand, when magento is set to "update on save" index never gets invalidated because indexes are supposed to get updated right away using the DB triggers.

And those triggers might become missing on many Magento installations, possibly due to database transfer via an incomplete dump (you can check on your DB using SHOW TRIGGERS; SQL command). See also #9036

Now, what @toddbc is saying means that switching the "update on save" mode on _and then back_ off seems to restore the trigger into the database. Please try if that workaround helps with your case.

[matin73]

hi @korostii , thanks for you reply - but i don not mixing up. I have opend an extra ticket on https://github.com/magento/magento2/issues/9174 and I was told that is same issue as here described. BUT there seems to be 2 bugs. one with reindexing not running correcty AND what I mean is database is not set to "invalid" index. so product never gets updated on frontend (also never been shown if new product). and that is realy a big big problem!!! and what makes me realy sad, is that magento core team is ignoring this two (or maybe more) bugs. that is realy bad - because many here have problems with this issue and nobody of core team is event trying to understand the problem nor trying to do anything against this bugs!!

[mironsoft75]

one with reindexing not running correcty AND what I mean is database is not set to "invalid" index. so product never gets updated on frontend (also never been shown if new product). and that is realy a big big problem!!!

Also other important bug

URL key for specified store already exists

So you can not use Magento. In Germany already many are converted to other shop systems. I hope it is not the end of Magento. Many bugs have not been fixed. With these bugs Magento can not be used.

[matin73]

1 year of realy big problems and none of the magento core team is saying a word about the bugs!! Hey guys where are you???

[romeof1980]

hi @matin73 ,
I was able to reproduce your issue on a dev 2.1.5 I set up and on my end I was also able to fix it (it's not properly a bug). I reckon you're NOT using a dedicated server but a shared service and it could be you're hitting resources. Please try to set to yes the following:
1 Stores -> Configuration -> Advanced -> System -> Cron (Scheduled Tasks) -> Cron configuration options for group:index -> Use Separate Process -> SET TO "YES"
2 Stores -> Configuration -> Advanced -> System -> Cron (Scheduled Tasks) -> Cron configuration options for group:default -> Use Separate Process -> SET TO "NO"

that should fix your problem. Please give us a feedback about the fact you're on a shared env.
cheers!

[matin73]

@romeof1980, thanks for helping. i am using 2 different server - both are root server, not shared. i have settings as you recommend - s. screenshot. ALL my settings are magento default settings. I just installed magento (upload tar via ftp, unziped via shell and installed via browser. no extensions, no extra confuguration! first products were shown correctly. after a few days i changed stock info of an existing product and nothing changes on shop. product is still shown out of stock, new products are not shown.

[romeof1980]

OK! Let us know if it works with those settings.
It works fine for me on a shared env, resource shouldn't be an issue for you then

---- On Tue, 16 May 2017 18:44:46 +0200 [email protected] wrote ----

@romeof1980, thanks for helping. i am using 2 different server - both are root server, not shared. i have settings as you recommend - s. screenshot

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[matin73]

@romeof1980 nope - no changes. I have changed price and nothing changes in frontend. old price still shown on listview and product also shown in stock - in admin set to out of stock. s. screenshots.

[romeof1980]

Gruess Gott @matin73 ,
your last screenshot shows "use separate process -> no" on "Cron configuration options for group:index", please set it to YES.
Additionally, would you mind to log to mysql and run the following command?
SELECT * FROM cron_schedule
(if you use table prefixing add the prefix in front of _cron_schedule).
let see if your cron is stuck. it seems it does not reindex prices and stock too, not just category-product association.
There is also a post in this thread that I wrote to provide a possible solution for this issue on Feb 20th where the issue is cron-related: I reckon for you it could be worth to have a look (see above).
IMPORTANT: your cron runs every minute: in order to exclude you're hitting performance set it every 2 minutes as per attachment.
IMPORTANT: just in case: give the system a few minutes before checking if the product gets updated. It won't be updated "on-the-fly" like with "update on save" but only when the cron run the indexing process.
Last but not least: DID YOU SET THE CRONS ON YOUR SERVER?

cheers!
crons.tar.gz

[matin73]

shure, crones are running ... "use separate process -> YES" was default setup - see firstscreenshot - same problem. s. screenshot of db. all success - problem still here. I thinks it is not an problem of cron.

[romeof1980]

hi @matin73 ,
I see from cron_schedule that between 2017-05-17 10-02 and 2017-05-17 11-30 no jobs were generated. At that time this morning you were hitting resources with some previous jobs that your system was trying to run (and in fact job 99575 and 99576 didn't run).
Hence, if you don't mind a last try, do as following:

1. set the cron every 2 minutes as per the 2 files I sent you in the previous post
2. clear completely the cron_schedule table (you shouldn't see ANY entry there)
3. try to edit your product and after a few minutes check if it gets updated correctly

cheers!
Note: if it does not work post again the cron_schedule table: I reckon that if you're using a clean Magento installation on a dedicated, there is little chance this is a bug: probably it's just a misconfiguration that can be fixed

[matin73]

so, today i had time to have again a look to this problem. setting cron to 2 minutes an trunk table did not help. changes price and no index was rebuilt again:

an realy realy strange thing is, that price on detail page has changes to old one - see screeshot above - the price has been € 16,00 - and now € 15,00 - but should be € 17,00 ???????

[dimple-vasoya]

Hello,

I have upgraded my Magento version from 2.1.0 to 2.1.7
Earlier the reindex time for Product Categories was 00:03:08 and now it has increased to 01:12:00
Can anyone help me understand the reason behind this?

[rho225]

I solved it by creating a few triggers and modifying others, make sure you have privileges. (I choosed reindex all, but you can modify it changing WHERE statement.)

delimiter //
CREATE TRIGGER custom_trg_catalog_product_index_price_after_delete AFTER DELETE ON catalog_product_index_price
FOR EACH ROW
BEGIN
UPDATE indexer_state SET status = "invalid" WHERE 1;
END//

delimiter //
CREATE TRIGGER custom_trg_catalog_product_index_price_after_insert AFTER INSERT ON catalog_product_index_price
FOR EACH ROW
BEGIN
UPDATE indexer_state SET status = "invalid" WHERE 1;
END//

delimiter //
CREATE TRIGGER custom_trg_catalog_product_index_price_after_update AFTER UPDATE ON catalog_product_index_price
FOR EACH ROW
BEGIN
UPDATE indexer_state SET status = 'invalid' WHERE 1;
END//

Add next line to this triggers:
UPDATE indexer_state SET status = "invalid" WHERE 1;

• trg_cataloginventory_stock_item_after_delete
• trg_cataloginventory_stock_item_after_insert
• trg_cataloginventory_stock_item_after_update

[matin73]

@rho225: can please explain where you created the triggers?

[rho225]

Directly on the db. This solution just force cron reindex because triggers set status = 'invalid' in all rows of indexer_state, the table that cron reads to know which index has to reindex. This triggers are executed when price or any stock attribute is modified.

[matin73]

thanks! so ALL products are reindexed each time if price or stock are modified, right? so if have 50.000 products it could be not so good?

[rho225]

You can modify triggers at your preference. You have just 10 indexes, for example, for price modifications you can invalide just the index of price (check for the right id on indexer_status table), for stock you have a few more. In our site, we don't have so many products, we still are on developer stage so I can't give you performance data yet. I just want to provide an approximate solution, perhaps someone with a better knowledge of indexes could help us to determinate which one reindex.

[pascaladriaansen]

We are experiencing similar problems. In our case however, it only occurs after there hasn't been any activity in the admin panel's catalog section for some days. So when no products or categories have been changed for a few days, and then we try to add a product to a category for example, the index will not be (partially) reindexed automatically. If we then check the System > Index Management page, we see that the 'Updated' column for almost all indexes is up to date, but not the 'Product Categories' and (sometimes) 'Category Products' and 'Category Flat Data' indexes. To make everything work as it's supposed to again, we have to manually reindex those indexes by separately running:

bin/magento index:reindex catalog_product_category
bin/magento index:reindex catalog_category_product
bin/magento index:reindex catalog_category_flat

After this, everything starts to work normally (meaning any catalog changes are usually visible on the frontend within a minute), until there haven't been any catalog changes in the admin panel again for some days.

It looks like somehow the index gets 'outdated' and will be skipped because of that.

EDIT: Ok, I figured out it has nothing to do with being outdated. Instead, it has to do with the version_id in the mview_state table not getting reset when any *_cl table is recreated upon changing an index from Update on Save to Update by Schedule. I created a separate issue for this #10204.

EDIT2: Hmm, turns out the issue I described in my edit above is caused by the Mirasvit_Misspell / Search Spell-Correction module (part of the Mirasvit Sphinx Search Ultimate module). I'll get them to fix this.

[gman-1986]

I have been marking products out of stock in my magento store using the admin panel. These items are not being shown as out of stock on my store front until I run a reindex.

Sorry @romeof1980 I thought you had fixed this issue but there still seems to be a problem.

[romeof1980]

ehi @gman-1986 ,
it appears clear there is 2 different issues here: if you followed my indication checking and cleaning cron_schedule, before troubleshooting anything too complicated you could now try these simple steps as suggested by others in this post:
set the indexer on "update on save"
re-set them on "update on schedule"
magento indexer:set-mode realtime && magento indexer:set-mode schedule
that should re-create the missing triggers as @toddbc correctly pointed out.
It seems it can't be excluded that there could be a bug arising after duplicating products: so when testing better go without it and creating products anew instead.

cheers!

[shamoon]

@romeof1980 holy cow I think you might have just solved a major issue for me (fingers crossed 🤞). Thanks!

[wehaveaplan]

Removing var/.update_error.flag solved this problem for me.

[romantomchak]

Hi to all,

I have the same issue with customer grid indexer. In case, when index mode setted as "Update on Schedule" and I delete or add new customers, this grid is not updated (customer_grid_flat table is not updated).

I have checked triggers on db and for customer tables there are no any triggers. Also, I have tried to switch indexer mode (using command indexer:set-mode realtime && magento indexer:set-mode schedule) but still no luck, no new triggers were created.

Magento: 2.1.8 EE
PHP: 7.1

[romeof1980]

please have a look at the other many suggestions in this thread and try them out.
If you'll have no luck, please detail what you tried.
thanks

---- On Mon, 04 Sep 2017 13:11:51 +0200 [email protected] wrote ----

Hi to all,

I have the same issue with customer grid indexer. In case, when index mode setted as "Update on Schedule" and I delete or add new customers, this grid is not updated (customer_grid_flat table is not updated).

I have checked triggers on db and for customer tables there are no any triggers. Also, I have tried to switch indexer mode (using command indexer:set-mode realtime && magento indexer:set-mode schedule) but still no luck, no new triggers were created.

Magento: 2.1.8 EE
PHP: 7.1

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[romeof1980]

hi there!
we can confirm we were able to reproduce the issue on 2.1.8 too (after upgrading from 2.1.5).
truncating the table cron_schedule, setting the indexer on "update on save" and re-setting them on "update on schedule" and finally clearing page_cache fixed the issue once again for us (at the first try).
For completeness' sake: we still run with the following settings:
1 Stores -> Configuration -> Advanced -> System -> Cron (Scheduled Tasks) -> Cron configuration options for group:index -> Use Separate Process -> SET TO "YES"
2 Stores -> Configuration -> Advanced -> System -> Cron (Scheduled Tasks) -> Cron configuration options for group:default -> Use Separate Process -> SET TO "NO")

cheers!

[karlsminton]

Can anyone confirm whether or not this is only affecting the indexers? Do any other scheduled tasks work normally?

Edit: From further research it seems to just be affecting indexers

[robbanl]

How is it possible that this issue have existed since summer 2016 and not have been fixed by Magento? Following...

[patadeloso]

If the indexers work from CLI, but not from cron. Check the default system ulimits. I've found that to be an issue on some systems.

[Nix-id]

I have problem on M2.2.0 with cron job "indexer_update_all_views"
it's stuck in "running" status
also exists process
20404 ? Ss 0:00 /bin/sh -c /usr/bin/php /home/mag/www//bin/magento cron:run | grep -v "Ran jobs by schedule" >> /home/mag/www//var/log/magento.cron.log
20437 ? S 0:05 /usr/bin/php7.0 /home/mag/www/*/bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

i have some configurable products with many simple products assined. ~5k simple products.

I don't see any errors in log files

[magento-engcom-team]

@crtsl, thank you for your report.
The issue is already fixed in 2.2.1

[dnadle]

Will the fix be backported to 2.1.x?

[jeffmalterre]

Not sure if this is related or not... I'm having trouble running cron (2.2.1) I can reindex myself but when I try to run cron it just hangs. I checked php/magento logs and there was nothing in them, however... when I looked into mysql logs im getting (Got an error writing communication packets) timestamped at the exact time I'm trying to run cron.

Thanks,
Jeff

[chlignell]

Hi everybody!

I had the same problem with indexes (my case product indexes) not being reindexed with cron:run. This caused new products to not being shown on frontend without a manual index:reindex.

The problem was that version_id in mview_state table had a value much higher than the autoincrement value in catalog_product_category_cl table. Everytime magento 2 cron runs it checks whether the changelogs tables have higher values than mview_state indexes (version_id). In my case I was able to resolve my issue by setting version_id in mview_state to 0 for the faulty indexes.

This command solved my problem:
UPDATE mview_state SET version_id = '0' WHERE view_id = 'catalog_product_category';

I hope this can help someone else out there, I hade to spend several frustrating hours on this issue.

/Chris

[gman-1986]

I still have the issue on 2.2.1 does anyone else?

[rahimalizada]

In my case column 'status' was stuck in 'suspended' for 'catalog_product_category' view in mview_state. Resetting the value to 'idle' fixed the issue. Additionally i set version_id to 0 and allowed the cron to reindex.
@chlignell Thanks for the tip!

[toddbc]

Yes - this can happen sometimes when the cron process either crashes or is terminated by oom-killer. For that scenario, you can also use magento indexer:reset instead of manually updating the database.

Note that before doing this, it's best to validate that no cron process is currently running.

[unicoder88]

@magento-engcom-team , is it possible to see some references to commit or task number to understand what can cause this?
Currently I'm having very strange behaviour that sometimes AUTO_INCREMENT numbers are reset back to 1 for some indexes (_cl tables) and update by schedule no longer works for them (M2.1.7).

[toddbc]

@unicoder88 I've actually just opened #13577 to fix exactly that issue. You can see an explanation of why it happens there.

[franckgarnier21]

You can check if the status of mview_state is not working after a script crash.

[ericvhileman]

We wrote an extension to fix these bugs, speed up performance, and control the execution of tasks: https://github.com/magemojo/m2-ce-cron

[somnath12]

No need to run re indexing. to get instant help please contact here [email protected]

[addinm]

@crtsl Please see my bug report and the solution here: https://github.com/magento/magento2/issues/19417

[FrankenSteinxD]

I still have this issue on Magento 2.3.0

Crons are setup correctly and my custom indexer is set to update on schedule.

I have to run index:reindex my_custom_indexer everytime I want to run my custom indexer, the indexer is not triggered on save too although I have correct subscriptions in my mview.xml

It only works fine calling it through the command line.

[JaromirMullerQuanti]

I have same issue and found that magento cannot run separated process to execute index job. I had to switch settings to Use Separate Process = No and then scheduled indexes executes fine.

I also noticed that indexer_update_all_views has status run.

Magento 2.3.1

[malwis]

Hello, i need your Help:
Magento 2.3.1
php /var/www/webroot/ROOT/bin/magento indexer:reindex

Results:
Design Config Grid index has been rebuilt successfully in 00:00:00
Customer Grid index has been rebuilt successfully in 00:00:00

I can't reindex following Position:
catalog_category_product
catalog_product_category

Please tell my what i can to reindex this 2 Position. Thanks

[AdeelMurtaza]

We are facing similar issue where after a customer purchases items on our store, items gets disappeared on our store front from category pages. Only solution is to manually reindex which is not a solution but strangely it is only happening to one of our stores.

Magneto support yet to come up with a positive answer. We are on Magneto cloud 2.3.1 and yet facing such issue. I was hopeful after Adobe acquisition but no luck.