Elasticsuite: error after upgrade Limit of total fields [1000] in index

[minikist]

Preconditions

Magento Version : 2.2.2

ElasticSuite Version: 2.5

Environment :

Third party modules :

Steps to reproduce

1. upgrade to elasticsearch 5.6 and elasticsuite 2.5
2. run indexer:reindex
   3.

Expected result

1. finish the reindex without errors

Actual result

1. [Screenshot, logs]
   Catalog Search indexer process unknown error:
   {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [magento2_default_catalog_product_20180203_190656] has been exceeded"}],"type":"illegal_argument_exception","reason":"Limit of total fields [1000] in index [magento2_default_catalog_product_20180203_190656] has been exceeded"},"status":400}

is there a way to set the attribute amount in some configs?

[southerncomputer]

PUT my_index/_settings
{
"index.mapping.total_fields.limit": 2000
}

Or during index creation:

PUT test
{
"settings": {
"index.mapping.total_fields.limit": 2000,
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
...
}
}

[minikist]

Thanks for the reply,
I found that too. But everytime i run the indexer it generate a new index. With the default as 1000. So i need to set it on the config somewhere. But i’m bit new on this. So do you have any ideas where to place it? Or other suggestions??

[southerncomputer]

getCreateIndexSettings in https://github.com/Smile-SA/elasticsuite/blob/cda127bd7b5ca72b67991a6c2c49cf699c6245e7/src/module-elasticsuite-core/Index/IndexSettings.php or /src/module-elasticsuite-core/Index/IndexSettings.php

[minikist]

Thanks that worked out. But the speed is very bad. It take about a minute to load a page the speed without the elastic was about 2,5 sec. is that possible to get it that fast too??

[southerncomputer]

Are you swapping? that can kill ES Performance!? only thing
i can think of if you have a nginx/php-fpm/redis/varnish set to production that could slow it down.

Need tons of ram for mysql/elasticsearch to run quickly with magento! Swapping kills - so does normal hard drive (versus ssd).

I've got 2.5 million sku's in my product on a single vm it is fast with 64gb reserved for mysql and 40/20gb reserved for elasticsearch!

[minikist]

that is serieus amount. We have 80K products 64GB mem 16GB go to mysql and i'm not sure how much goes to elasticsearch. But do you use mage 2.2.2 and elastic suite 2.5 with elastic 5.6? We changed it now to single website > store > store view. i don't know where to start.

[minikist]

sorry i forgot to say that i'm disabled swapping already. the setup is
apache > varnish > nginx and redis for the backend

[southerncomputer]

try using the perl script from mysqltuner.com and see if you are critically low on memory?

[afoucret]

PR #725 increase the default number of fields allowed in the indices created in ElasticSearch with ElasticSuite. It have been merged in branches 2.5.x and master. I close the issue since it's main topic have been addressed.

For performance issue, auditing your ElasticSearch setup could be a good start.
Maybe your setup is swapping as explained by @southerncomputer. You can find some reference to the garbage collector in ES logs (gc). Default ES heap size is 1Gb and you can feel a little short with it in some situation.

Additionally we can provide some audit service as part of our commercial paid support for the extension if you are interested.

BR,