Magento2: Magento 2.3.5p-1 and Elasticsearch 6.8.11 fielddata error

[AleksandrPlotnikov]

Preconditions (*)

1. Magento 2.3.5p-1
2. Elasticsearch 6.8.11

Steps to reproduce (*)

1. Install magento using
   composer create-project --repository-url=https://repo.magento.com/ magento/project-enterprise-edition=2.3.5-p1
2. install Elasticsearch 6.8.11

Expected result (*)

Catalog Product page works well with updated_at sorting

Actual result (*)

Catalog Product page throw an error
[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [updated_at] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}]

We have default sorting on the catalog by updated_at desc, if I remove this sorting all works well. Elasticsearch displays updated_at as
"updated_at" : { "type" : "text", "index" : false },
So I am not sure why this is happening if updated_at is not indexed and obviously can not be set as fielddata=true.

Previous version o magento that we have was 2.3.0 (magento2ce, now we moved to the project-community-edition) with elasticsearch 5.6 and all works well with such config.

[m2-assistant[bot]]

Hi @AleksandrPlotnikov. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

  :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

:clock10: You can find the schedule on the Magento Community Calendar page.

:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

:movie_camera: You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[hostep]

We're running into a very similar problem with Magento OS 2.3.3 and ElasticSearch 6.8.11, but I haven't found the time and haven't got the knowledge yet to figure out what this is about.

@AleksandrPlotnikov: have you already found out more about this issue in the mean while?

[AleksandrPlotnikov]

@hostep kind of. I`ve updated reindex mechanism to set updated_at as keyword field, included this field into indexed fields and also couple of files was overrided with old elasticsearch magento module, this is a workaround but works well with sorting. But I still dont understand why it was working on previous version.

[hostep]

Okay, we've figured out the issue on our side, it's not really related to the problem mentioned here I think. But let me briefly explain it just in case other people run into similar problems:

• We have a custom product import which can also create attributes
• One attribute which was created had type "text" but also had the flag "is_filterable" set to "yes" (normally Magento backend doesn't allow this, but our custom import did allow this)
• ElasticSearch then threw one of two exceptions:
  
  
  
  • Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [our_custom_attribute] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
  
  
  • Fielddata is disabled on text fields by default. Set fielddata=true on [our_custom_attribute] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
  
  
• When removing the is_filterable flag from the attribute, ElasticSearch no longer threw errors.

Sorry for the noise, hope you find a proper solution for date attributes being used in the sorting, I don't know if Magento's implementation of ElasticSearch supports that or not.

[rparsi-boxycharm]

Just letting everyone know that I'm getting this same error with m2.3.5-p2 (enterprise/whatever the paid version is called) with ES 7.0

[JelleGe]

Getting the same error on m2.4.1. Any solution to this error?

[rparsi-boxycharm]

@JelleGe Depending on which ES version you're using, you need to create a "before" plugin on method addFieldsMapping, changing the ES field type to keyword.
In our case we made "after" plugin for \Magento\Elasticsearch7\Model\Client\Elasticsearch, so our beforeAddFieldsMapping looks like this

public function beforeAddFieldsMapping(
        Elasticsearch $subject,
        array $fields,
        string $index,
        string $entityType
    ) {
        if (!isset($fields['is_salable'])) {
            $fields['is_salable'] = [
                'type' => 'keyword'
            ];
        }

        return [$fields, $index, $entityType];
    }

You'll have to do this for any category or product attributes for which magento uses the wrong ES field type

[msyhr]

@JelleGe Depending on which ES version you're using, you need to create a "before" plugin on method addFieldsMapping, changing the ES field type to keyword.
In our case we made "after" plugin for \Magento\Elasticsearch7\Model\Client\Elasticsearch, so our beforeAddFieldsMapping looks like this

public function beforeAddFieldsMapping(
        Elasticsearch $subject,
        array $fields,
        string $index,
        string $entityType
    ) {
        if (!isset($fields['is_salable'])) {
            $fields['is_salable'] = [
                'type' => 'keyword'
            ];
        }

        return [$fields, $index, $entityType];
    }

You'll have to do this for any category or product attributes for which magento uses the wrong ES field type

I can confirm this works, however in my instance the array key was already set. So the below code was needed instead of !isset (your results may vary)

if (array_key_exists('your_attribute', $fields)) { $fields['your_attribute] = [ 'type' => 'keyword' ]; }

[mlaurense]

Thanks @rparsi-boxycharm and @msyhr for your input. I ended up combining both your additions to fix our issue. Complete example for the one's running into the same issue:
di.xml:

    <type name="Magento\Elasticsearch7\Model\Client\Elasticsearch">
        <plugin name="fix_elasticsearch_index" type="<Vendor>\<Module>\Plugin\Elasticsearch" sortOrder="1"/>
    </type>

Plugin/Elasticsearch.php:
```

namespace Plugin;

class Elasticsearch
{
public function beforeAddFieldsMapping(
MagentoElasticsearch7ModelClientElasticsearch $subject,
array $fields,
string $index,
string $entityType
) {
$skip = [
'image_label',
'links_title',
'samples_title',
'small_image_label',
'thumbnail_label',
];
foreach ($skip as $field) {
if (!isset($fields[$field]) || array_key_exists($field, $fields)) {
$fields[$field] = [
'type' => 'keyword'
];
}
}

    return [$fields, $index, $entityType];
}

}