Elasticsearch-dsl-py: "No handler for type [string] declared on field [test]" after update to ES6

[edmorley]

Hi!

After updating from Elasticsearch 5.5.1 -> 6.1.1 (and corresponding elasticsearch-py and elasticsearch-dsl-py 5.x -> 6.x updates; running under Python 2.7) I get:

...
  File "/home/vagrant/treeherder/treeherder/model/search.py", line 197, in _init
    item.init()
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch_dsl/document.py", line 152, in init
    cls._doc_type.init(index, using)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch_dsl/document.py", line 85, in init
    self.mapping.save(index or self.index, using=using or self.using)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch_dsl/mapping.py", line 116, in save
    return index.save()
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch_dsl/index.py", line 219, in save
    return self.create()
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch_dsl/index.py", line 203, in create
    self.connection.indices.create(index=self._name, body=self.to_dict(), **kwargs)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/client/indices.py", line 91, in create
    params=params, body=body)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/transport.py", line 314, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 161, in perform_request
    self._raise_error(response.status, raw_data)
  File "/home/vagrant/python/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: TransportError(400, u'mapper_parsing_exception', u'No handler for type [string] declared on field [test]')

Enabling elasticsearch.trace logging I see the request being made was:

[2018-01-02 14:38:52,749] INFO [elasticsearch.trace:67] curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/test-test-failure-line?pretty' -d '{
  "mappings": {
    "doc": {
      "properties": {
        "best_classification": {
          "index": "not_analyzed",
          "type": "integer"
        },
        "best_is_verified": {
          "index": "not_analyzed",
          "type": "boolean"
        },
        "expected": {
          "index": "not_analyzed",
          "type": "string"
        },
        "job_guid": {
          "index": "not_analyzed",
          "type": "string"
        },
        "message": {
          "analyzer": "message_analyzer",
          "type": "string"
        },
        "status": {
          "index": "not_analyzed",
          "type": "string"
        },
        "subtest": {
          "index": "not_analyzed",
          "type": "string"
        },
        "test": {
          "index": "not_analyzed",
          "type": "string"
        }
      }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "message_analyzer": {
          "filters": [],
          "tokenizer": "message_tokenizer",
          "type": "custom"
        }
      },
      "tokenizer": {
        "message_tokenizer": {
          "pattern": "0x[0-9a-fA-F]+|[\\W0-9]+?",
          "type": "pattern"
        }
      }
    }
  }
}'
[2018-01-02 14:38:52,749] DEBUG [elasticsearch.trace:70] #[400] (0.016s)
#{
#  "error": {
#    "caused_by": {
#      "reason": "No handler for type [string] declared on field [test]",
#      "type": "mapper_parsing_exception"
#    },
#    "reason": "Failed to parse mapping [doc]: No handler for type [string] declared on field [test]",
#    "root_cause": [
#      {
#        "reason": "No handler for type [string] declared on field [test]",
#        "type": "mapper_parsing_exception"
#      }
#    ],
#    "type": "mapper_parsing_exception"
#  },
#  "status": 400
#}

I believe this is coming from the usage of the field type String, here:
https://github.com/mozilla/treeherder/blob/2adb7b93dc9148eda66998e85a0ca20d1e986ecd/treeherder/model/search.py#L68-L75

Looking at the changelog for elasticsearch-dsl 5.0.0, I see:
String field type has been deprecated in favor of Text and Keyword

...so presumably this is related to that?

However if Elasticsearch 6 no longer supports string, then:

1. it seems like it should be removed from elasticsearch-dsl to prevent confusion / reduce the time spent debugging this? (Or at least a visible deprecation message logged somewhere?)
2. it would be great for the elasticsearch-dsl 6.0 release notes to mention it's now changed from deprecated to "doesn't work"

After figuring that out, I tried importing/using Text instead of String, however I then get:
elasticsearch.exceptions.RequestError: TransportError(400, u'mapper_parsing_exception', u'Failed to parse mapping [doc]: Could not convert [test.index] to boolean')

Both of these feel like things that perhaps would be best caught in the client to ease debugging?

Many thanks :-)

[HonzaKral]

Hi, indeed it is caused by that - string is deprecated and you should be using text or keyword instead.

string can still, however, exist in a mapping in a 6.x cluster if the index was created before the upgrade, that is why it is still present in the library. We will most likely remove it soon though in order to avoid similar confusion, thank you for the report!

Hope this helps!

[edmorley]

Thank you for the clarification!

[willtu]

So how to solve the problem occurs after changing from "string" to "text"?

elasticsearch.exceptions.RequestError: TransportError(400, u'mapper_parsing_exception', u'Failed to parse mapping [doc]: Could not convert [test.index] to boolean')

[HonzaKral]

@willtu there is something else going wrong, Looks like a boolean field you have cannot be converted properly, could you please provide your mappings and what operation (and the data) you are trying to do?

Thanks!

[ravi-score]

As mentioned, a string is deprecated and replaced by text and keyword. Index of text is 'analyzed' and whereas for keyword is 'not_analyzed'. You don't need to mention "index" when you are using 'text' or 'string'

[Raunak-Agrawal]

Not able to map mapper parsing exception says no handler for type string. Can you please help for it

[HonzaKral]

@Raunak-Agrawal see the original answer here - string fields have been removed by elasticsearch and either keyword or text type need to be used instead

[Raunak-Agrawal]

I have tried both, they are also not working!! what extra field should i add ?

[HonzaKral]

@Raunak-Agrawal tried how? What is the code that produces the above error? Without details we cannot really help you, please share a complete example of what you are doing.