Elasticsearch-py: RequestError: TransportError(400, u'mapper_parsing_exception', u'failed to parse')

[abhigenie92]

Creating an index:

def create_index(index_name):
    es=create_elastic_search_object()
    entry_mapping = {
    'entry-type': {
        'properties': {
            'text': {'type': 'string'},
            'coordinates': {'type': 'geo_point'},
            'username':{'type': 'string'} }
            }
            }
    es.indices.create(index_name,body={'mappings':entry_mapping})

Inserting into the index

coordinates= str(tweet[0][0])+","+str(tweet[0][1])
es.index(index=index_name, doc_type=keyword, id=start_id+ind, body={'text': tweet[1],'coordinates': coordinates,'username': tweet[2]})

Error:

*** RequestError: TransportError(400, u'mapper_parsing_exception', u'failed to parse')

Debugging:

(Pdb) body={'text': tweet[1],'coordinates': coordinates,'username': tweet[2]} 
(Pdb) print body
{'username': 'csd', 'text': 'RT @funder: Court Doc:Trump evicted a disabled US Veteran because he had a therapy dog\n\n@votevets #trumprussia #resist #theresistance #russ...', 'coordinates': '-117.1304909,32.7211149'}

All formats seem correct to me, what am I missing?

Libraries used:

from elasticsearch import Elasticsearch

https://elasticsearch-py.readthedocs.io/en/master/

[HonzaKral]

You have a field in your document that doesn't match your mappings. Inspect the .info attribute of the exception to get the whole information on which field it is.

[nbajason]

just got same error like you, looks like ES has some trouble when parse the dictionary, resolved by json.dumps(*).

[HonzaKral]

@nbajason what do you mean by "resolved by json.dumps"? The error has typically nothing to do with the document itself, instead referring to a field that failed to parse, so an individual value in your document doesn't match the type elasticsearch expects in its mappings - like trying to index a string as an integer...

[bicepjai]

how to ignore issues and insert other fields when non existing fields are there in the json data, when indexing ?

[HonzaKral]

You can set dynamic to False in the mappings and have elasticsearch ignore new fields. You can also set existing fields to be lenient and ignore malformed values by setting ignore_malformed to True - https://www.elastic.co/guide/en/elasticsearch/reference/6.1/ignore-malformed.html

[goolig]

The problem was solved for me by renaming the field '_id' to 'id'.

[apogre]

You have a field in your document that doesn't match your mappings. Inspect the .info attribute of the exception to get the whole information on which field it is.

@HonzaKral I set the ignore_malformed to True but the issue persists. How can I locate the field that is giving the issue?
I can only get:
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: RequestError(400, u'mapper_parsing_exception', u'failed to parse')

[HonzaKral]

@apogre You need to capture the exception instance and look at it's .info:

try:
    # whatever causes the exception here
except TransportError as e:
    print(e.info)