Elasticsearch-py: Error: TransportError(400, 'mapper_parsing_exception', 'failed to parse')

[juancar1979]

Hi!

I have the following error:

Error: TransportError(400, 'mapper_parsing_exception', 'failed to parse')

It happens when I try to load in elasticsearch a search in shodan. The part of the code where I do it is:

    index = keyType + "-" + name_emp
    doc_type = "tipo_" + keyType + "_" + name_emp
    jsontry = json.dumps(result['matches'])
    bodyT = json.loads(jsontry)
    idM = page
    try:
          es.index(index=index, doc_type=doc_type,id= idM, body= bodyT)  
    except ElasticsearchException as es1:
          print('Error: ', es1)

I have put the logging:
import logging
logging.basicConfig(level=logging.DEBUG)

With the following result:

DEBUG:elasticsearch:< {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"not_x_content_exception","reason":"Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}},"status":400}

I try to enter a simple search for shodan "org:"

Thanks!!!!! :)

JC

[fxdgear]

Thanks for reporting the issue. I feel like this might be a mapping issue. (trying to index a document with a filed that doesn't match the type of the data you're trying to index.

If you are able to share your mapping and a copy of the document you're trying to index we can probably quickly determine where the mismatch is.

For reference this issue might have some relevant information for you.

[juancar1979]

Thanks for your attention!

It is not a file itself, it is an object directly. I try to pass it through a .json file and I pass it.

It is a search in shodan, for example "org: siemens". I add some debug lines in the images.

Thank you very much!

[fxdgear]

@juancar1979

so maybe the issue is the data coming from shodan isn't very uniform.

IE Elasticsearch is attempting to create a mapping for the first doc in your bulk.
It assumes that there's a data type but then the data type changes in a different document and as a result we get this mapping error.

can you share some of your documents bodyT?

[fxdgear]

@juancar1979 I'm going to close this due to inactivity. If you find you're still having issues please feel free to re-open.