Elasticsearch-py: Set the Content-Type header on requests

[bgroff]

Maybe I am misunderstanding the code, but it seems as if the actual http request (either urllib3 or requests) do no set the Content-Type. I would expect that at the very least, if the JSONSerializer was used that the Content-Type header would be set to "application/json" and something like "text/plain" if the payload is not JSON.

If my understanding is incorrect, then my apologies and feel free to close this ticket. If this is a valid concern and you are looking for a PR then I would be able to supply something in the coming weeks.

As a side note, it looks like you have added support for arbitrary headers in the master branch, I think this could probably be used to set the Content-Type, but it seems to me like something more fundamental that you would always want set.

[HonzaKral]

Can I ask you for the reasons behind this request? Elasticsearch itself assumes json by default and actually does it's own detection. There is no need for us to add a few bytes to every single request.

Not to mention that we cannot always guarantee what the body will contain since in some cases (user passing in a string as body parameter for example) we don't have control and in other cases (bulk, mget, msearch, ...) the body is not a valid json anyway.

Thank you!

[bgroff]

Sure, we have a custom middleware that looks at the particular request and figures out if going directly to certain indices might be faster. We are using Django (and Django Rest Framework) which looks at the Content-Type to figure out which deserializer to use on the incoming request. The work around is not terrible, basically for each route just tell drf what to use.

I know it adds extra information to the http header, and figuring out the Content-Type is extra work. It also seems like using the headers method would allow us to set the Content-Type ourselves, it would just be nice to have the header set automatically if possible.

[HonzaKral]

I am afraid that this is currently out of scope for the project. Since we cannot guarantee any of the content types we also cannot set the appropriate headers. Feel free to set them yourself globally using the headers parameter or on a per-request basis by extending the connection class and using your own.

[opentokix]

When you run Elasticsearch 6.x it will not assume Json and throw error 406 instead.

(406, u'Content-Type header [] is not supported', {u'status': 406, u'error': u'Content-Type header [] is not supported'})

https://www.elastic.co/blog/strict-content-type-checking-for-elasticsearch-rest-requests

"Starting from Elasticsearch 6.0, all REST requests that include a body must also provide the correct content-type for that body.

In earlier releases of Elasticsearch, the content-type was optional, and if it was missing or not recognised, then the server would sniff the content and make an educated guess regarding the content-type. That will no longer happen in Elasticsearch 6.0 - every incoming request needs to have the correct content-type for the body it contains.

This ability to enforce strict content-type checking has existed since Elasticsearch 5.3 via the http.content_type.required configuration setting. In 5.x it is optional, and defaults to false, in Elasticsearch 6.0, that setting defaults to true, and there is no way to disable it."

[HonzaKral]

@opentokix this had been implemented in 5.2.0 by d862c8a.

For elasticsearch 6.x you wil have to use the master version of the elasticsearch-py library so it shouldn't be an issue.

[opentokix]

Oh, awesome - I was running the package-provided library.

[WakeDown-M]

Hello guys, could you please help me how to set that library to use master version? Where can I modify that requirements.txt file?

(Im now trying to learn how to use ES with Python to create some visualisations in Kibana so Im trying to import some data from online StarWars API :) ) IM getting transport error 406.

Thank you in advance!

[rjurney]

I keep getting this error lately: pyelasticsearch.exceptions.ElasticHttpError: (406, 'Content-Type header [] is not supported')

Can this be fixed please?

[HonzaKral]

@rjurney I am sorry but I cannot help you there, it is a different library, please open an issue there or migrate your code to use elasticsearch-py instead. pyelasticsearch hasn't been maintained for years and it is definitely not recommended for use.

[AdarshaG]

@HonzaKral .. i tried installing elasticsearch-py, but get below error..

Could not find a version that satisfies the requirement elasticsearch-py (from versions: )
No matching distribution found for elasticsearch-py

[yami14]

@HonzaKral @AdarshaG .. were you able to solve this error , even I am facing the same issue.

TransportError(406, 'Content-Type header [] is not supported')
and when I am trying to install elasticsearch-py instead of pyelasticsearch , it gives me same error,

__"Could not find a version that satisfies the requirement elasticsearch-py (from versions: )
No matching distribution found for elasticsearch-py"__

[HonzaKral]

elasticsearch-py is a name of the project, to install it just use pip install elasticsearch

[yami14]

@HonzaKral I have already installed pip install elasticsearch. But still getting this
TransportError(406, 'Content-Type header [] is not supported') error.

Not Sure why it is not allowing me to create index.

[HonzaKral]

and what version of elasticsearch did you install? Can yo try upgrading via pip install -U elasticsearch? Thanks!

[yami14]

@HonzaKral I am using elasticsearch 2.0 , and when I am trying to upgrade the version it show below logs:

(base) C:UsersXYZ>pip install -U elasticsearch
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 11001] getaddrinfo failed',)': /simple/elasticsearch/
Requirement already up-to-date: elasticsearch in c:usersdocumentspythonlibsite-packages
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno 11001] getaddrinfo failed',)': /simple/urllib3/
Requirement already up-to-date: urllib3<2.0,>=1.8 in c:usersXYZdocumentspythonlibsite-packages (from elasticsearch)
You are using pip version 9.0.1, however version 9.0.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

[HonzaKral]

@yami14 this is a problem with your pip configuration where it cannot reach the PyPI index. Either way I am sorry, but we cannot help you there. The problem is outside of this library.

[chuttipayan]

@HonzaKral
My elastic search is 6.2.2 and elasticsearch-py is 6.2.0 , I am trying to create an index use es.create am also getting the same error any thoughts would help
es.create(index="test",id=pageid ,doc_type="articles", body={"lineNum": 1,"content": "What is the best Loan Origination Software (LOS)?"})

[chuttipayan]

i downgraded the python library to 6.0.0 and then it worked

[wincle]

1.pip install elasticsearch==5.2.0
2.in lib/python2.7/site-packages/pyelasticsearch/client.py line:271
change _,prepped_response = ...
to prepped_response = ...
solve the problem