Elasticsearch-py: UTF-8 serialization in python 2

[peta15]

I'm running python 2.7 connecting to AWS elasticsearch service using the 2.2 release of elasticsearch-py. To connect I use requests_aws4auth as recommended in your docs (thanks for integrating that!).

When writing to elasticsearch (bulk upload, creating a doc etc) I get this error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 2102: ordinal not in range(128)

I know that the library was changed a couple months ago to stop hiding unicode errors but this change coincides with the introduction of support for requests_aws4auth as they are both first seen in the 2.2 release and therefore downgrading is not an option for me. Handling unicode conversion myself piecemeal is non-trivial. Upgrading to python 3 is not an option yet given other dependencies.

Therefore, I have come up with a workaround for now using a custom serializer that essentially reverts the unicode change made earlier to this codebase:

from elasticsearch import Elasticsearch, RequestsHttpConnection, serializer, compat, exceptions

class JSONSerializerPython2(serializer.JSONSerializer):
    """Override elasticsearch library serializer to ensure it encodes utf characters during json dump.
    See original at: https://github.com/elastic/elasticsearch-py/blob/master/elasticsearch/serializer.py#L42
    A description of how ensure_ascii encodes unicode characters to ensure they can be sent across the wire
    as ascii can be found here: https://docs.python.org/2/library/json.html#basic-usage
    """
    def dumps(self, data):
        # don't serialize strings
        if isinstance(data, compat.string_types):
            return data
        try:
            return json.dumps(data, default=self.default, ensure_ascii=True)
        except (ValueError, TypeError) as e:
            raise exceptions.SerializationError(data, e)

I hope this helps anyone else that runs into this issue.

[HonzaKral]

Thanks for posting this here! I hope people will find it useful.

For the record other solution might be to just call super() and then encode the strings as bytes explicitly using utf-8 encoding, then the rest of the library will leave it untouched.

[vinitkumar]

@peta15 Hi, We are facing the same issue. My question to you is this, have you forked the elasticsearch repo and used this override or did you override this some other way?

[HonzaKral]

@vinitkumar you should never need to fork the repo - you can pass in your own serializer very simply:

from elasticsearch import Elasticsearch
es = Elasticsearch(..., serializer=JSONSerializerPython2())

[adamhadani]

👍 this caught me by surprise, solution offered here seems to work for Python 2.7.x, thanks!

[LucasBerbesson]

The JSONSerializerPython2 solution worked for me too ! thanks !
However I'm curious to know how I could call super() to encode the strings as bytes explicitly ?

[HonzaKral]

@LucasBerbesson you'd have to convert everything to unicode before passing the data into super(), otherwise you'd get the same issue.