Elasticsearch-py: GZip compressed bulk indexing

[gcornut]

Hi,

Could it be possible to send GZip compressed JSON documents for bulk indexing using this python API?
I'm currently doing it using bash and curl but a python script would be better.

Thanks,
Guillaume

[HonzaKral]

Hi, gzip option is currently not supported. You could add the support yourself (just subclassing Urllib3HttpConnection and adding the gzip logic to perform_request essentially just wrapping the original method and adding gzip. Then just pass this subclass to the constructor of Elasticsearch - es = Elasticsearch(..., connection_class=GzipEnabledConnection).

Do you have any numbers on how efficient this is, if it's worth it we can explore the option of adding the capability to the library itself.

Thanks

[gcornut]

Well in my first tests with gzip compressed documents for bulk indexing I though I gained speed during HTTP transfer but I made a mistake and didn't index the same documents with or without compression. It turns out the the gain is negligible. It's only interesting if the connection to the remote ES server has limited speed (which is not my case).

[HonzaKral]

Yeah, in my tests the compression makes things slower, in both ways - from python to es and even just requesting a gzipped response by setting the appropriate headers.

This to me means that it is not a good candidate for inclusion into the client, especially since it's possible to achieve completely outside by just creating a custom connection class. Closing this ticket.

Thanks for the issue, please reopen if you (or anyone else) think it should indeed be included.

[ei-grad]

Sorry for digging into the old issue, but I just want to add my 50 cents here.

compression makes things slower

It is only true when the bottleneck is in client code. But in case of limited network capacity between bulk indexing clients and the ES cluster it is crucial to use the compression, which could reduce the size of bulk index requests by something like 10 times in many workloads related to logs and metrics, or 2 times in worst cases (it is still the json with its repeat-every-key-each-time syntax).

But anyway, if one wants to get around the client code bottleneck in bulk indexing process, then it would be a good idea to switch from python to another language, or at least keep away from elasticsearch-py which makes things more complicated than they are - I mean its http client and encoder abstractions with their default implementations, different bulk operation helpers and related configuration mess. It would took 20 lines of python code to get a clean implementation of the bulk indexing just for some concrete case, and it could reduce the python code overhead at least by 50%. Or it could be 50 lines of golang code, which would reduce the CPU usage by another 50% or more, depending on document composition logic.

So I'd basically agree that it is ok to have no parameter for client-side compression here.

[fxdgear]

@ei-grad

we added Gzip support already: https://github.com/elastic/elasticsearch-py/pull/704

But thanks for tagging this issue. Now the next person who comes looking will know that it's been supported now.