Requests: Support method to gzip or otherwise compress HTTP request body?

[dkador]

It doesn't appear that requests currently supports any built-in mechanism for automatically compressing a given HTTP body payload (e.g. JSON). Correct me if I missed something terribly obvious here, and apologies if I did.

I could certainly just implement the compression part myself, but I was wondering if the requests community thinks that a feature like this would be useful in general or is too specific to each integration's problem domain and so should remain there?

If people think this sounds interesting I'd be happy to take a stab at implementing it. Thanks!

[sigmavirus24]

I think it is a good idea to at least have methods for doing this documented (if not packaged in a separate library) but I'm pretty sure it doesn't belong in requests proper.

[Lukasa]

Agreed. The fact of the matter is that it isn't very hard to compress a request body yourself, and it's hard to see how we'd have a nice API for it. I'm fairly sure we don't need a separate library for it because of how trivial it is, but I'm not averse to documenting it (maybe informally):

import zlib
import json

data = {'some': {'json': 'data'}}
postdata = zlib.compress(json.dumps(data))
r = requests.post('http://posturl.com/', data=postdata, headers={'Content-Encoding': 'gzip'})

[ionrock]

It might be worthwhile to include a GzipAdapter:

import gzip
import zlib
import json

import requests

from requests.adapters import HTTPAdapter

from pprint import pprint


class GzipAdapter(HTTPAdapter):

    def add_headers(self, request, **kw):
        request.headers['Content-Encoding'] = 'gzip'

    def send(self, request, stream=False, **kw):
        if stream is True:
            request.body = gzip.GzipFile(filename=request.url,
                                         fileobj=request.body)
        else:
            request.body = zlib.compress(request.body)

        return super(GzipAdapter, self).send(request, stream, **kw)


if __name__ == '__main__':
    sess = requests.Session()
    sess.mount('http://', GzipAdapter())

    msg = {
        'msg': 'hello world',
    }

    resp = sess.put('http://httpbin.org/put', data=json.dumps(msg))
    pprint(resp.json())

I haven't tested this very extensively, but if it seems like a helpful tool, an adapter seems like a nice place to put it. I'd be happy to take a more in depth stab at it if people are interested.

Just throwing it out there.

[Lukasa]

You should absolutely write one, but Requests core won't include it. =)

[sigmavirus24]

You should absolutely write one, but Requests core won't include it. =)

This.

[tapionx]

I landed here looking for a way to gzip a POST payload using requests.
Since requests is HTTP for human being, I think that it should include a parameter to handle gzip automatically

[sigmavirus24]

Thank you for your opinion @tapionx but given the nearly 5 years from the first suggestion o fthis until now, I don't think this is actually worth the surface area to the core library. Further, adding a parameter implies an expansion of a Frozen API - one which has been frozen (successfully) for years. The library that tends to include these patterns outside of the core, requests-toolbelt is happy to include this if someone wants to write it.

[cmin764]

I would advise more an adapter like this:

COMPRESSION_SCHEMES = [
    "http://",
    "https://",
]


class CompressionAdapter(HTTPAdapter):

    """Adapter used with `requests` library for sending compressed data."""

    CONTENT_LENGTH = "Content-Length"

    def add_headers(self, request, **kwargs):
        """Tell the server that we support compression."""
        super(CompressionAdapter, self).add_headers(request, **kwargs)

        body = request.body
        if isinstance(body, bytes):
            content_length = len(body)
        else:
            content_length = body.seek(0, 2)
            body.seek(0, 0)

        headers = {
            Compression.CONTENT_ENCODING: Compression.GZIP,
            Compression.ACCEPT_ENCODING: Compression.GZIP,
            self.CONTENT_LENGTH: content_length,
        }
        request.headers.update(headers)

    def send(self, request, stream=False, **kwargs):
        """Compress data before sending."""
        if stream:
            # Having a file-like object, therefore we need to stream the
            # content into a new one through the compressor.
            compressed_body = io.BytesIO()
            compressed_body.name = request.url
            compressor = gzip.open(compressed_body, mode="wb")
            # Read, write and compress the content at the same time.
            compressor.write(request.body.read())
            compressor.flush()
            compressor.close()
            # Make available the new compressed file-like object as the new
            # request body.
            compressed_body.seek(0, 0)    # make it readable
            request.body = compressed_body
        else:
            # We're dealing with a plain bytes content, so compress it.
            request.body = gzip.compress(request.body)

        return super(CompressionAdapter, self).send(
            request,
            stream=stream,
            **kwargs
        )


def get_session():
    """Get a requests session supporting compression."""
    # Prepare the adapter and the session.
    compression_adapter = CompressionAdapter()
    session = requests.Session()

    # Mount the adapter to all affected schemes.
    for scheme in COMPRESSION_SCHEMES:
        session.mount(scheme, compression_adapter)

    # Use this sessions for CRUD-ing.
    return session