Requests: Easy printing of Requests and Responses, two new utils?

[EmilStenstrom]

Several times now I've stumbled over cases where I would like to print the request and response objects as strings to the console. Simply to see when headers and content are sent. Seems more people than I have had this problem: http://stackoverflow.com/questions/20658572/python-requests-print-entire-http-request-raw

My suggestion: add two util methods that print requests and responses according to the HTTP spec. They are fully optional to use, and would not break backwards compatibility:

To be clear, I'm suggesting something link this be added to requests.utils:

def print_request(req):
    print('HTTP/1.1 {method} {url}\n{headers}\n\n{body}'.format(
        method=req.method,
        url=req.url,
        headers='\n'.join('{}: {}'.format(k, v) for k, v in req.headers.items()),
        body=req.body,
    ))

def print_response(res):
    print('HTTP/1.1 {status_code}\n{headers}\n\n{body}'.format(
        status_code=res.status_code,
        headers='\n'.join('{}: {}'.format(k, v) for k, v in res.headers.items()),
        body=res.content,
    ))

Is this a good idea?

[Lukasa]

@EmilStenstrom Generally this is a good idea, but we tend to oppose having 'utility' functions be merged into requests itself: we want to restrict the scope of the main library so that it remains really good at doing its specific roles.

For that reason, you should check out the requests toolbelt, which has exactly the utilities you want.

[sigmavirus24]

Didn't we just have another issue about this a couple days ago?

[kennethreitz42]

I actually wouldn't be against having something like this added directly to the PreparedRequest and Responses classes. I think that functionality is useful enough to warrant inclusion.

Would have to think hard about the name, though.

[kennethreitz42]

PreparedRequest.render(body=False) (defaults to True).

I would want it to be a near exact representation of what we're going to send across the wire (minus any post processing that occurs in a transport adapter, of course).

[Lukasa]

@kennethreitz How "near exact" is "near exact"?

Note, for example, that the PreparedRequest does not have a Host: header: this is because httplib attaches it for us. It is also missing some Accept-Encoding headers that urllib3 adds for us. The reality is that Requests does not have access to enough information to render this out unless we want to guess at what the rest of the stack will do, and if we do that then we'll start getting bugs raised when the output of render does not match the actual transmitted bytes.

[kennethreitz42]

That may be okay, as we'd be rendering a representation of the PreparedRequest object, not the request itself. It would have to be documented as such.

Part of me thinks this is a great idea, the other part has hesitation.

It would be nice to be able to print requests.get(...).request.render() to get a quick view of what was sent, rather than having to print a bunch of request attrs.

[Lukasa]

@kennethreitz I have no objection to rendering a representation of the preparedrequest object, but I suspect it's _extremely_ unwise to render in a structure that suggests completeness.

In particular, the structure given here (which renders out to a 'valid' HTTP/1.1 request/response) doesn't work in the following cases:

1. If using HTTP/2, when the request/response are structured entirely differently and made of binary frames.
2. If we send a file or send using chunked transfer-encoding, where we cannot render the body before we send it.
3. If we _receive_ data using chunked transfer-encoding, where we cannot accurately represent it because urllib3 has transparently removed the chunking.
4. If the request is sent via a proxy there's an extra CONNECT request we're simply not printing.

There's nothing wrong with wanting to have a printable representation of all the data on the PreparedRequest, but we should be very wary before we format that like a HTTP/1.1 request.

[kennethreitz42]

Agreed. I think we could make it _look like_ an HTTP/1.1 request/response, but clearly not be one. That would present the data in a friendly and familiar format, but would make it clear this is not a representation of the wire layer.

For example, instead of opening with HTTP/1.1, we'd open with REQUESTS/2.9.2.

[Lukasa]

I'm open to that direction.

[kennethreitz42]

Cool. I like this idea.

[EmilStenstrom]

Thanks for continuing the discussion and making a very incomplete idea into something awesome.

Is this something you think someone inexperienced with contributing to requests could work on? It looks fairly straightforward with my newbie eyes and I think some code could be inspired by requests_toolbelt. Should I start working on a pull request? Do you want to see something more of a spec first?

[kennethreitz42]

@EmilStenstrom what do you think of this? https://github.com/kennethreitz/requests/pull/3014