Requests: Can't set headers per part of multipart requests

[sholsapp]

I should be able to specify per-part headers for parts of a multipart requests. E.g., in the following examples, I should be able to specify different values for 'Content-Disposition' and 'Content-Type'.

Request generated with:

requests.post(url , files=dict(('file1', '{"example": 1}'), ('file2', '{"example": 2}')))

Request on wire:

POST /route/foobar HTTP/1.1
Host: sholsapp.com:11936
Content-Length: 682
Content-Type: multipart/form-data; boundary=586c8e04f40444d28160739615e0b7a7
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.1.0 CPython/2.6.6 Linux/2.6.32-131.4.1.el6.x86_64

--586c8e04f40444d28160739615e0b7a7
Content-Disposition: form-data; name="file1"; filename="file"
Content-Type: application/octet-stream

{"example": 1}
--586c8e04f40444d28160739615e0b7a7--
Content-Disposition: form-data; name="file2"; filename="file2"
Content-Type: application/octet-stream

{"example": 2}
--586c8e04f40444d28160739615e0b7a7

I can't see an obvious way to do this using requests. Does anyone know how to accomplish this?

_Would anyone be open to reviewing a patch that would add this functionality? If so, what things do they want to see considered, what API exposed, etc.?_

Thanks!

[Lukasa]

Yeah, you can do this (sort of), but it's not a very clearly documented feature.

We don't allow you to set _arbitrary_ values of those headers, but we do allow you to change the name, filename and content-type values, like so:

>>> import requests
>>> files = {'file1': ('filename', 'data', 'text/plain'), 'file2': ('otherfilename', 'data2', 'text/other')}
>>> r = requests.post('http://httpbin.org/post', files=files)
>>> r.status_code
200

On the wire, this looks like:

POST /post HTTP/1.1
Host: httpbin.org
Content-Length: 318
Content-Type: multipart/form-data; boundary=e1b0cf4b5e114bf088118fc0bbf4ee4c
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.2.3 CPython/2.7.2 Darwin/12.4.0

--e1b0cf4b5e114bf088118fc0bbf4ee4c
Content-Disposition: form-data; name="file2"; filename="otherfilename"
Content-Type: text/other

data2
--e1b0cf4b5e114bf088118fc0bbf4ee4c
Content-Disposition: form-data; name="file1"; filename="filename"
Content-Type: text/plain

data
--e1b0cf4b5e114bf088118fc0bbf4ee4c--

Does that provide the functionality you wanted?

[sholsapp]

Almost, but not quite. To work with the system I'm communicating with, I would need the ability to set the "Content-Location" header per part. Is there a way to set that header?

I figured we could abstract this feature out and allow one to add _any_ headers they would want per part, but it's not immediately obvious to me how that feature would be designed. I haven't seen the part of code that allows me to do what you showed, but I suppose you could extend it to allow a headers dict, too.

[Lukasa]

The answer there is no. =) I also sincerely doubt there is going to be enthusiasm for adding it to the API. Each addition to the API is an increase in complexity and needs to be strongly justified by a compelling (and reasonably common) use-case. I'm prepared to be convinced, but I doubt such a use-case exists here.

[sholsapp]

I think at a minimum the ability to change the "Content-Location" header is a must-have for library writers. I also like most of the arguments in http://tools.ietf.org/html/rfc2557.

I agree it is probably not a super popular use-case... I'll do some more digging and see if there would be a non-obtrusive way of adding the feature.

[Lukasa]

You should feel free to investigate, but the friction involved here is high. We delegate our building of multipart-encoded bodies to urllib3 via this function call here. That function is defined here in urllib3. Any change you wanted to make would need to be implemented in urllib3 first, and would then be exposed (or not) in Requests.

As for whether or not changing Content-Location is a must-have, I'm not totally sure I agree: after all, this is the first time we've been asked for it! =D

[sholsapp]

Thanks for the links. Being that urllib3 isn't flexible enough to support this, either, I think I'll have to look into rolling my own request for this thing. It's a shame, requests was _soo_ close.

Or I'll go bug the urllib3 people now. =)

[sigmavirus24]

@sholsapp is it possible to close this? You can open a pull request if and when you get these features accepted into urllib3. You can still reference this in that pull request but I cannot guarantee this will be accepted either way.

[sholsapp]

Sure, yes, we can close this. I'm not sure if I'll continue using requests for this task so if there is uncertainty over whether people want this feature or not, I'll not send a pull request.

[Lukasa]

The challenge is always to provide an implementation that does not add complexity to the mainline (standard multipart file uploads), _and_ that is intuitive for the unusual case. If you can provide such an API, you've got a much better chance of getting it accepted. =)

[sharpshadow]

In my opinion it is crucial to offer a full useable way to add any headers to the request.
I have the same problem and it's very anoying.

At least you guys could display a solution to archive this without adding it to the stable release.
I would appreciate it.

[Lukasa]

@sharpshadow You can do this today, and if you'd looked around a little bit longer: say, at the urllib3 issue linked above, or at the docs, you'd have found that out.

Files takes a dictionary of keys to tuples, with the following meanings of each place in the tuple:

# 1-tuple (not a tuple at all)
{fieldname: file_object}

# 2-tuple
{fieldname: (filename, file_object)}

# 3-tuple
{fieldname: (filename, file_object, content_type)}

# 4-tuple
{fieldname: (filename, file_object, content_type, headers)}

[sharpshadow]

@Lukasa
Thank you very much. As mostly it was not the problem of missing ressources or bad dokumentation.