Requests: POST converts request data from UTF-8 to ISO-Latin

[rhoerbe]

Posting a chunk of XML in utf-8 encoding arrives garbled (iso-latin) at the target server.
It should arrive as encoded.

Reproduction Steps

import requests

request = '''
<?xml version="1.0" encoding="UTF-8"?>
<Requ>
<GivenName>Özdemam</GivenName>
<FamilyNameGrünzl</FamilyName>
</Requ>
'''
print(request)
headers = {'Content-Type': 'text/xml; charset=utf-8', }
req = requests.Request('POST', 'http://192.168.1.5:8888/api',
                       headers=headers,
                       data=request)
prepped_requ = req.prepare()
s = requests.Session()
http_response = s.send(prepped_requ)

Running `nc -l 8888' will show the garbled Umlaut characters, whereas print or command-line curl will produce the expected result.

System Information

Tested on CentOS7/ Python 3.4.6 (EPEL) and OSX 10.12 MacPorts Python 3.5.3

Seems to be similar to #3476, but is unresolved.

[Lukasa]

You haven't encoded it. ;)

On Python 3 the native string type is unicode: it has no encoding. We have to auto-encode the data, and in practice that's done by httplib with no reference to whatever you put in your header field. Try changing your code to this:

import requests

request = '''
<?xml version="1.0" encoding="UTF-8"?>
<Requ>
<GivenName>Özdemam</GivenName>
<FamilyNameGrünzl</FamilyName>
</Requ>
'''.encode('utf-8')
headers = {'Content-Type': 'text/xml; charset=utf-8', }
req = requests.Request('POST', 'http://192.168.1.5:8888/api',
                       headers=headers,
                       data=request)
prepped_requ = req.prepare()
s = requests.Session()
http_response = s.send(prepped_requ)