Requests: sending post request data nested object

[wiryonolau]

Unable to send post data from nested object, some of the data is truncated

payload = {
   "config": {
      "data" : "test"
   }
}

r = request.post('http://httpbin.org/post', data=payload)
print(r.text)

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "config": "data"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "11", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.8.1"
  }, 
  "json": null, 
  "origin": "xxx.xxx.xxx.xxx", 
  "url": "http://httpbin.org/post"
}

[Lukasa]

What are you expecting the result to be here?

[wiryonolau]

the post data that send become config : data instead of the full object

[Lukasa]

I'm sorry, I wasn't clear enough.

When you use the data keyword, we encode the data using HTTP form-encoding. This is not capable of representing nested data structures, only flat ones. So my question is: what are you trying to achieve, exactly? Because what you're asking requests to do does not make any sense.

[wiryonolau]

If I urlencode the payload it appear like this

config=%7B%27data%27%3A+%27test%27%7

which is what should I get on the server

[Lukasa]

So you literally want to represent the config as a urlencoded JSON string?

[wiryonolau]

Ah sorry shouldn't use urlencode, in php they have http_build_query that convert the payload become like this

config%5Bdata%5D=test

[Lukasa]

@Wiryono That does not match what the urlencoded payload you copied decodes to. In fact, your urlencoded payload corresponds to config={'data': 'test'}, which is a JSON-serialized representation of the nested data structure. To get that, you'd want to use:

import json

payload = {
   "config": json.dumps({
      "data" : "test"
   })
}

r = request.post('http://httpbin.org/post', data=payload)
print(r.text)

That will work appropriately.

[Lukasa]

Ok, so to represent that in requests you want to use:

payload = {
   "config[data]": 'test',
}

There is no consistent, standardised way to represent nested data in form-encoding, and PHP just happily invented one. Requests doesn't guess what you want here, you need to come up with a conversion function yourself I'm afraid.

[sigmavirus24]

See also https://github.com/sigmavirus24/requests-toolbelt/issues/45 (which I hope to have time to tackle soon).

[martimlobao]

The solution is non-obvious, but all you have to do is change the data argument to json:

r = request.post('http://httpbin.org/post', json=payload)

It's not clear to me why the two arguments exist when they seem to be functionally identical. At least, a warning should be given to the user when trying to send a nested dictionary with the data argument (see #5058).

[sigmavirus24]

They're not functionally identical in the slightest, otherwise both would just work. Please read the documentation.