Got: JSON option when posting data

[paglias]

Would it be possible to use the json: true option when posting data to set Content-Type to application/json and stringify the body with JSON.stringify?

[floatdrop]

@paglias it is possible, but json option is responsible only for toggle parsing response from server and it will be kinda major change to make it stringify the body for request also.

[spencerhakim]

It would be a major change, but not necessarily a bad one. At least having the option to enable it would be nice. Like, if json is an object instead of just true seems like a reasonable compromise.

_Edit_ - Tagging @sindresorhus, since it's his project

[sindresorhus]

@spencerhakim Can you provide a code example of how it would look like?

[sindresorhus]

Reopening, but we're not likely to do another major in the near future.

[spencerhakim]

Current way:

var options = {
  json: true, // if truthy, parse *response* as JSON
  headers: {
    'Content-type': 'application/json'
  },
  body: JSON.stringify({ // have to manually stringify object in body
    name: 'whatever',
    category: 'toy',
    description: 'Hello World'
  })
};

got.post(`http://example.com/api/createThing`, options)
  .then((resp) => {
    // ...
  });

Possible new way (should be functionally equivalent to the above):

var options = {
  json: { // typeof json === 'object', not 'boolean', so send body as JSON instead of form-encoded
    name: 'whatever',
    category: 'toy',
    description: 'Hello World'
  }
};

got.post(`http://example.com/api/createThing`, options)
  .then((resp) => {
    // ...
  });

Coming from request, I was surprised that got didn't follow the same semantics for handling JSON or offer any way of automatically serializing sent objects as JSON. This approach should be backwards compatible with current usage, so I don't think it requires a major version bump.

[sindresorhus]

Wouldn't it make more sense to pass the object literal in body?

[spencerhakim]

Yes, but that approach would likely require a major version bump.

[bishtawi]

@sindresorhus What about making the json attribute an object that takes req and res as keys. Setting req to true means the request body should be JSON encoded, setting res to true means the response body should be JSON parsed.

The current json attribute being a boolean could be deprecated but not removed till the next major rev bump. Setting json to true would be translated to { req: false, res: true}. Omitting the json attribute or setting it to false would translate to { req: false, res: false}. This would keep backwards compatibility while easily supporting a way to JSON encode the request body.

var options = {
  json: {
    req: true,
    res: true
  },
  body: {
    name: 'whatever',
    category: 'toy',
    description: 'Hello World'
  }
};

The above example would JSON encode/decode the request/response body.

[sindresorhus]

Not interested adding a temporary workaround when it's possible for you to stringify it yourself. We'll add support for passing object literal in body when json: true in the next major release.

[floatdrop]

@sindresorhus now passing object literal in body work as serializing it as application/x-www-form-urlencoded – wouldn't it make json option a bit confusing?

[sindresorhus]

@floatdrop I don't really see the confusion. json: true makes it clear it has different behavior. Just need clear docs.

request supports the same:

body - entity body for PATCH, POST and PUT requests. Must be a Buffer, String or ReadStream. If json is true, then body must be a JSON-serializable object.

[floatdrop]

@sindresorhus it is clear, that json: true will change behaviour, but not in obvious way – without it Object in body was serialized as form, but with json: true it morphing to JSON object, which is not expected in many usecases (for example API that accepts form in body, but returns JSON).

Maybe it is better way to choose serialization based on content-type header?

[sindresorhus]

for example API that accepts form in body, but returns JSON

Hm, I didn't think of that use-case.

request has a separate form option for posting an object serialized as application/x-www-form-urlencoded. I think that would be a less surprising way. Since the JSON thing would be a breaking change, it might worth doing this too.

[bisubus]

@sindresorhus The flexibility seems to be appropriate in this case. Switched to got from superagent recently. The latter has got 3 different methods (type, accept, parse) for setting serialization type, accept header and forced parsing.

[MarkHerhold]

I just converted from request-promise and had the same need as others here. request-promise would automatically stringify JSON bodies and set the appropriate header. I solved this by wrapping got with a local module and doing the same. Here's a snippet from my wrapper:

Stringify JSON Objects and Arrays and set Content-Type

    // stringify object and array bodies
    if (options.json === true && options.body && (typeof options.body === 'object' || Array.isArray(options.body))) {
        options.headers = options.headers || {};
        options.headers['Content-Type'] = 'application/json';
        options.body = JSON.stringify(options.body);
    }

[alextes]

Since this is a very common use case in my experience and the json flag works very differently from request/request which is the dominant http / request lib. Similarly visionmedia/superagent doesn't require the coder do much thinking with regards to JSON posts.

I think many coders will require the same 5 - 10 min of messing about before they realise you do it manually with got. Perhaps even running into an unexpected error.

Adding documentation for this common use case makes a lot of sense to me. I think the cost is low, even if only considering the benefit to the maintainers not having to deal with people creating issues like this. Would a PR be welcome?

Unrelated but the same question for adding an index to the documentation. PR welcome? ^^

Thanks for creating what will undoubtedly be my new http lib goto.

[MarkHerhold]

Alex, I think you are on point. I too wrote a wrapper around got for
serializing JSON bodies.

[sindresorhus]

PR welcome for the change described in https://github.com/sindresorhus/got/issues/174#issuecomment-227001985. Would be nice to have this included in the next major version.

[alextes]

After giving the code a thorough read (so concise<3) I can say I would love to take a stab at it @sindresorhus. I also feel I might be out of my depth with all the streaming going on. I'm hoping the greatest cats in nodejs won't mind that.

Maybe good to summarize.

• add to current json flag effect / body behaviour.
  
  
  
  • If body is a plain object, it will be stringified with querystring.stringify and sent as application/x-www-form-urlencoded.
  
  
  • If json is true, and body is a plain object, it will be stringified with JSON.stringify and sent as application/json.
  
  

This poses a problem for people wanting to send a querystring but receive json and have it parsed. Which is to be solved as follows.

• add form option
  
  
  
  • type: object(maybe string)
  
  
  • takes a plain object which will be stringified with querystring.stringify and sent as application/x-www-form-urlencoded.
  
  

This does leave the hairy situation where a user sets both json and form and then passes a plain object or string as the body. Since content types are mutually exclusive (I think) what happens?
This is assuming the default body behaviour is to stay the same.

Maybe you're going on the assumption querystring and json would both require explicit setting to affect the request. In that case, what is to be the default behaviour of body? Pass to req.end? What happens when passing an object which node doesn't support?

Final question, what about people wanting to send json but not receive it? Many will use json flag as they did with other libs, and perhaps be surprised when an error is thrown because we tried to parse their plain text response. These are probably few and far between. Still, asking the user to set a header and stringify the body themselves because you want to control your parsing on receiving does not seem straightforward.

Again, if it isn't too much trouble answering, I would _love_ to take a stab at this PR ^^

[MarkHerhold]

There are a lot of valid what-if questions here. My fear is that we implement this or another approach, someone ends up not liking it, then we add another set of options to accommodate that use case, and soon the got becomes 500k lines long, competing in the got vs request bloat competition. :laughing:

Alternatively, we could devise a plugin architecture, possibly via beforeRequest()/afterRequest() hooks, making this PR a plugin that ships with got, but not enabled by default. This would allow others to write their own JSON plugin that fits other use cases without bloating the core.

[alextes]

Hm. I think considering to generalize this case makes sense @MarkHerhold. I've also got to add that's a bigger PR than I think I can take on, and wouldn't without the cats weighing in, and they seem busy. I think I'll simply open a PR, making the most straightforward choices I can. I like the general idea of having optional pre / post request processing. I wouldn't know the best way to implement it.

[alextes]

I thought about this quite a bit when making my first attempt at implementing and now am pretty convinced it's a bad idea. I think having the 'shortcut props' makes more sense. So form, lets you easily send application/x-www-form-urlencoded, maybe formData for multipart/form-data, JSON for application/json. That leaves parsing freed up to be a wholly separate thing. Creating a clear distinction between serializing and deserializing which we shouldn't mix I'd say. Most of the inspiration for this thinking comes from looking around, mostly at superagent. A small bonus is that you can start treating deserializing manually by specifying a parsing function or you can start looking at the content-type in the response headers for default behavior parsing which would make a lot of sense too.

[sindresorhus]

I feel we're overthinking it a bit here and trying to consider too many imaginary edge-cases. The JSON option is just meant to cover the common case. If someone wants more advanced configurability they can just not use it.

Proposal:

• If json: false, body will be required to be a string, buffer, readableStream.
• If json: true, body will be required to be a plain object and will be JSON stringified and correct headers set and response will be JSON parsed.
• If form: true, body will be required to be a plain object and will be stringified with querystring.stringify and sent as application/x-www-form-urlencoded. If json: true, only the response will be parsed.

I'm open to suggestions on how we could simplify this further.

[alextes]

Fair enough Sindre.
I'll PR the above. It would make any switchover to got a lot smoother.
Just to be clear, your proposal means dropping the default querystring serializing of object values passed as the body, like I proposed in #265 right?

[alextes]

I want to suggest we support all inputs that querystring.stringify and JSON.stringify support, or maybe their shared subset.

[sindresorhus]

your proposal means dropping the default querystring serializing of object values passed as the body, like I proposed in #265 right?

👍

I want to suggest we support all inputs that querystring.stringify and JSON.stringify support, or maybe their shared subset.

I'm not sure what this means. Can you elaborate?

[alextes]

I'm not sure what this means. Can you elaborate?

I like when things work the way I imagine them to work. When I use a request library, specify a body, and ask to stringify the body, I imagine it:

• accepts any input the underlying stringify fn accepts
• (bonus) is helpful and refuses to stringify something that might be accepted by the stringify fn but isn't what I meant to do.

Current proposal: only accept a plain object.
querystring.stringify accepts everything:

> qstr('alex')
''
> qstr(24)
''
> qstr(['alex'])
'0=alex'
> qstr({name: 'alex'})
'name=alex'
> qstr({name: null})
'name='
> qstr(null)
''
> qstr(undefined)
''

JSON.stringify accepts everything (sorta):

> jstr('alex')
'"alex"'
> jstr(24)
'24'
> jstr(['alex'])
'["alex"]'
> jstr({name: 'alex'})
'{"name":"alex"}'
> jstr(null)
'null'
> jstr(undefined)
undefined

Proposals:

1. Just support their common subset. Keep the API simple. body validation wouldn't depend on which of the flags you're using. Their common subset would be (Array and Object).
2. Stringify what the Node supports but _don't stringify null or undefined_, the user probably meant 'don't send a body'.
3. Don't stringify null or undefined. Stringify what makes sense for each. Querystring that's Array and Object. (shh, don't worry about deep objects). For JSON that's Number, String, Array, and Object. I for one do use the plain values in my API. Example: add ten to a user's credits, or shift the position of an item in a list.

I vote 3 🙋.

Sidenote: wouldn't mind leaving this out of the milestone.

[sindresorhus]

I vote 1. I'm ok with also supporting an array, but querystring.stringify is IMHO too loose.

I for one do use the plain values.

Really? I've never seen that before.

[alextes]

Alright, I'll do 1. but when the people come knocking for us to be less opinionated and support what is valid JSON, I'm with them ✊.

[sindresorhus]

@AlexTes That's actually a good way to do development. Let's call it Complaint-Based-Development. It's so easy to try to solve every imaginary use-case and get bloated. Implementing the minimum and rather change later if many people complain means you only implement what most people need.

[alextes]

@sindresorhus Complaint-Based-Development 😁. Maybe call it Wish-Based-Development ✨ instead 🙏. Fully agree with all your points!

[ash0080]

Not interested adding a temporary workaround when it's possible for you to stringify it yourself. We'll add support for passing object literal in body when json: true in the next major release.

@sindresorhus So how to make a request with json content but response is a html?

[szmarczak]

const instance = got.extend({
    hooks: {
        beforeRequest: [
            options => {
                options.body = JSON.stringify(options.body);
            }
        ]
});

instance(url, options);

[alextes]

I'd go with something simpler 😊:

got(url, {
    headers: {
        'content-type': 'application/json'
    },
    body: JSON.stringify(body);
});

[szmarczak]

Custom instances are even simpler ;) No need to set headers every time or stringify the body :P

[alextes]

Alright, alright 😄 . Still like this version better in that case. No need to understand got's request lifecycle. Although, you got to hand it to you (pun intended?), not having to stringify the body is pretty cool 😎 .

const jsonGot = got.extend({
    headers: {
        'content-type': 'application/json'
    }
})
jsonGot({
    url,
    body: JSON.stringify(post)
})

[ash0080]

Excuse me? what u discussed is requesting a json, but what I asked is post a json, responsed is html :joy:

[szmarczak]

So how to make a request with json content but response is a html?

As far as I'm good with English, IOW you said to make a request which body is JSON, and the response body is HTML. So I think you're confused here :)

[alextes]

Ah, we should've been a bit clearer @ash0080 .

got(url, {
    method: 'POST',
    headers: {
        'content-type': 'application/json'
    },
    body: JSON.stringify(body);
});

However, got is quite smart. It understands that if you're adding a body, you probably meant to post, and will set the method to POST all by itself, unless explicitly instructed otherwise.

[ash0080]

Ahhha~ Sorry, I am messed up. what we discussed is right.
Sorry for my chaotic mind :joy: