Msw: Support alternative JSON library

[JesusTheHun]

Is your feature request related to a problem? Please describe.

This is both a feature request and a bug report. Consider the following :

import { rest } from 'msw';
import { setupServer } from 'msw/node';
import axios from 'axios';
import JSONbig from 'json-bigint';

// This is required by Jest to work properly
BigInt.prototype.toJSON = function() { return this.toString(); };

const mockBaseUrl = "http://localhost/services/acme/api";

const customer = {
  "username": "Dude",
  "balance": BigInt(1597928668063727616) // yes, the Dude is rich
};

const server = setupServer(
  rest.get(`${mockBaseUrl}/customers/me`, (req, res, ctx) => {
    return res(
      ctx.status(200),
      ctx.json(customer),
    )
  }),
)

beforeAll(() => server.listen({ onUnhandledRequest: 'error' }))
afterEach(() => server.resetHandlers())
afterAll(() => server.close())

const apiAxios = axios.create({
  transformResponse: [data => data ? JSONbig.parse(data) : data],
});

describe("Mocks", () => {

  it("should mock a basic GET call", () => {
    const url = `${mockBaseUrl}/customers/me/`;

    return expect(apiAxios.get(url).then(r => r.data)).resolves.toEqual(customer);
  });
});

This test result in the following error :

Error: expect(received).resolves.toEqual(expected) // deep equality

- Expected
+ Received

  Object {
-   "balance": 1597928668063727616n,
+   "balance": "1597928668063727616",
    "username": "Dude",
  }

Describe the solution you'd like

One simple solution would be to be able to set the JSON library to use while transforming the response, as it looks like you internally transform to json.
In my case the common library to handle BigInt is json-bigint.

Describe alternatives you've considered
I'm still looking into creating my own response transformer to handle BigInt but there is no documentation about it so it's quite time consuming.

[kettanaito]

Hey, @JesusTheHun. Thanks for reporting this.

Basically, this could be solved with a custom response serialization function, do I get it correctly? Can you give me an example of an API you'd like MSW to expose?

[JesusTheHun]

Hi @kettanaito, thank you for the fast response.

You are correct.
Since this kind of setup is to be applied application wide, I think the best approach is to expose a method at the prototype level:

msw.setSerializer(body => JSONbig.stringify(body));
// or
msw.config({
  serializer: body => JSONbig.stringify(body);
})

Or if you want to keep the prototype untouched, which I can understand, return the configured object like axios does :

const myMsw = msw.create({
  serializer: body => JSONbig.stringify(body);
});

[kettanaito]

On the contrary, I think such serialization should be configurable per-response. Otherwise you'd have to opt-out of it in certain responses, as opposed to opt-in, which would be more explicitly and thus more straightforward for the codebase maintainers.

This most likely should be a custom res composition function with the proper serialization built-in.

const customRes = () => {...}

// In a request handler
return customRes(ctx.json(...))

I was thinking about a few factors to enable such feature:

1. Add a support to opt-out of the built-in response body serialization. This way you can tell a certain response to ignore the built-in JSON.stringify.
2. Add an API on the res object to supply a custom serializer/deserializer.

Now here's where challenge arises: one cannot pass a function (deserialization function) to the worker. At the same time the entire response needs to be stringified (serialized) in order to be sent to the worker to respond with it. This means that although you would be able to serialize a response body, on the worker's side JSON.parse would likely fail, or produce a mismatch.

[marcosvega91]

I was thinking about this yesterday. Are we sure that the worker could cause problem?

1) The body will be serialized JSONbig.stringify(body) by a custom response function
2) The response will be serialized to be sent through channel with JSON.stringify(response)
3) The message received through the channel will be unserialized with JSON.parse(message).

In the last step the body should be the same at first step, or not?

[kettanaito]

@marcosvega91, while may work in this particular use case, I was suggesting a more versatile serialization customization. As in serialize/deserialize a mocked response however you want. Can we be sure that arbitrary serialization/deserialization logic would play nicely with the worker?

The issue here is that the worker is the place where we deserialize the stringified mocked response from the client. If deserialize is even an option in the configuration, I don't see a viable way to propagate that option (function) to the worker to execute.

Do you see my concern here?

[JesusTheHun]

I strongly disagree with the opt-in approach. As an architect when you lay a block of code it is for a good reason, you know and control the impact for your fellow developers. They are either informed or do not need to know. In this case, the new serialization function is a drop-in replacement that does not come with any drawback and is here to handle a native JS object. There is no reason to opt-out. Opt'ing-out from this custom serializer is actually a mistake and will likely leads to error throughout the app.
Now I can understand that one may want to set a custom response handler for a particular response, to handle base64 or binary content for example, but this is an additional case to handle, not a case that superset the others.

Anyway, having to create a "global" setter and a local one is actually not a big deal so we could totally do both.

For your worker issue, we could use dynamic import to load the custom deserialization function. You could be something like :

// my_custom_module.js
export const serialize = (d) => JSONbig.stringify(d);
export const deserialize = (d) => JSONbig.parse(d);

// user land
const server = setupServer(/* ... */)
server.registerSerializationModule('a-unique-name', './my_custom_module.js');

// msw internal
serviceWorker.controller.sendMessage({
  cmd: 'REGISTER_SERIALIZATION_MODULE',
  payload: {
    name: moduleName, // a-unique-name
    path: modulePath, // ./my_custom_module.je
  }
});

// worker land
self.addEventListener('message', async function (event) {
  // ...
  switch (event.data.cmd) {
    // ...
    case 'REGISTER_SERIALIZATION_MODULE': {
      import(event.data.payload.path).then(module => {
        try {
          registerSerializationModule(event.data.payload.name, module)
          sendToClient(client, {
            type: 'SERIALIZATION_REGISTERED',
            payload: true,
          })
        } catch (e) {
          // name is already taken
          sendToClient(client, {
            type: 'SERIALIZATION_REGISTERED',
            payload: false,
          })
        }
      }).catch(err => {
        // path not found
        sendToClient(client, {
          type: 'SERIALIZATION_REGISTERED',
          payload: false,
        })
      });
      break
    }
    // ...
  }
})

// user land again
// app wide
server.setDefaultSerializationModule('a-unique-name');

// opt-in
rest.get(`${mockBaseUrl}/customers/me`, (req, res, ctx) => {
  return res(
    ctx.status(200),
    ctx.json(customer, { serializationModule: 'a-unique-name'}), // maybe not ideal, I don't know the msw api
  )
});

[marcosvega91]

Yes I understand the point. When you talk about client you mean the browser or the handler?

[kettanaito]

@JesusTheHun one of the core design principles of the library is for the worker script to remain as small as possible, containing only the most crucial functionality. While dynamic imports may work, I'm sure that a decent portion of users won't even use this feature. In other words, the worker script shipped by the library should contain only the logic used by every user of the library.

That is why the current worker script mainly contains internal implementations, like self-destroying the worker, supporting per-client mocking, etc. In its majority that code is not even directly consumer-facing. Adding a serialization message handling and a dynamic import looks like a huge architectural overkill to satisfy a single use case.

That being said, the worker script is open to modifications and you are welcome to modify it to suit your usage. I hope you understand our precautions of adding feature-specific implementations to the worker in general.

Would the suggestion from @marcosvega91 work in your case?

[kettanaito]

In the past, ctx.json performed the response body serialization. We've moved away from this internally to support merging of multiple JSON chunks of the response body (#403). Now response serialization is in the res function, done at the very end of the response composition chain as a built-in transformer.

I find the recent change sensible, as context utilities may affect how the response is constructed and handled. This means that the serialization shouldn't be provided to the ctx.json, or any other context utility, but specified either on the res function level, or the entire worker/server instance.

Using the res composition

This looks like the best option to put the serialization into, as with a custom serialization you describe how the _response_ should be serialized.

How I see this feature:

import { response } from 'msw'

const customRes = (transformers) => {
  // Pseudo-code, somehow opt-out of the default usage of "JSON.stringify"
  response.optOutOfDefaultSerialization()

  // Apply the response transformers (compose the mocked response)
  const originalResponse = response(...transformers)

  // Provide custom response body serialization
  originalResponse.body = JSONBig.stringify(originalResponse.body)

  return originalResponse
}

import JSONbig from 'json-bigint';
import { customRes } from './customRes'

rest.get('/data', (req, _, ctx) => {
  return customRes(ctx.json({ balance: BigInt(1597928668063727616) }))
})

Using the worker/server instance

Providing custom serialization to these instances may feel right, as they represent something global, but from the API perspective I think it's not the best solution. These instances represent the layer of interception itself, not particular request/response handling (_that_ is controlled by request handlers that utilize res composition).

Configuration to these instances should control how the _interception_ behaves. For example, adding new handlers, erroring on unhandled requests, but not affecting the request/response.

[marcosvega91]

@kettanaito I was thinking about this. All MSW logic will be inside the createResponseComposition. With your function you can manipulate the response as you wish

import { setupWorker, rest, createResponseComposition } from 'msw'
import JSONbig from 'json-bigint'

const customReponse = createResponseComposition((res) => {
  if (res.body && res.headers?.get('content-type')?.endsWith('json')) {
    res.body = JSONbig.stringify(res.body)
  }

  return res
})

const worker = setupWorker(
  rest.get('/me', (req, res, ctx) => {
    const me = {
      username: 'Dude',
      balance: BigInt(1597928668063727616),
    }
    return customReponse(ctx.json(me))
  }),
)

worker.start()

[JesusTheHun]

@marcosvega91 with this approach I would need to set the body to a string with ctx.body(JSONbig.stringify(customer)) and use a custom response function will deserialize the string response again. Did I understand correctly ?

@kettanaito I understand you want to keep things simple. I presented you a real production use case for a native javascript object.
If you are working with an API that send a long and do not use json-bigint, JS will silently fail every operation you do with that data.
Using json-bigint when working with 3rd party API is the only right way to do it.

I see msw as the new standard when it comes to HTTP mocking. I would be very sad if it does not support the javascript state of the art.

Please, reconsider your position on this matter.

[JesusTheHun]

@marcosvega91 your latest suggestion works for me. From a DX perspective I don't like to use a custom function everywhere when it could be set in one place but at least it covers the case !

[kettanaito]

@JesusTheHun, to be clear, I'm not against finding an API to suit this use case. I just want to find a good API to do that. With enough dedication and discussion we will find the proper solution.

[kettanaito]

Marco's suggestion is great, but it still doesn't cover the deserialization issue. However, when speaking of custom serialization function that's also how I see it working. I understand that having to use a custom response function may feel strange, but if you give it time you see it granting you much more control than some option you feed to the library.

[marcosvega91]

deserialization should be covered by user. Maybe I'm wrong but for example using fetch you could do this

const res = await fetch('http://MY_CUSTOM_API/me')
const body = JSONbig.parse(await res.text())

[kettanaito]

Since response body is text, I suppose it's safe to assume that a custom serialization function will always produce text. Then, calling JSON.parse on it within the worker script can also be assumed as safe. @marcosvega91, you make a perfect point that the worker doesn't actually care about the response body, but the client consuming the response does.

[marcosvega91]

Yes, even if the worker will parse the body it should work anyway.

I have created a test, it seams working

[kettanaito]

Note that createCustomResponse doesn't behave as you've shown in the example above. It accepts an initial state of the response and returns a response composition function to alter that state:

https://github.com/mswjs/msw/blob/c9855a88ba41a8ae757c297312399e0bc8c6f3d1/src/response.ts#L49-L71

A custom response composition function would look like this. There's no way to opt-out of the default JSON body serialization as of now. However, would that even be necessary?

[marcosvega91]

Yes, I was thinking about this.

export function createResponseComposition(
  responseTransformer: ResponseTransformer = stringifyJsonBody,
  overrides: Partial<MockedResponse> = {},
): ResponseFunction {
  return (...transformers) => {
    const initialResponse: MockedResponse = Object.assign(
      {},
      defaultResponse,
      {
        headers: new Headers({
          'x-powered-by': 'msw',
        }),
      },
      overrides,
    )

    const resolvedResponse =
      transformers.length > 0
        ? compose(...transformers)(initialResponse)
        : initialResponse

    return responseTransformer(resolvedResponse)
  }
}

export const response = Object.assign(createResponseComposition(), {
  once: createResponseComposition(stringifyJsonBody, { once: true }),
  networkError(message: string) {
    throw new NetworkError(message)
  },
})

You can pass your own responseTransformer function . By default stringifyJsonBody will be used

[kettanaito]

The responseTransformer argument is essentially the same function as you would supply to the returned composition function (in ...transformers), which makes the arguments somewhat reversed:

function createResponseTransformer(transformer) {
  return (...transformers) => {}
}

const res = createResponseTransformer(iTransformResponse)
res(iTransformResponseAsWell, andMeToo)

I think we should keep the list of transformers as the arguments to the res function directly. At the same time that stringifyJsonBody is something purely internal, and although it's the same response transformer technically, it's different by the time it gets applied.

What do you think about something like this:

interface MockedResponse {
  // ...previous properties
  // Keep the list of such built-in transformers on the response itself
  transformers: ResponseTransformer[]
}

const res = createResponseComposition({
  // That way a custom response composition can override
  // the default "transformers" functions, such as "stringifyJsonBody"
  transformers: [serializeMockedResponse]
})

Let's do a follow up of #401 and think about how to distribute those built-in response transformers internally, as well as exposing them publicly. Such API should:

• Allow to override built-in response transformers (partially or completely).
• Give a clear understanding over when built-in transformers are applied.
• Consider separating pre- and post-response composition transformers. Would this be useful?

[kettanaito]

With those points in mind, what do you think about this kind of API?

interface MockedResponse {
  // A list of response transformers executed before the first
  // transformer from the "res()" call (in request handlers).
  before?: ResponseTransformer[]

  // A list of response transformers executed after the entire
  // mocked response has been composed in request handlers. 
  after?: ResponseTransformer[]
}

[marcosvega91]

I don't find the real benefit to split transformers in both before and after ( do you have some use case in mind? ) but your solution of using transformers could be great. Could we add the default transformers in the StartOptions?

[kettanaito]

I think that the default transformers should reside on the res composition function. This should be clear, as the default res function stringifies responses using JSON.stringify, if you wish to have a custom stringification, create a custom composition function. I think this is the right message to give to the users, as they shouldn't think of the res functions as something magical. It's a plain function you should be able to compose into another function, or replace entirely.

I like your approach of creating a custom response function via createResponseComposition. We should expose this function publicly and allow users to customize built-in transformers that way. As of whether those transformers should be on the MockedResponse, or accepted by the createResponseComposition as an argument—I'm not sure.

• Set on the response.
  
  
  
  • Pros: accessible from within any response transformer, including custom ones.
  
  
  • Cons: would most likely contain the same set of built-in transformers on each response object.
  
  
• Set on the createResponseComposition
  
  
  
  • Pros: conceptually isolates built-in transformers.
  
  
  • Cons: slightly unintuitive that you give certain transformers as an argument to createResponseComposition, while other transformers straight to the created res function.
  
  

Worker and server instances represent interception layer as a whole, they shouldn't care about how response is stringified. On the implementation level it would be a huge argument drilling chain to get options.transformers down to each res function, which is a good sign it doesn't belong to StartOptions.

[marcosvega91]

Yes analyzing the code it could be very tricky pass transformers to the response.

I think that it is a clear solution :)

[kettanaito]

@marcosvega91, would you be interested in taking over this feature? Feel free to experiment with the API as you find suitable. We can always discuss things in the code review :) I think we are on the right track now.

[marcosvega91]

yes of course

[JesusTheHun]

I have read everything very fast.
Regarding the before & after I think it's simpler to just expose the array as a mutable object (or return a copy) so the user can access default transformers and fully manipulate the order of execution. We could also see the api to accept a new array, in this case the default transformers would be exposed from the package but then in case of a simple case such as a append or prepend it would require more knowledge of msw as you would have to stack the default transformers in the array.

[kettanaito]

Let's dump an idea of before/after, as there seem to be no use case for the before application. Starting simple is a good choice. For now let it be an array of transformers, fully rewritable on demand. We shall expose the default set of transformers, so you can extend it, instead of overriding.