Msw: WebSocket / Server-sent Events support

[Doesntmeananything]

Is it possible to use msw to mock server-sent events (SSE) and WebSocket (WS) connections?

My use case is to have a somewhat full client-side mocking story (including REST, SSE, and WS functionality), and since msw is such a joy to use when mocking out REST APIs, I was wondering if it makes sense to use it to mock more specialised server interactions.

Have you thought about this? I admit that I haven't looked much into whether it's possible just to use custom request handlers to add this functionality, emulating SSE and WS behaviour in some way. I wanted to know if you already had something in mind regarding this question. Thanks!

[kettanaito]

Hey, @Doesntmeananything, thanks for bringing this topic up. I'd love to bring SSE and WS support to the users of MSW. I admit I haven't researched the topic yet, but would use this thread for this.

Technically, it comes down to the ability of Service Worker to intercept those requests/events. If the spec supports it, there shouldn't be much changes needed on the MSW side.

Here's some useful resources:

• https://github.com/w3c/ServiceWorker/issues/885, EventSource interception in Service Worker.
• https://github.com/w3c/ServiceWorker/issues/947, a discussion whether Service Worker should access WebSocket events.

Could you please try to set up a proof of concept, if those events can be intercepted in the worker's fetch event?

You're right about the custom request handler, we can use it to log all intercepted requests:

setupWorker(
  {
    predicate(req) {
      console.log(req)
      // log captured requests, but bypass them
      return false
    },
    resolver: () => null
  }
)

If we confirm it working, I'd be glad to discuss the API for a dedicated request handler for WebSocket/SSE. I can read on their specification meanwhile.

[Doesntmeananything]

Sounds like a plan! At a cursory glance, it does indeed seem quite doable. Let me PoC this, and I'll get back to you with my results as soon as I can.

[Doesntmeananything]

Hi, @kettanaito! I've set up a (very quick and dirty) repository to test these interactions over at https://github.com/Doesntmeananything/msw-sse-ws.

My initial findings are the following:

• I am able to intercept SSE connections, however this occurs only when a stream ends. For example, if a stream consists of 3 messages, logging happens only after all 3 messages have been received, including the final end event.
• I am not able to intercept WS connections. I tried to make sure that the WS connection is established from the client only after the mock service worker had been initialised, but it didn't seem to help. I want to investigate this further.

I'm a bit concerned about WS events, although I hope that with some additional work it'd possible to intercept them.

[kettanaito]

@Doesntmeananything, thank you for the investigation! I'm excited to hear that SSE can be intercepted! Wonder if there's anything we can do it intercept events as they go.

I'm currently working on a NodeJS support, but can switch to this issue to help you once I'm done. I'm always open to questions or discussions, so please don't hesitate to raise those here.

Also, if you don't mind, we could then move your proof of concept repo under "msw" to serve as an example how to work with SSE/WS. That'd be awesome.

[kettanaito]

I'm trying to get my head around the SSE example. It seems MSW should intercept the hi from client event, so then it can mock the server response to it. I can see the once the WS connection is established, all the messages are inspectable live in DevTools. However, the webSocket.send("hi from client") is not intercepted by the Service Worker. I'm reading through https://github.com/w3c/ServiceWorker/issues/947, trying to figure out if it's technically possible to access WS communication in a service worker.

API-wise, I think there should be at least two types of request handlers: event-based handler, and persistent handler (pulsing back messages to the client, like you have in your example using AsyncIterator).

[kettanaito]

One of the most useful pieces of code I've found in the w3c discussion (https://github.com/w3c/ServiceWorker/issues/947#issuecomment-410816076) was that the Service Worker file can _establish_ a WebSocket connection. It appears that the WS events are not subjected to be intercepted in the fetch event, but one can establish a socket connection an (?) intercept events that way.

If it comes down to the manual WS connection, I'd suggest to do that on the client's side, not in the worker file. There's no technical reason to move this logic to worker, at least as of how I understand such implementation now.

[Doesntmeananything]

Thanks very much for taking the time to look further into this!

Since I've hit the wall in regards to intercepting WS connections, your suggestions come in handy. Will definitely look into this.

To be clear, are you saying that mocking WS connections falls strictly outside of MSW concerns? My investigations lead me to believe this, and I would certainly not want to push for something that doesn't make sense neither on technical nor on conceptual level.

[kettanaito]

Not necessarily. What I was trying to say is that a WebSocket event is not intercepted by the fetch event in a Service Worker. That's a per-spec behavior. However, I've mentioned an example above, that creates a WS connection within the worker file, which I suppose allows to intervene the communication in some way. I haven't tried that approach out, whether it's actually possible to mock the response of an event.

[kettanaito]

I've received a suggestion to look at mock-socket. We may get some inspiration from how it's implemented, and see if a similar approach can be done in MSW.

[kettanaito]

Update: I've started with the WebSocket support and will keep you updated as I progress. For those interested I will post some technical insights into what that support means, what technical challenges I've faced, and what API to expect as the result.

Session 1: It's all about sockets

No service for the worker

Unfortunately, WebSocket events cannot be intercepted in the fetch event of the Service Worker. That is an intentional limitation and there's no way to circumvent it. This means a few things:

• WebSocket events won't be visible in the "Network" tab;
• WebSocket support does not require the worker and can live outside of setupWorker context.
• WebSocket events interception should be done by patching the underlying logic (i.e. a WebSocket class).

Goodbye, handlers!

WebSocket operates with _events_, not requests, making the concept of request handler in this context redundant. Instead, you should be able to receive and send messages from ws _anywhere in your app_, including your mock definition.

import { rest, ws, setupWorker } from 'msw'

// Create an interception "server" at the given URL.
const todos = ws.link('wss://api.github.com/todos')

setupWorker(
  rest.put('/todo', (req, res, ctx) => {
    const nextTodos = prevTodos.concat(req.body)

    // Send the data to all WebSocket clients,
    // for example from within a request handler.
    todos.send(nextTodos)
    return res(ctx.json(nextTodos))
  })
)

// Or as a part of arbitrary logic.
setInterval(() => todos.send(Math.random()), 5000)

URL that got away

When constructing a WebSocket instance you must provide a URL that points to a WebSocket server. Providing a URL to a non-existing server yields and exception that you cannot circumvent by patching WebSocket class, as the URL validation lives in its constructor.

I've ended up re-implementing a WebSocket class, effectively making a polyfill out of it. That way it can have its custom constructor that would tolerate non-existing URLs if there is a ws.link interception server declared for that URL.

Persisting WebSocket clients

The entire idea of WebSocket is to sync data between multiple clients in real time. When you dispatch a mocked ws.send event to send some data to all clients, you need to let all the clients know they should receive the data (trigger their message event listener). However, there's no way to know and persist a list of WebSocket clients on runtime, since each page has its own runtime.

Usually a solution to this kind of problems is to lift the state up and maintain a record of clients in the upper context shared with all the clients (pages). However, in JavaScript there isn't that may ways to share and persist data between clients. In case of WebSocket clients one needs to store references to WebSocket instances—basically, object references. I've considered:

• localStorage/sessionStorage. Good for sharing textual data, not that suitable for storing objects with prototypes. Objects flushed here effectively lose their references, making them completely different objects.
• Web Worker/Service Worker. A great way to have a detached process in a browser that can control and communicate with multiple pages. However, as stated before, worker API cannot intercept WebSocket events, and using it only for the sake of storing some records is utterly redundant (I don't wish to ask people to copy any more worker scripts). Utilizing an existing mock worker may be an option, however, I'm afraid it would put too many logic into it, increasing its maintenance, and violating its purpose of containing a bare minimum logic that you seldom need to update.
• BroadcastChannel. Turns out the API that allows workers to communicate with clients exists standalone and it's awesome. You can create a broadcast channel as a part of page's runtime, and as long as another page _on the same host_ creates a channel with the same name they can send data between them.

const channel = new BroadcastChannel('ws-support')

// One client sends a data.
channel.send('some-data')

// All clients can react to it.
channel.addEventListener('message', (event) => {
  event.data // "some-data"
})

I find BroadcastChannel a great choice to mimic the real time data synchronization functionality of WebSocket. I've chosen it to spawn a single channel between all clients and notify them when they should trigger their message event listeners.

[Sun-2]

@kettanaito

URL that got away
When constructing a WebSocket instance you must provide a URL that points to a WebSocket server. Providing a URL to a non-existing server yields and exception that you cannot circumvent by patching WebSocket class, as the URL validation lives in its constructor.
I've ended up re-implementing a WebSocket class, effectively making a polyfill out of it. That way it can have its custom constructor that would tolerate non-existing URLs if there is a ws.link interception server declared for that URL.

You could use an ES6 Proxy. It can mess with ctors.

Link.

[BlackGlory]

SSE and WebSockets are different issues.
If msw supports response streams (such as ReadableStream), it can support SSE.

[kettanaito]

@BlackGlory, MSW should support ReadableStream as the mocked response body. Would you have some time to try to which extent that's true, and whether SSE would be supported now?

[BlackGlory]

@kettanaito Although ctx.body supports ReadableStream, it does not seem to work.

export const worker = setupWorker(
  rest.get('/sse', (req, res, ctx) => {
    return res(
      ctx.status(200)
    , ctx.set('Content-Type', 'text/event-stream')
    , ctx.body(sse(function* () {
        yield 'message1'
        yield 'message2'
      }))
    )
  })
)

function sse(gfn) {
  let iter
  return new ReadableStream({
    start() {
      iter = gfn()
    }
  , pull(controller) {
      controller.enqueue(`data: ${iter.next().value}\n\n`)
    }
  })
}

Browser:

[MSW] Request handler function for "GET http://localhost:8080/sse" has thrown the following exception:

DOMException: Failed to execute 'postMessage' on 'MessagePort': ReadableStream object could not be cloned.
(see more detailed error stack trace in the mocked response body)

Node.js:

TypeError: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received an instance of ReadableStreamTypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received an instance of ReadableStream

      at ClientRequestOverride.<anonymous> (node_modules/node-request-interceptor/src/interceptors/ClientRequest/ClientRequestOverride.ts:216:34)
      at step (node_modules/node-request-interceptor/lib/interceptors/ClientRequest/ClientRequestOverride.js:33:23)
      at Object.next (node_modules/node-request-interceptor/lib/interceptors/ClientRequest/ClientRequestOverride.js:14:53)
      at fulfilled (node_modules/node-request-interceptor/lib/interceptors/ClientRequest/ClientRequestOverride.js:5:58)