Feathers: SocketCluster support

[IngwiePhoenix]

Hello!

Over the time I have been a rare-time contributor to SocketCluster - but that doesnt mean I am actually using it. No; in fact, my whole bigger project is basically centered around it. To give you a few shortnotes on what SocketCluster is:

• Scale horizontally and vertically with ease.
• Separate worker and broker processes to manage incoming traffic and backend work.
• Integrated key-value storage: nData
• Event-based approach, where back- and frontend are synced, even across nodes; heck, you could probably sync two buildings full of serves with this!
• WebSockets, and no backwards compatibility to older transports - so, its a yes-or-no situation.

There are a lot more features, which is why I would recommend their website for more information on the subject.

But basically, what I will focus on is the worker process - that one has access to HTTP and the WebSockets stuff. And in fact, I can even provide actual code to demonstrate how I am currently using it with Express!

So first to note is that each process gets a pre-configured HTTP server instance. So Feathers' .setup() method is the proper approach in any way, since that is already a given. Instantiating a new server would defeat the purpose of using SC in the first place. This is how you do it: https://github.com/DragonsInn/BIRD3/blob/843736bea08d22316009558b4119df5e1dbf981c/app/Backend/SocketCluster/Worker.js#L17

Next, to get access to the SocketCluster server itself - the WebSockets stuff - we can do so with a separate method too. To see that, just go ~3 lines deeper on the previous link.

Now that these two are gotten, that is basically all that would be required to make Feathers compatible:

• The user has to use .setup instead of .listen
• Feathers needs to learn to speak with SC's Real-Time engine.

In most - but not all cases - it is very similar to that of Socket.IO. But some things are different intentionally - callback layout, for instance - since SC supports stuff like "answering" to an event - much like returning data int hat sense. And to see how it is used in its most basic way, see my link above and go towards the bottom; you'll see me setting up a "ping" event and an empty "disconnect" event. There are more events like authentication. One can provide SC with a custom authentication engine - allowing Feathers to throw itself right in and providing the user with a unified mechanism.

So, thats as much of my SC introduction. I might try my luck by forking the feathers-socketio component and hacking it into the right way, cleaning it up and returning it. But I would like to see what you think about it! :)

Hope my english didn't fail me (I am actually german ^^) and you understood what I wanted to write about. I personally really want to move from plain Express to a more fleshed-out backend service - and Feathers seems to provide that. I really like your error handling, for a start. And the other things I have seen look really interesting as well! So we'll see what comes along.

Kind regards,
Ingwie

[ekryski]

Hey @IngwiePhoenix thanks for the detailed description, and your english is excellent! I would have guessed you were a native speaker. :smile:

I had sort of looked at Socket Cluster as an alternative to Feathers but I think that is more my lack of understanding of how it works. Some of the resiliency and failure mechanisms it has look pretty great and from what you described it doesn't look like it would be a ton of work to be able to use it as the transport layer for Feathers sockets. @daffl will definitely need to weigh in here.

Feathers itself, achieves socket clustering via two mechanisms:

The crude manual method

You can register a client on each server manually for every other server you want to communicate with. It does work with JWT auth and will do retries and reconnects.

Using feathers-sync

You can use a central pub-sub mechanism via https://github.com/feathersjs/feathers-sync. It totally works but we'll be spending more time in the near future on the scaling part of Feathers.

As for forking and doing a PR, by all means go right ahead. It would be interesting to see what you come up with. You may want to wait for @daffl to chime in and see if he has any ideas that could help it along.

Thanks for bringing this up!

[IngwiePhoenix]

You are very welcome! Ill do some local poking then, and can wait for @daffl in the meantime :)

[kulakowka]

I am currently studying Socket Cluster. It works pretty cool. I would be interested to see an example of integration with feathers.

[jondubois]

@ekryski @kulakowka @IngwiePhoenix Main SocketCluster author here.

Looking at FeathersJS code, I don't think you would be able to create a plugin for the full SocketCluster framework (with multi-core scalability and all) BUT you could easily make a plugin for the standalone (more lightweight) socketcluster-server (https://github.com/socketcluster/socketcluster-server). Users will be able to connect to it using socketcluster-client (https://github.com/socketcluster/socketcluster-client) as normal.

socketcluster-server behaves exactly the same as the full SocketCluster framework except that you don't get the automatic scalability and opinionated structure so it offers more flexibility when it comes to integration into existing systems and high-level structural frameworks like Feathers.

[IngwiePhoenix]

@jondubois I do see what you're going at here. But what if you actually turned this upside down? When a user initiates an SC instance, and within an SCWorker initiated a FeatherJS app which "inherited" the SC HTTP and WS server(s)... What would let us do that, in this scenario?

[jondubois]

@IngwiePhoenix Oh so you mean ultimately you just want to be able to attach Feathers to a raw Node.js http server like Express allows? E.g. https://github.com/SocketCluster/socketcluster/blob/master/sample/worker.js#L9-L16

I guess that should be feasible :)

[IngwiePhoenix]

@jondubois Not just the HTTP server - that's a no-brainer :). But also the ability to interact with SC's "real time API" through the uniform API of FeatherJS. For instance, take this example of socket.io into example:

import feathers from 'feathers';
import socketio from 'feathers-socketio';

const app = feathers()
  .configure(socketio(function(io) {
    io.on('connection', function(socket) {
      socket.emit('news', { hello: 'world' });
      socket.on('my other event', function (data) {
        console.log(data);
      });
    });

    io.use(function(socket, next) {
      socket.feathers.data = 'Hello world';
      next();
    });

    io.use(function (socket, next) {
      // Authorize using the /users service
      app.service('users').find({
        username: socket.request.username,
        password: socket.request.password
      }, next);
    });
  }));

app.use('/todos', {
  get: function(id, params) {
    console.log(params.data); // -> 'Hello world'

    return Promise.resolve({
      id,
      description: `You have to do ${name}!`
    });
  }
});

(Copied from the feathersjs-socketio repo's README.)

My idea was originally to fork from this module and extend on the way socket.io is integrated - and instead pass the SCWorker instance, from which FeathersJS can obtain the HTTP server for .setup() and for doing real-time related activity.

Basically, take this as a pseudo-version of an SC worker:

import feathers from 'feathers';
import sc from 'feathers-socketcluster';

export function run(/* SCWorker */worker) {
    const app = feathers()
      .configure(sc(worker, function(sc) {
        sc.on('connection', function(socket) {
          socket.emit('news', { hello: 'world' });
          socket.on('my other event', function (data) {
            console.log(data);
          });
        });

        // Other SC stuff...

      }));

    app.use('/todos', {
      get: function(id, params) {
        console.log(params.data); // -> 'Hello world'

        return Promise.resolve({
          id,
          description: `You have to do ${name}!`
        });
      }
    });

    // I don't exactly know if FeathersJS allows a module
    // to define a new HTTP instance to use or not.
    // But if it does, the above could already pull
    // worker.httpServer into account.
    app.setup();
}

The basic idea is to teach FeathersJS to understand SC's basics and obtain the information it requires from it to run. The user can still define all the usual things possible with SC.

I just woke up, so just letting you know I could be babbling some weird english...sorry if that happened! :)

[bytecodedan]

Is that all it takes? Is this a working example? This would be huge! I'll have to try this later tonight.

I'm a big fan of Feathers and Socket.io but didn't realize until last week that clustering (standard Node way) breaks the WS communication. Which makes sense because the master proc dispatches requests to worker procs via RR (round robin). There's no way for master to know which WS belonged to which worker (at least that's how I'm interpreting the problem). And that's a bummer because you loose the performance benefit of "clustering" Node.

But if Feathers could easily integrate with SocketCluster, as mentioned here... that would be HUGE!

Thanks for posting this!

[bytecodedan]

Nevermind, I just realized that it's pseudo-code - a proposal of what it could possibly take to integrate Feathers with SocketCluster.

Well, anyway... let's try to keep this moving forward! There's definitely a need.

[daffl]

We're still happy to support anybody interested in implementing this but I am going to close this issue since it has been inactive for a while and the team is going to focus on the already supported transport providers.

[devel-pa]

Feathers integration with SC Worker: https://github.com/polst/feathers-socketcluster

[daffl]

@devel-pa Awesome! Would you mind adding it to the showcase section at https://docs.feathersjs.com/why/showcase.html?

[devel-pa]

There must be a bug, as I'm packing the requested data on client into an array and not unpacking on server side. Ufff

[devel-pa]

@daffl I don't mind :) Not quite tested. What should I do?

[devel-pa]

PR done for docs

[devel-pa]

@daffl Now looks like it's working. Unfortunately I had to modify https://github.com/polst/feathers-socket-commons as I didn't found other way to solve the issue with only one data argument in SC

[daffl]

@devel-pa Can you make a PR with those changes and tests for it?

[devel-pa]

@daffl PR done.

[areknawo]

Does it work with FeathersJS 3.x?

[lock[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue with a link to this issue for related bugs.