Boardgame.io: Horizontal scaling

[JoHuang]

Hi

Does this framework support horizontal scaling?
Which means

1. stateless multiple node.js instances behind a load balancer.
2. central state storage (ex. MongoDB, redis , ...)

It seems that single point of failure might happen in this framework?
Correct me if I'm wrong.
Thanks

[jasonharrison]

Hi Jo,

For 1), it is possible to run multiple BGIO servers behind a load balancer, however, you may run into issues with socket.io's websocket broadcasts. It may be possible to overcome this (for example, with https://github.com/socketio/socket.io-redis ), but I have not looked into what this would entail.

Note that games and content can be served separately from BGIO (and served on multiple servers without issue); the only point of failure could be with multiplayer mode which uses BGIO's server api.

For 2), BGIO's server does support external databases such as MongoDB (these can be set through environment variables). See:

https://github.com/nicolodavis/boardgame.io/tree/master/src/server/db
https://github.com/nicolodavis/boardgame.io/blob/master/src/server/db/index.js

[JoHuang]

For 1), if I have A, B, C 3 servers behind a load balancer, players a, b, c connected to A, B, C respectively. Can these 3 players play the same game?

If the answer is true. I have another question about 2), especially for MongoDB implementation.
There is a LRU for preventing race condition, but the LRU is only used locally. Is that be able to prevent race condition caused by different servers?

Thanks

[nicolodavis]

@JoHuang There is another issue open to discuss this: #277

You can try to use a load balancer with sticky connections to ensure that all the traffic for a particular game goes to the same server, but I think it would be better if we added support for something like SocketCluster.

[nicolodavis]

@JoHuang Is this something that you'd like to contribute to? We could definitely use some help in this area.

[JoHuang]

Thanks for reply.
So the MongoDB implementation is only for record / backup, players have to connect to the server/instance to join the same game.

I'm still evaluating this framework. For now I'm not sure if I would put effect on it.

To achieve horizontal scaling, there are 2 todo items:

1. Stateless node.js instances behind load balancer. (load & save to database for every move)
2. Atomic update game states in a central state storage (Mongodb atomic ops like findAndModify).
3. Broadcasting update data to players connected to different instance. (Socket.io-redis ..., for non-open source solution, maybe Pubnub, Pusher, Ably.io these kind of services can help for rapidly integration)

My 2 cents. :)

[leocaseiro]

For Socket.io, you can sticky lazy loading with socket.io-redis, as mentioned before by @jasonharrison.

Read more here: https://socket.io/docs/using-multiple-nodes/#Sticky-load-balancing

In that case, redis will pass events between nodes.

Here is an example https://github.com/socketio/socket.io/tree/master/examples/cluster-haproxy

[leocaseiro]

one more nice article https://medium.com/containers-on-aws/scaling-a-realtime-chat-app-on-aws-using-socket-io-redis-and-aws-fargate-4ed63fb1b681

[nicolodavis]

Thanks for sharing @leocaseiro

[gk-patel]

I am not an expert for Scaling, but let me give a try.

We generate a session-id/hash for each Online session, we could use this hash to reroute the user to a specific instance. Thus what we need is an algorithm that maps hash to number between 1 to n, where n is the number of servers. This way we can make sure that users are routed to the same host.

[leocaseiro]

That’s exactly what this lib does, using IP address for hashing. Although it’s no longer maintained: https://github.com/wzrdtales/socket-io-sticky-session

There’s also an explanation in details here: https://github.com/elad/node-cluster-socket.io

[JoHuang]

It means hosts are stateful, and the hosts can not come and go anytime as long as there are users connected. So single point of failure could be a potential issue, and scaling down is limited.
It could be a solution for now, but maybe not the final goal IMHO.

[gk-patel]

@JoHuang you are right. I think that is the reason why some solutions suggested above use an ACID database like Redis. Nevertheless, this hash-based routing could still work in an environment where new nodes can be spawned if it is known that a node has failed and the new node can get old data from redis/database. (again, I am not an expert, I was just giving a suggestion, thanks)

[nicolodavis]

Marking this as a duplicate of #277