Gun: Multi Process/Worker support ?

[corporatepiyush]

What is the best way to use all CPU cores on a single machine and distribute the computational and IO tasks but not duplicate the data on the same machine?

Can Cluster module be helpful ? in what way ?

Recently NodeJS added stable support for worker threads also.
https://nodejs.org/api/worker_threads.html

An example would be great.

[go1dfish]

Really depends on what you want to accomplish. If the issue is that you need to read from and write to gun and the in-between processing requires a ton of CPU, I would just spawn new processes to connect to your other gun peers (without their own persistence) and do the work in those process.

They could be long lived with their own job queues, or start/stop for each unit of work. Long lived would reduce the number of connections and overhead I expect.

If you want to do a cluster of processes acting as one to listen for connections I don't think this is possible with current gun.js

It's still in early dev, and I'm the only one currently using it; but I built gun-receiver to allow for running a gun server as a pm2 cluster with multiple processes serving the same port.

https://github.com/notabugio/gun-receiver

[bugs181]

Something to be on the lookout is that the upcoming Joy project will handle clusters fine by using singletons and a shared resource pool made possible by Frame.

[corporatepiyush]

Thanks both of you.

In a nutshell, I wish to achieve vertical scaling of GunDB process instance across all CPU cores and RAM memory as it would help me embed my business logic processing layer on top of GunDB API along with radisk module for disk persistence in the same instance.

It would be great if GunDB core itself or any other library/framework on top of it supports the functionality using worker threads and shared memory.

I know it can be done by using Redis or Sqlite backend perhaps but i wish to stick to pure JS implementation.

[bugs181]

Re:

pure JS implementation

I'm assuming this isn't a node-specific environment? If it is, you could accomplish this by running Gun in a vm context and retrieve the gun instance from that context. Then you'd share the context across thread pools.

The primary issue you're going to run into here is that RADISK and other adapters don't account for multiple instances of Gun. This is a known issue and one we intend to solve using singletons with JoyDB

[corporatepiyush]

Oh I see. Perhaps, I would rather wait for JoyDB to have its first stable release.

[bugs181]

Keep in mind, the alternative that @go1dfish suggested would be an excellent workaround. Spawn a new process without any storage adapters, and connect to the primary worker thread as a peer. Looking at the specs of Gun, it should realistically handle any workload you throw at it without running it across multiple CPU cores.