Bull: Regarding sandboxed processes and scaling

[ahmedoverflow]

not a bug - just asking for directions

I'm using sandboxed processes by defining concurrency

Task.process(25, async job => {
    return new Promise(async (resolve, reject) => {
        try {
            await TaskFunction.start(job.data.id)
            resolve()
        } catch (error) {
            reject(error)
        }

    });
})

so im assuming it would scale by itself according to the documentation

but does it scale accross multiple cpus?
this is a screenshot of htop

im not doing any clustering right now as i read in some other topics in this issues area that sandboxing method is preferred but what do i do if i wanna have...lets say 500 concurrency for heavy tasks

any help is appreciated!

[manast]

Concurrency does not activate sandboxed processes, you need to define the processor in a separate file and specify that in the processor: https://github.com/optimalbits/bull#separate-processes

[manast]

btw, for many heavy tasks in parallel you probably want to launch more workers instead of increasing the concurrency of only one of them.

[ahmedoverflow]

Concurrency does not activate sandboxed processes, you need to define the processor in a separate file and specify that in the processor: https://github.com/optimalbits/bull#separate-processes

Already was doing that and it was an easy fix

btw, for many heavy tasks in parallel you probably want to launch more workers instead of increasing the concurrency of only one of them.

can you direct me in the right way to do this?

[manast]

start more machines with processors?

[ahmedoverflow]

start more machines with processors?

Ah okay got it so basically id be seperating the queue producer that has its own redis

then creating multiple consumer based nodejs apps that connect to the producer's redis and pickout a job

if this is the right way of doing it would there be any chance that multiple machines pick out the same task?

[himanshusinghs]

@manast What about arguments for sandboxed processors ? Documentation doesn't specify a way to pass arguments to sandbox processes. Is that even possible ?

[manast]

@himanshusinghs why would you need that? can you explain your usecase?

[vm2908]

I have a similar question. In the sandboxed process, I would like to import classes or functions ...
For example, if in the sandbox process, I want to import a mongoose object Account, how would I do that?

In any other file, I would do something like this
import Account from '../../account/model'
And then I would do Account.findById({})

Or if I want to call a function, I would do
import { processAccount } from '../../account/controller'

However, doing that in the sandboxed process file gives an error.

[himanshusinghs]

@himanshusinghs why would you need that? can you explain your usecase?

@manast I was overthinking my use case. What I wanted to achieve was easily manageable with jobs object available in processor function. Sorry to bother.
But I am interested now. Do you think there could be possible use cases where it would make sense to pass arguments to the sandboxed process?

[himanshusinghs]

@vm2908 The sandboxed process can't share the mongoose model registered in the main node process. One thing that you could do is - register mongoose models again somewhere in the top of your processor file and then simply use them as you would normally.
Example:

const mongoose = require("mongoose");
const fs = require("fs");
const path = require("path");
const modelsDirectory = path.join(__dirname, "path", "to", "model", "directory");
fs.readDirSync(modelsDirectory)
  .forEach((filename) => require(path.join(modelsDirectory, filename)));

const SomeMongooseModel = mongoose.model("SomeMongooseModel");
module.exports = job => {
  SomeMongooseModel.find({})
    .then(() => {
      // Do something
    })
};

One thing to consider is that your mongoose model registration shouldn't error out because then your job will move to FAILED state.
Whether this op is heavy for a sandbox processor is definitely a question. @manast any views on this ?

[manast]

your explanation is completely valid. I do not see any drawback with this, sandboxed processors are reused so it should not matter if they are heavy in the sense of many dependencies or initialization, however you would probably need to have a promise in your processor function flagging that the mongo connection is correctly stablished, and so on.

[vm2908]

Thank you! @himanshusinghs and @manast. This makes me wonder though if I am using Bull for its intended purpose. Basically, whenever the server gets a request from the client that requires a long operation, I add the request to Bull and send back a response to the client immediately. The request can have CPU intensive operations and/or could require connecting to the database -- essentially, anything that the main app can do. So far I had been using Agenda but I am looking at switching to Bull because I could not find a way in Agenda to run the operation in a separate process. And I want to run the operation in a separate process because the operation otherwise blocks the event queue (and the operation is not easily splittable into smaller chunks). I havent yet explored worker_threads but that could be one option. Another option would be to create a microservice. Is this the intended use case for Bull or am I missing something here. I am not a very experienced node developer so I could be getting this all wrong.

[naz]

Hey @vm2908 :wave: Was wondering if you could share the outcomes of your research and if you had any luck with worker_threads?

[vm2908]

Hi @naz I did not try worker_threads. I ended up creating a separate service for handling the CPU intensive blocking operations. For any light-weight operations that need to run in the background, I am still using agenda.