Kue: More than one jobs were executed

[nguyenxuantuong]

I have an application in which each job consumes lots of CPU. Therefore, only one job should be executed at a time. Only if the current job has been executed and finished (with/without error), then next job can be taken from the queue and be executed. But it seems that in my case, there is always more than one job being executed at a time.

I used jobQueue.process("", function (job, done)....) so by right, it should only run one job.
I do using cluster modules; but even if i disabled the cluster module, still more than one jobs being executed. In case of cluster, number of job being executed > #core. So i really don't have any clue.

I did try redis also: var jobQueue = kue.createQueue({
prefix: 'q',
redis: {
port: Config.Redis.port,
host: Config.Redis.host,
auth: Config.Redis.pass,
db: Config.Redis.database
}
}); but no help.

How to limit number of job to be exact one. It would also be more desirable if we can limit # of concurrent job to be exact 1; even if some one using cluster modules (which i think it's possible with redis as data store).

One more question, the document mentioned that the default setting is one concurrent job being run. By concurrent (in contrast with parallel), does it means that there might be multiple running jobs (they might all start at the same time -- long running process); but at a time, only one job being executed (by using thread context switching eg.). But if it's case, what happens if my jobs are IO-bound, meaning it might spawn other expensive CPU-bound processes (which in desired case, should have at most one running).

[behrad]

each jobQueue.process("my_job", function (job, done){...}) ensures one job being processed at a time, and this means from when we your callback is called, until you call done
But if you are calling .process multiple times (being in a single node.js process, or a cluster) you will increase job processing concurrency.

One more question, the document mentioned that the default setting is one concurrent job being run. By concurrent (in contrast with parallel), does it means that there might be multiple running jobs (they might all start at the same time -- long running process); but at a time, only one job being executed (by using thread context switching eg.). But if it's case, what happens

node.js is a single threaded process, so all your stuff will be run sequentially with no thread context switches.

if my jobs are IO-bound, meaning it might spawn other expensive CPU-bound processes (which in desired case, should have at most one running).

you start processing your job, do your IO/CPU bound stuff and when it finishes, you call done and this guards you against your heavy resource utilization.

[nguyenxuantuong]

Thanks. It does seem that in my code the jobQueue.process has been executed multiple of time due to cluster modules. So multiple processes spawn by master process have created jobs and consume jobs concurrently. So that's why there are always more than one jobs running at a time.

For someone having the same problem, I have managed to solve it by only allow the master process to consume the jobs (but all other process to create jobs) which can be done by checking: MASTER = cluster.isMaster; if(MASTER) {//put your logic here -- jobQueue.process} ...

Thanks for prompt answer!