Agenda: Cluster with 2 or 3 workers, jobs are not getting "distributed" evenly

[leob]

Maybe I overlooked something because this seems pretty basic. I've used the code example from the README (https://github.com/agenda/agenda#spawning--forking-processes) to set up a cluster with N workers, each worker (Node cluster process) has an agenda instance.

Now suppose I have 2 workers (processes) and I call "agenda.now()" from worker 1, then it can be picked up (processed) by any of the 2 workers, because all of them are monitoring the queue, right? However I always see the job being picked up by the first worker, the other one(s) aren't picking up the job.

What am I overlooking?

[OmgImAlexis]

@lushc might be able to answer this. I'm guessing jobs are only added to the next worker if the first one is currently active/full instead of just using a round robin configuration like you're expecting.

[leob]

Yes that could be the case. I'm curious then what the algorithm is. Maybe I should do a little bit more (realistic) testing with heavier loads and see what the behavior is. Is there a DEBUG flag I could set in my nodejs startup command to enable some more detailed/verbose output?

[OmgImAlexis]

Add DEBUG=agenda before your app and it should enable debug messages.

For example DEBUG=agenda node ./index.js.

[leob]

Thanks! Yes that should work although for some very vague and fuzzy reason it doesn't ... maybe I should go back to the drawing board and check my setup.

[OmgImAlexis]

What version of agenda are you using?

[simison]

@leob Agenda debugger should work only with latest version which was released a couple days ago.

[leob]

@simison Thanks for the tip, I upgraded Agenda to the newest version (0.10.2), however unfortunately I STILL don't get debug output. I'm using this command:

NODE_ENV=development NODE_PATH=src/ DEBUG=agenda node index.js

However, I think I'm a step closer to the answer, @OmgImAlexis has put me on the right track.

Apparently Agenda will only use the second processor (worker) if the first one is "full" (completely "occupied") - this depends on how high you set the "defaultConcurrency" parameter. If I set "defaultConcurrency" to 1 (the standard value is 10 I think) then I do see the second worker being used.

So, apparently, Agenda first "fills up" the first Node worker (agenda instance) and only after that starts using the second (and third, etc) worker. This SEEMS to be how it's working.

I find that a bit strange, it means that the second (third, etc) cluster workers go underutilized while the first one is super busy. Wouldn't it be more logical (and better for performance) if any worker (at random) could just pick up and accept jobs until its "concurrency threshold" is reached instead of "filling up" the first one?

If this is doable (and deemed desirable) then I'd be willing to make an attempt at a code change and a PR.

[simison]

@emhagman any idea why debugging might not work here?

[OmgImAlexis]

My bad, use DEBUG=agenda:*.

[leob]

Yes that works! I'm getting plenty debug output now.

So, what about the idea of adapting the algorithm of how the workers are selected that pick up a job ... is that an idea that makes sense?

[simison]

@leob I like the idea, the rest depends on implementation. ;-) PR's welcome!

[wingsbob]

I'm noticing a lot of confusion surrounding the difference between concurrency and lockLimit.

The default lockLimit for an agenda instance is 0 which means it will keep locking jobs in the DB until it has locked all of them.

These locked jobs are tracked in-memory, and the agenda instance will allow concurrency of them to be processed using the jobDefinition.

I've been playing with this over the last few days, and found that to get horizontal scaling to work sensibly, you need to set lockLimit to a value which is a bit higher than concurrency. Not sure why, but if you set them both to the same value, my experience was that agenda spends a lot of time idle. If you set it too much higher, then the jobs all get locked aggressively, and wait in memory.

Hopefully that makes some sense 😄