Luigi: Feature suggestion: luigid --max-workers

[ocschwar]

Hi, all.

This is a particular feature that I am trying to wedge into my own working copy of Luigi in order to obviate needing a queue manager with my Luigi pipelines. The idea is simply to start luigid with a configuration that tells it to have no more than 8 workers running regardless of how many Luigi master jobs are running. Naturally that number will usually be the number of cores available. THat way if I have an 8 worker job going, and it's down to 5 running jobs, and someone else also starts a Luigi joba nd asks for 8 workers, he initially gets only 3, until mine are completed, and he gets more workers started up one by one as mine drop out.

A bigger reach woudl be to enable a Luigi Task instance to have an idle-workers parameter, so that if the Task is going to start up a CPU hogging or RAM hogging job, it can ask for n other workers to be idled at that time.

Is there any interest in this? I think such a feature would enable many academic Beowulf users to jettison their Torque/Slurm configurations and just use Luigi for a lot of jobs.

[dgronskij]

@ocschwar, have you looked at resources mechanism? Doesn't it fit your needs?

• https://luigi.readthedocs.io/en/stable/luigi_patterns.html#tasks-that-regularly-overwrite-the-same-data-source
• https://luigi.readthedocs.io/en/stable/luigi_patterns.html#avoiding-concurrent-writes-to-a-single-file
• https://luigi.readthedocs.io/en/stable/luigi_patterns.html#decreasing-resources-of-running-tasks
• https://luigi.readthedocs.io/en/stable/configuration.html#resources

[ocschwar]

Testing it right now. Wow, this is a game changer. Thank you!

[Tarrasch]

@dgronskij - thank you very much for helping other community members. :)