Luigi: Access the number of workers from within a Task?

[gioelelm]

Is there a way to access, from within a task, the number of total workers that have been made available to the command line call that eventually triggered the task?

Basically this would be the the value specified for the --workers parameter.

[ACarrasco1009]

While not ideal, you can do this with a "global" definition, and using the luigi.run() function:

import luigi

WORKERS = 2

class Task_(luigi.Task):
    def requires(self):
        return []

    def run(self):
        print(WORKERS)

if __name__ == '__main__':
    #f-string will only work in Python 3.6, use .format() instead if using older version
    luigi.run(main_task_cls=Task_, cmdline_args=[f'--workers={WORKERS}'], local_scheduler=True)

[jeanmn]

Yes, there is a way
@gioelelm @ACarrasco1009

import luigi

class DemoTask(luigi.Task):

    def run(self):
         # config vars parsed in order specified by the luigi docs
        config = luigi.configuration.get_config()

         # the additional core defaults, when not specifying config
        core = luigi.interface.core(config)

        print(core.workers)

[ACarrasco1009]

@jeanmn That's great, never knew about it. Thanks!

[Tarrasch]

I would probably do:

import luigi
import luigi.interface

class DemoTask(luigi.Task):

    def run(self):
        print(luigi.interface.core().workers)

This will also work if you specify the workers by the command line (--workers). Also see these docs.

[jeanmn]

Thanks for pointing this out @Tarrasch; made me revisit the internals one extra time :)
(where the magic happens)