Luigi: how to define dynamic resources according to luigi parameters?

[leafjungle]

for example, x = luigi.Parameter()
if x==1:
resource = {"r1": 1}
elif x==2:
resource = {"r2": 1}

What can I do? redefine the init function?

[erikbern]

not sure how to do it. maybe define a property decorator?

[daveFNbuck]

I actually do this a lot and a property decorator is the answer:

class ExampleTask(luigi.Task):
    x = luigi.IntParameter()

    @property
    def resources(self):
        if self.x == 1:
            return {'r1': 1}
        else:
            return {'r2': 2}

[quentinsf]

@daveFNbuck - many thanks! That's just what I needed!

I'm updating HDF5 files, and I need to make sure that no two tasks try to write to the same file at once. Following your example of using a property, I can do (effectively):

class HDF5TestTable(luigi.Task):
    ...

    @property
    def resources(self):
        return {self.hdf_file_name: 1}

Seems to work nicely! I'll do a pull request to get something like this into the documentation, I think.

[visjedi]

How could we check this for correctness? Is there a small test/hello world example to show this indeed works and the file is held by only worker at a time? I shall be glad to help if someone is interested.

[daveFNbuck]

@visjedi we have unit tests for resources. Luigi can only guarantee that you won't exceed your resource limit. It's on you to set up your resources such that this means you won't have multiple jobs touching the same file at the same time.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.

[dlstadther]

This looks already resolved