Luigi: Passing lists in on the command-line doesn't work properly

[alexbw]

If you have a parameter that's declared as a list, e.g.

import luigi
class ListTest(luigi.Task):
    my_list = luigi.Parameter([1,2,3], is_list=True)
    def run(self):
        print "\n"*5
        print self.my_list
        print "\n"*5

As long as you don't touch the defaults, or you pass in lists in Python, everything works fine.

python list_test.py ListTest --local-scheduler

prints (without the extra scheduling stuff)

(1, 2, 3)

But when you try to pass in the list in the command-line, the list is never parsed properly.

python list_test.py ListTest --my-list=[4,5,6] --local-scheduler

('[4,5,6]',)

I've tried all brands of character-escaping, but the list always comes in as a string. A solution I'm using as a stopgap is this:

import ast
class TupleParameter(luigi.Parameter):
    def parse(self, x):
        return tuple(ast.literal_eval(x))


class ListTestWorking(luigi.Task):
    my_list = TupleParameter([1,2,3])
    def run(self):
        print "\n"
        print self.my_list
        print "\n"

python list_test.py ListTestWorking --my-list=[4,5,6] --local-scheduler

now works properly

(4, 5, 6)

ast.literal_eval is supposed to be a "safe" version of eval, meaning it'll only work on valid Python literal structures.

[erikbern]

I think you need to pass --my-list 4 --my-list 5 --my-list 6 (see https://github.com/spotify/luigi/blob/master/luigi/interface.py#L192)

Would also like a more compact way of passing lists though (and even better would be to have an option to just take remaining args on the cmd)

[alexbw]

What do you mean by remaining args on the command?

Any thoughts on the TupleParameter class pasted above?

[erikbern]

I think it would be awesome to do something like this:

class ListTestWorking(luigi.Task):
    my_list = IntParameter(args=True)

python list_test.py ListTestWorking --local-scheduler 4 5 6

Regarding TupleParameter, I think it sounds useful, but why not call it a more general PythonParameter since it will really parse any structure

[alexbw]

I agree re: PythonParameter, but I'd be in favor of exposing subclasses like TupleParameter and ListParameter and NumpyParameter that cast the results, and make it clear in code what a parameter's type will end up being.

And about the args=True approach, how would multiple parameters work?

class ListTest(luigi.Task):
    my_list1 = IntParameter(args=True)
    my_list2 = IntParameter(args=True)

python list_test.py ListTest --local-scheduler --my-list1 4 5 6 --my-list2 7 8 9

It doesn't feel quite right, but if this is somehow standard practice in the shell, or how the shell usually interprets lists, I'm all in. Is this somehow standard behavior for argparse or the CLI?

[erikbern]

No it wouldn't work with multiple args=True, that's correct. It would be something similar to how *args work in Python

[jmmk]

I ran into this today adding a list parameter to a task. The current "append" syntax seems clunky to me. The cleanest way I've found to do it with argparse is to use nargs='+'. (Documentation for nargs)

In interface.py, ArgparseInterface.add_parameter, this would look something like:

@classmethod
def add_parameter(cls, parser, param_name, param, prefix=None):
    description = []
    if prefix:
        description.append('%s.%s' % (prefix, param_name))
    else:
        description.append(param_name)
    if param.description:
        description.append(param.description)
    if param.has_value:
        description.append(" [default: %s]" % (param.value,))

    if param.is_list:
        parser.add_argument('--' + param_name.replace('_', '-'), help=' '.join(description), default=None, nargs='+')
    else:
        if param.is_boolean:
            action = "store_true"
        else:
            action = "store"
        parser.add_argument('--' + param_name.replace('_', '-'), help=' '.join(description), default=None, action=action)

Then you can pass the arguments like python myscript.py MyTask --listarg arg_one arg_two arg_three

[bp256r1]

I was able to get around this issue by simply defining a custom parameter type

class ListParameter(luigi.Parameter):
    def parse(self, arguments):
        return arguments.split(' ')

Then passing a given ListParameter to a given luigi Task as follows:

...
class Fetch(luigi.Task):    
     my_list = helper.luigi.ListParameter(default=[])
...

This allows you to pass in arguments as follows: python myscript.py MyTask --my-list 'one two'

¯\_(ツ)_/¯

[Tarrasch]

Yes. That's the correct way @TylerJFisher, only that you must also implement serialize() (the opposite of parse).

I'm closing this. Since it's old and is_list is removed/deprecated core feature. If you want list like stuff do similar like what @TylerJFisher did above.

[palmeida]

I was able to pass a list of strings in the command line by enclosing it in quotes:

luigi --module my_module MyTask --local-scheduler --my_parameter '["par1", "par2", "par3"]'

I also tried with a list of ints (--my_parameter '[1, 2, 3]') and it worked. I know the issue is closed, just leaving this here in case it's useful for anyone who is creating a parameter type just for this.