Luigi: assistant shouldn't require specifying task

[demoray]

Using '--assistant' should not require a task.

As is, spinning up assistant runners requires pre-knowledge of an existing task. I'd prefer to be able to spin up workers remote nodes without pre-knowledge of a defined task.

[erikbern]

Great point. Afaik @daveFNbuck and the Houzz crew are the only ones using the assistant feature in production. Is this an issue you encountered?

Is it easy to fix?

[daveFNbuck]

We're not currently using it due to some bugs I don't have time to figure out right now. My workaround is to just use a dummy task. I use my log processing task because it never hurts to process extra logs, but you can just as easily use Task.

[erikbern]

What are the bugs you were struggling with? Would be good to know

[daveFNbuck]

It's more about missing features than outright bugs. I think the biggest issue is that when we alter the code to remove tasks from scheduling there isn't a good way to get them out of the scheduler with an active assistant. I think there were one or two smaller issues with the actual running of the assistant but I can't remember what they were.

[daveFNbuck]

I think a real bug we encountered was that the assistant would sometimes get into a state where it wasn't running new tasks. I forget exactly what this looked like because I just reverted back to having workers run their own jobs.

[davemt]

when we alter the code to remove tasks from scheduling there isn't a good way to get them out of the scheduler with an active assistant

@daveFNbuck Is this resolved in #861?

the assistant would sometimes get into a state where it wasn't running new tasks

Do you know if this is still an issue or was resolved? Are you folks using assistants now?

[davemt]

Sorry I just looked at dates and realized that merge was older than this!

Anyways, I guess the question becomes, are both of these still issues?

[daveFNbuck]

We're not currently using assistants, but I'd like to try again when I have the time to work out the bugs.

[davemt]

We are thinking of using this feature and may have some time to work on bug fixes.

If there are any bugs that you have enough background info to log luigi issues for, it would be really helpful.

[erikbern]

Would be amazing if you have some time for this. Usable assistants would be tremendously useful

[daveFNbuck]

I unfortunately didn't have time to document the bugs sufficiently to share them.

[Tarrasch]

I'm currently using the assistants feature at VNG. I too feel it's buggy. I haven't encountered that assistants just stop working on tasks. But I have great issues understanding when or if ever tasks will be cleaned.

Another serious headache (for me as a luigi contributor, users maybe don't care) is that we introduced task module. I wish it could be unified with task namespace.

[Tarrasch]

Given that rant, I'm still planning to use the feature as currently I find it to the be the best option for me. I have only one machine which is both the only machine with workers and the only luigid server. I have hundreds of cronlines, but I want to ensure that at most 10 tasks are running at the same time. The best solution I found for this requirement was to use assistants.

Naturally I'm more daring in using new and untested luigi features because I know the code base well already.

Here's my (simplified) cronline:

# Try to start it every minute. This isn't really that costly.
* * * * * luigi --module package_with_all_my_code Task --assistant --workers 10

And my other cron lines that only upload the tasks but don't run them (hence --workers 0):

CMD=luigi --module package_with_all_my_code.some_module package_with_all_my_code.some_module.UmbrellaTask --workers 0 --some-param
00,30 * * * * $CMD 10026
01,31 * * * * $CMD 10073
02,32 * * * * $CMD 10079
03,33 * * * * $CMD 10085
04,34 * * * * $CMD 10086
05,35 * * * * $CMD 10090
...

[fmorency]

I've been playing with Luigi assistant for the past few days and it seems promising.

I agree with the principal thread subject that the required base task given to the luigi utility feels wierd in the context of assistant.

We have long running pipeline/jobs (~2h) and the workflow seems to hold. However, the assistant service where workers are kept alive terminates as soon as a task generates an error. This is a bit problematic for us.

[mfcabrera]

I tried to lookup information on the assistant. I couldn't find any reference on the documentation. What is it? Is this feature still maintained?

[Tarrasch]

I'll try to fix this and create some docs for this. :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.