Luigi: Assistant dynamic resources

[fmorency]

Use case: Spawn new assistant instances on-the-fly (scale up/down) and connect them to the central planner
Issue: Resource pool is allocated when central planner is starting via configuration file. Dynamic resource pool modification is currently not possible from the Luigi client
Solution: It would be nice to be able to push new host-based resources pool dynamically when a new assistant is launched. For example,

Assistant A has 8 resources available
Assistant B has 6 resources available
Assistant C has 2 resources available

Task X requires 4 resources
Task Y requires 1 resources

Scheduler should be smart enough to schedule X on A or B (but not C) and Y on A, B or C. Lets spawn a new assistant

Assistant D has 12 resources available

The Luigi client should be able to notify the central planner that new resources are available and start scheduling jobs on it. The same mechanic applies when an assistant goes offline.

This is mostly food for tough and I would like the community's opinion on such a feature.

[Tarrasch]

Thanks for the nice description and interesting issue @fmorency. Have you thought about leaving out the assistants concept from this? I think worker-based or host-based resource limitations is what we look for. That is it's regardless if a worker is an assistant or not. What do you think?

[fmorency]

You are correct @Tarrasch. I took an example from what I'm working on at the moment, but ultimately we are looking for {host/worker}-based resources pushed via the Luigi client and without restarting the central planner.

[Tarrasch]

As far as I remember, this has been requested many times by multiple parties. If you would go ahead and implement it. That would be great for sure. :)

[daveFNbuck]

I missed this until now, but I have a fairly longstanding PR #1669 that deals with this.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.