Flux-core: Job Scheduling within a Hierarchy

[SteVwonder]

Full transcript of our email exchange can be found here: https://gist.github.com/SteVwonder/73faf0dcb61d0bf205fe0d4e047ee67c

Also see #395 for context

My attempted summary (sorry if I misquote anyone):

@stevwonder asked about alternative approaches to hierarchical scheduling that were in discussion
@garlick pointed to issue #395 and advocated for the child instance to have a special initial program that accepts/pulls jobs from the enclosing instance. This would keep the child scheduler ignorant to the source of the jobs. He also suggested keeping it loosely coupled and possibly using capacitor
@lipari mentioned that Suraj's code is a platform for research and can branch out to do different things in search of an optimal model. For production codes, he hopes that the scheduler will know as little as possible about the hierarchy and the activities of other schedulers in order to maximize the distribution of scheduling intelligence.
@stevwonder agreed with @grondo and @garlick's original points of keeping the schedulers as ignorant as possible to the source of jobs. He mentioned that the initial program acting as a proxy was odd, and suggested that the parent submit jobs directly to the child scheduler (child would still remain ignorant of the source). He also mentioned that this opens the door to race conditions which would require additional scheduler functionality to avoid (or the use of the proxy).
@stevwonder agreed with @lipari on his points and gave an example of a change to make to the current hierarchical scheduling code to bring it in line with these ideas
@garlick agrees that if the child provides the job submission "service" to the parent that trouble will follow. He suggested that the parent provide the "service". For example, the child's initial program pulling jobs from the parent until it receives an error/EOF. He mentioned that maybe the functionality should be put in a new comms module in order to keep the scheduler out of the process as much as possible. He reiterated the desire for the instances to have as little awareness of the hierarchy as possible. He suggested parents need to know about their children, but perhaps the opposite is not necessary.

[SteVwonder]

Going back through, I like @garlick's idea of building the initial program on top of capacitor. For the research, the initial program simply needs to read in jobs from a file and submit them to the scheduler (which sounds like exactly what capacitor was built for). I also like the idea of replacing my "submission" comms module with capacitor.

The parent "service" is also a good one. It would eliminate the race conditions, and as long as the child polls the parent frequently enough, the incurred delay should be reasonable. Since the parent might not always have a job available, there would need to be a code for "no work right now, but check back later if you are still alive". This of course would mean that if we want to support online job submissions, the instances would have to know about their parent/enclosing instance.

I warming up to the idea of putting this functionality inside of the initial program or a separate module. This way, the scheduler only knows about the child instances (but not its parent), and the initial program/module only knows about the parent instance (but not the child instances).

Maybe we should also consider putting the dynamic scheduling communications inside of a separate module as well (rather than integrating it directly into the scheduler). Not quite sure how that module would interface with the scheduler, but it should be feasible. It would also keep everything modular and allow us to change the functionality of the scheduler by adding/removing modules.

[garlick]

Since the parent might not always have a job available, there would need to be a code for "no work right now, but check back later if you are still alive".

Or the service can simply not respond until it has a job for the client. The client can be dumb and just block there, or if it needs to do other things, it can manage that RPC asynchronously (e.g. see flux_rpc_then(3))

[SteVwonder]

@trws's email:

I'm probably just missing it going through this post-mortem, but what is
the use-case you're after? The way I've been conceptualizing this for
the most part is that in general a child wouldn't get new jobs from a
parent after launch, but yield resources to the parent to allow it to
run jobs. The parent then offloads work by taking a number of, possibly
related, jobs and creating a new child responsible for running them.

This doesn't really handle the case where you have a top-level
orchestrating instance, then one per cluster, or similar, but it seems
to handle the other cases reasonably. I'm wondering what the use-case
is due to the discussion of a parent passing work down the tree rather
than passing resources.

Also, why would the child expect to get its resources back? If a user
has given the wrong time for their job, or step, it's on them. This is
not to say that it shouldn't ask the parent if the resource is still
unassigned, but if it has been assigned, I don't see any reason to do
anything special about that, at least not until we can show that it
causes an overall job throughput degradation in the system.

[trws]

Using capacitor sounds like a good idea, but before you do much with it @SteVwonder, feel free to poke me repeatedly until I get the cleaned up refactored version submitted... I've had it in a mostly complete state for quite a while now, but it hasn't made it up.

[garlick]

At the highest level isn't it "I have some work to do and I will handle it by spawning a hierarchy of instances (because you told me to, or because I have deemed it the best way to run this workload). Then somehow, jobs known to the parent have to be offloaded to child instances?

[SteVwonder]

For the current dynamic scheduling research, there is no use-case. One hypothetical scenario Suraj and I discussed is the case where a user submits 200 job in quick succession and the running Flux instance has some pre-processor that aggregates these jobs into 1 large, hierarchical job. Let's say that job begin running (which causes another Flux instance to be spawned) and then shortly after the same user submits 5 more jobs. It would be cool if those 5 jobs could be submitted to the newly spawned Flux instance (and essentially be aggregated with the other jobs on-the-fly)

In general though, it seems a little restrictive to forbid hierarchical Flux instances from receiving new work and force them to essentially be offline schedulers.

[SteVwonder]

@garlick, that's a good point. I was imagining a 2 stage hand-off. The 1st stage happens when the child instance is created (which is a batch transfer of jobs) and the 2nd is an ongoing process provided by the parent's "jobs" service, but really, it could all go through that service (no need to special case that 1st stage).

[trws]

Fair point. It occurred to me after starting to write a reply that there is no reason the parent couldn't do that, it just never occurred to me as something the parent would want to do since it would have a more complete picture of the resource situation.

One nice thing about using capacitor for this actually, is if you give it a pipe as its standard input, it will just keep sucking in jobs until you close the other end of that pipe. Might make it easy to control if you're creating a prototype, while directly using the submission interface would probably be the way to go for a long-term solution.

[garlick]

Credit goes to @SteVwonder for the capacitor idea. All I did was mention that specialized initial program could be written in a high level language like capacitor.

On driving capacitor with stdin: if the enclosing instance grows some new service like sched.offload, maybe the easy prototype would be to write a flux-getjobs util that retrieves jobs from the parent and presents them on stdout in a way capacitor can ingest. Then the hierarchical job initial program could be a shell script that runs flux getjobs | flux capacitor. (Again avoiding the parent directly driving the child)

[SteVwonder]

Any objection to closing this issue? Seems like all the questions have been answered (and continued in #1151).

[garlick]

No let's close.