Cylc-flow: --start-task: ignore pre-initial dependencies?

[hjoliver]

Raised by @dpmatthews in the last meeting.

On a clean start at the initial (or other) cycle point we ignore pre-start-cycle dependencies as a convenience for users. Should we do the same if starting a new run from one or more individual tasks in the graph? e.g. cylc play --start-task=blah.2

[hjoliver]

Firstly, for the "start tasks" themselves, we do already ignore pre-initial (and any other) dependencies. At start-up the start tasks are put into the pool with their prerequisites set to satisfied.

[scheduling]   
    cycling mode = integer  
    initial cycle point = 1  
    [[graph]]   
        P1 = "foo[-P1] => foo => bar & baz"                                          
[runtime]               
    [[foo, bar, baz]]  
        script = "sleep 10" 

Here, cylc play --start-task=foo.10 will not stall immediately, because the dependence on foo.9 is (in effect) ignored.

[hjoliver]

But cylc play --start-task=foo.10 will stall at foo.11 on this:

[scheduling]
    cycling mode = integer
    initial cycle point = 1
    [[graph]]
        P1 = "foo[-P1] & bar[-P2] => foo => bar & baz"
[runtime]
    [[foo, bar, baz]]
        script = "sleep 10"

because the next instance of foo still depends on a pre-initial point. Result:

2021-06-11T14:29:05+12:00 INFO - Workflow shutting down - AUTOMATIC
2021-06-11T14:29:05+12:00 WARNING - Partially satisfied prerequisites left over:
        foo.11 is waiting on:
        * bar.9 succeeded
2021-06-11T14:29:06+12:00 INFO - DONE

[hjoliver]

This one's more interesting:

[scheduling]
   cycling mode = integer
   initial cycle point = 1
   [[graph]]
      P1 = """
         foo[-P1] => foo
         bar[-P1] => bar
         foo => bar & baz => qux
      """
[runtime]
   [[foo, bar, baz, qux]]
      script = "sleep 2"

cylc play --start-task=foo.10 will cause foo and baz to cycle on, but partially satisfied prerequisites will accumulate for bar and qux at each point. (The scheduler will shut down at the runahead limit on current master, or keep going on #4258).

It's probably safe to assume that the user would have intended bar's pre-initial deps to be satisfied here.

[hjoliver]

OK, I think I've convinced myself that we should ignore pre-initial dependence when starting the scheduler with --start-task :+1:

[dpmatthews]

Are we going to apply the same logic to any new flow (i.e. one created via cylc trigger --re-flow or cylc spawn)?

What happens if you specify multiple --start-task options which cover different cycle points?
I assume we should only ignore pre-initial dependencies for the earliest of the cycle points in this case.

[hjoliver]

What happens if you specify multiple --start-task options which cover different cycle points?
I assume we should only ignore pre-initial dependencies for the earliest of the cycle points in this case.

Yes, that's what I've implemented in #4259

[hjoliver]

Are we going to apply the same logic to any new flow (i.e. one created via cylc trigger --re-flow

Hmm, a possible future enhancement maybe. I guess each flow would need its own pre-initial-ignore point.

Cycle point-based pre-initial ignore isn't a cure-all for stalling with --start-task or trigger --reflow BTW. There's also the possibility of off-flow dependence within the same cycle point, which is a more difficult problem. Graph traversal would be required to figure out if particular dependencies should be ignored or not.

[dpmatthews]

Are we going to apply the same logic to any new flow (i.e. one created via cylc trigger --re-flow

Hmm, a possible future enhancement maybe. I guess each flow would need its own pre-initial-ignore point.

I think it's just as important to support this for cylc trigger and cylc spawn as it is for cylc play.
At the moment we use warm starts when we have HPC problems and need to re-start from an earlier cycle. Using --start-task will give the ability to start part way through a cycle which will certainly help. However, this still requires starting a new run. I think the best way to handle this situation in the future will be to stop the existing flow and start a new flow using cylc trigger --re-flow. This will be much nicer since we'll retain the full history and all the old job logs. However, we'll need it to ignore pre-initial dependencies in order to be able to do this.

[hjoliver]

OK, agreed. I'll bang up a new issue for that... #4270