Cylc-flow: make main loop more asynchronous

[oliver-sanders]

Supersedes #2866

As of #3492 the main loop is a asyncronous coroutine, however, the code is still synchronous.

Identify areas which can be made asyncronous and convert them.

Pull requests welcome!

[hjoliver]

Identify areas which can be made syncronous and convert them.

(Presumably you mean "asynchronous" there?)

[oliver-sanders]

The Scheduler main loop as it currently stands:

1. on:reload Reload TaskDefs
2. Process Command Queue
3. Release Runahead Tasks
4. Process SubProcPool (run callbacks?)
5. Should Process Tasks?
   
   
   
   • Check XTriggers
   
   
   • Check ext triggers
   
   
   • Task Pool matching
   
   
6. Process Task Pool
7. Process Task Messages
8. Process Command Queue
9. Process Task Events
10. on:reload Rebuild Data Store
11. DB write task event timers
12. Update data store
    
    
    
    • Update Data Store
    
    
    • Publish Data Store
    
    
13. DB write everything else
14. Check DB Health
15. Check Timeout
16. Check Shutdown
17. Check Filesystem Health
18. Check Suite Server Health
19. Profile
20. Sleep

A cursory glance suggests we might be able to run with these groupings:

(async items should be parallel safe with their neighbours)

1. on:reload
   
   
   
   • Reload TaskDefs
   
   
   • Rebuild Data Store
   
   
2. Process Task Pool
   
   
   
   • Release Runahead Tasks
   
   
   • Task Pool matching
   
   
   • Process Task Pool
   
   
3. Process Everything Else
   
   
   
   • async Task Messages
   
   
   • async Command Queue
   
   
   • async Task Events
   
   
   • asyc Process SubProcPool (run callbacks?)
   
   
4. Data Stores
   
   
   
   • async DB Writes
   
   
   • async update data store
   
   
   • publish data store
   
   
5. Check
   
   
   
   • async DB Health
   
   
   • async Timeout
   
   
   • async Shutdown
   
   
   • async Filesystem Health
   
   
   • async Suite Server Health
   
   
   • async Profile
   
   
6. Sleep

At the moment we don't have the best handle on what takes the longest, the main gains would probably be with the DB writes, and DB/FS health check routines.

I may do some follow up work to the Main Loop Plugins PR to make all main loop functions asynchronous coroutines and run them through the cylc.flow.main_loop infrastructure (so we can time and monitor them) but run them synchronously.

[oliver-sanders]

Scratch That!

After some whiteboard time I think this is the wrong way of approaching the problem. Will follow-up, probably with a new issue to explain this.

[oliver-sanders]

Converting main-loop functions to coroutines is of very little benefit unless we can break the main-loop lock-step.

Superseded by #3498