Cylc-flow: Replace scheduler.py main loop by asyncio loop

[kinow]

When working on the Tornado PoC (#2825) one of the tasks involved making the scheduler.py main loop to run alongside Tornado's main loop (i.e. an asyncio loop since Python 3.x).

It worked after replacing scheduler.py main loop by Tornado's loop. However, there are several blocking calls in Cylc's Suite Server Program main loop; calls to read and write files to disk, calls to sqlite, even network calls. These could all be parallelized through coroutines with asyncio.

I spent some time looking at the code, and I think it could work. Though I would need more time with the code in order to do some further tests and come up with some benchmarks and pros/cons.

[matthewrmshin]

Agreed on principle.

Just another emphasis on the logic that writes to the primary SQLite file. It is important that we design the asynchronous to block the suite work flow to ensure the integrity of the persistent data for unscheduled stop+restart. Obviously, we can still design this logic to be asynchronous relative to the web server, but have enough logic to stop the suite workflow from moving on until we have a successful write to the primary SQLite file.

[kinow]

I think this might be superseded by a new issue to do the same but with focus on pyzmq/ZeroMQ, and priority queues? If that's the case, we can close this one :+1:

[matthewrmshin]

Very different scope here, I suppose. The main loop is not just about communication, so we may want to repurpose this issue to have an emphasis on the parts that are not to do with 0mq. Unless the new issue is going to fully supersede those as well?

[oliver-sanders]

Superseded by #3495