Cylc-flow: Better "cylc monitor" - rewrite for ncurses?

[hjoliver]

The cylc monitor utility seems to be viewed as quite useful. It's currently quite unsophisticated, however: it simply prints strings to the terminal, sleeps for a second and calls os.system("clear") before re-display, and there is no capacity for interaction with the suite or manipulating the view at run time. It would be nice to have a more sophisticated monitor - with functionality akin to the GUI - where for e.g. you could change sort order at run time, or hit "k" and then type in the name of tasks to kill, etc. Use of ncurses (which has a Python interface) would presumably make it easy to do this sort of thing. A commonly used ncurses application with some similar functionality is htop - for monitoring Linux processes: http://hisham.hm/htop/

[hjoliver]

Note comments in #1739 - we should also consider making the terminal monitor conform to the user's GUI colour choices, or if that's not possible (and it probably isn't) at least make the terminal colours easily customizable.

[oliver-sanders]

I took a quick look at this during jet-lag recovery.

I had Cylc monitor in mind when thinking about the Tree View, it should be pretty easy to implement GScan + TreeView for a new cylc monitor, it's just a basic tree after all. The task and job icons can all be replicated with unicode.

Implementation options:

1. Implement a tree widget in ncurses
   
   
   
   • This will be a bit of a headache and probably take a little while
   
   
2. Use the built-in tree widget in urwid
   
   
   
   • This widget is not responsive so it would be a global update (like the current cylc monitor)
   
   
   • There are certain annoying colour restrictions
   
   
   • urwid can produce terminal applications but has other outputs as well
   
   
3. Use Vue Blessed - https://github.com/lyonlai/blessed-vue
   
   
   
   • Lets us share the UI code for building trees, setting the state of groups of tasks etc
   
   
   • May provide responsive features for us out of the box
   
   
   • Would require a node installation on users machines (at present we only have one on the suite servers). This is actually quite a big barrier due to security etc.
   
   
   • Would be slightly awkward to fit into the cylc flow click-based CLI.
   
   
   • Would probably require a separate Vue project with the common components factored out of the Cylc UI project.
   
   
   • Would have to go via the UI server, the option to go directly to the suite would be tricky, though this makes it easier as the data provisioning is already taken care of.
   
   

Thoughts:

• Option (3) is probably the most compelling but requiring Nodejs on users machines might be a barrier to adoption at larger sites. We would probably have to spin it off into another repo which is kinda annoying as Cylc Flow should ship with a decent monitoring tool really.
• I knocked together a POC for option (2) in a couple of hours, quick, simple.

[kinow]

Would need more time to consider the other two options as I've not well versed in shell 😬 thoughts on some items of the 3rd option below.

Use Vue Blessed - https://github.com/lyonlai/blessed-vue

Hadn't heard about this one! Interesting. Shame it hasn't seen much activity since 2018, and it's using Vue 2.5 (we are on 2.6 which has extra stuff that we could be using and is not available on 2.5 - https://github.com/lyonlai/blessed-vue/issues/11).

Would require a node installation on users machines (at present we only have one on the suite servers). This is actually quite a big barrier due to security etc.

Looking at the readme, looks like they mention npm when installing the dependency. If it's not a runtime dependency, then maybe it wouldn't be necessary.

Would be slightly awkward to fit into the cylc flow click-based CLI.

True.

Would probably require a separate Vue project with the common components factored out of the Cylc UI project.

I have a post-it somewhere about doing something similar. I think I mentioned to @hjoliver before that in the future once we have Cylc 8 done, I would like to explore creating Jupyter Notebook widgets.

The idea would be to have something that would allow users to create cells like

%%cylc tree "kinow@niwa-001/five"

Or so, and that would add the tree view component into the notebook. Allowing users to embed images, monitor workflows, use for publication, etc.

But instead of a separate project, I was thinking in having the packages folder. Lerna manages that for you (jupyterlab/lumino use that), but you can do that with Yarn too (vuejs/vue does that). This way you can have cylc-ui/package.json to manage the whole solution, and then create cylc-ui/packages/components/tree, cylc-ui/packages/components/graph, etc... these components could then even be written in TypeScript (e.g. Vuetify).

[oliver-sanders]

Shame it hasn't seen much activity since 2018

Kinda annoying. I wonder how necessary it is to use a Vue plugin to Blessed. Blessed doesn't seem to have a tree view (though blessed-contrib does) so we would probably end up writing our own anyway...

Dammit blessed is cool.

[hjoliver]

Damn, that is cool.

[hjoliver]

If the way forward via blessed or hand-coded ncurses isn't clear or easy, I'd suggest:

• Quickly finish off @oliver-sanders' urwid treeview POC
  
  
  
  • I thought it was quite impressive
  
  
  • if it doesn't perform well for enormous suites, it would still be useful for most suites
  
  
• Look at more sophisticated and performant options on the back-burner

[kinow]

+1 to @hjoliver's suggestion above. We still have time before the 8.0 release, so that will give us time to test and improve it if necessary.

[dwsutherland]

blessed references an extension that seems much more active:
https://github.com/yaronn/blessed-contrib/blob/master/README.md

(and popular... 13k stars!)

[dwsutherland]

Also, would be nice for it to hook up to either workflow or uiserver graphql endpoint (so it can be used with the workflow in isolation of the rest of the components)