Cylc-flow: The future of check-software

[sadielbartholomew]

Following on from a point arising from reviewing https://github.com/cylc/cylc-doc/pull/26#issuecomment-510081950, after the:

1. splitting up of the cylc/cylc mono-repo to inter-connected multi-repositories; &
2. a number of large-scale package dependency changes from Cylc 7 to 8 (& beyond), so far notably the:
   
   
   
   • Python 2 -> 3 port;
   
   
   • docs conversion from LaTeX to Sphinx;
   
   
   • architectural changes;
   
   
   • development of the web UI,
   
   

we have completely changed the external software dependencies for our full ecosystem (see my outputs below), & divided them into subsets of requirements based on each sub-component.

The check-software command currently sits under cylc-flow, & reports what is needed for this. But its place going forward is into clear & merits discussion:

1. Is it still worth having, noting that:
   
   
   
   • the setup.py of setuptools we now use has settings managing dependency specifications, both for compulsory ones & optional ones for subsidiary features.
   
   
   • in the future we might even want, & be able to, provide ready-made environments e.g. conda via conda-forge that can be sourced, so that the user does not even have to consider dependencies & can just source those.
   
   
   • it may still be useful as a diagnostic tool, i.e. an admin command, if it is not useful to users.
   
   
2. What repository will it sit in?
3. What repositories should it capture dependencies for? Noting that:
   
   
   
   • users will generally not want to use every single (non-admin) component under the cylc organisation. For example, they are unlikely to have need to build the docs themselves, therefore sphinx as a docs requirement is not relevant to most users.
   
   
4. Should we amend it in any way, taking the above decisions into account?
   
   
   
   • How to organise under headings? Perhaps report on a repo-by-repo basis?
   
   
   • Most of the packages reported now are Python-based, so it seems sensible to drop the Python: descriptor prepended to each package in the output.
   
   

---

Requirements tested for

Cylc 8.0a0 i.e. master (in cylc/cylc-flow)

Outputs:

Checking your software...

Individual results:
================================================================================
Package (version requirements)                           Outcome (version found)
================================================================================
                              *REQUIRED SOFTWARE*                              
Python (3.7+)...........................FOUND & min. version MET (3.7.3.final.0)
Python:zmq (any)..................................................FOUND (18.0.0)
Python:jose (any)..................................................FOUND (3.0.1)
Python:graphene (any)..............................................FOUND (2.1.6)
Python:colorama (any)..............................................FOUND (0.4.1)
Python:ansimarkup (any)............................................FOUND (1.4.0)
Python:protobuf (3.7.0+)........................FOUND & min. version MET (3.7.1)

              *OPTIONAL SOFTWARE for the configuration templating*              
Python:EmPy (3.3.4+)...............................................NOT FOUND (-)
================================================================================

Summary:
                          ****************************                          
                             Core requirements: ok                             
                           Full-functionality: not ok                           
                          **************************** 

Cylc 7.8.x (the old cylc/cylc)

Outputs:

Checking your software...

Individual results:
================================================================================
Package (version requirements)                           Outcome (version found)
================================================================================
                              *REQUIRED SOFTWARE*                              
Python (2.6+, <3)......................FOUND & min. version MET (2.7.16.final.0)

        *OPTIONAL SOFTWARE for the GUI & dependency graph visualisation*        
Python:pygraphviz (any)............................................NOT FOUND (-)
graphviz (any)....................................................FOUND (2.30.1)
Python:pygtk (2.0+)................................................NOT FOUND (-)

             *OPTIONAL SOFTWARE for the HTTPS communications layer*             
Python:requests (2.4.2+)...........................................NOT FOUND (-)
Python:urllib3 (any)...............................................NOT FOUND (-)
Python:OpenSSL (any)...............................................NOT FOUND (-)

              *OPTIONAL SOFTWARE for the configuration templating*              
Python:EmPy (any)..................................................NOT FOUND (-)

                 *OPTIONAL SOFTWARE for the HTML documentation*                 
Python:sphinx (1.5.3+).............................................NOT FOUND (-)
================================================================================

Summary:
                          ****************************                          
                             Core requirements: ok                             
                           Full-functionality: not ok                           
                          **************************** 

[kinow]

I think it is still useful. Even with setuptools, we may still have dependendy issues (deps updated post installations, wrong version range, missing system dep, etc).

So having a command that tells whether the environment is OK or not seems useful to me - helpful especially for troubleshooting I think.

But I think we don't want to check for docs dependencies like sphinx. IMHO we should display what is required for running Cylc only.

Bruni

[matthewrmshin]

I guess it can become a report on OS, local environment, site configuration, plugins, etc that may be useful for a support team to understand local issues.

I'll also imagine that we can integrate the command into cylc --version. E.g. The old perl -v prints the basic version information, whereas perl -V prints the detailed version information.

Do we need a separate command for the web UI? Perhaps we can have an About link to print out detailed version information?

[kinow]

Do we need a separate command for the web UI? Perhaps we can have an About link to print out detailed version information?

Right now we have only the Cylc version in the footer. Later would be good to start semver in the cylc-ui too, then include just that information for support. For reporting issues, OS+browser versions, a description of the issue, the cylc-ui version, and the output of the browser console should be enough I guess.

[hjoliver]

cylc check-software could potentially go, along with logo/icon image files etc., in a repo-spanning cylc/cylc-common project ... and report on what's installed for the whole system. (Just an idea). Ref: discussion about cylc-common on Riot chat (I think) ... in which we decided against it because it seemed that it would only contain logos.

[kinow]

Mistakenly created a duplicated issue. Closed it, will use this one to discuss the future of check-software. IMO (from the other ticket), we should focus on workflow/graph/etc, and let the tooling&dependency check to conda/pip/rpm/etc, by telling users in the docs about possible ways for doing that. Especially given that we now have Python + JS dependencies in the project, and some projects may use version ranges that may not guarantee that transitive dependencies are correct.

[hjoliver]

I guess conda can't tell us the installed UI version, because it isn't a conda package itself?

$ conda list | grep cylc
# packages in environment at /home/oliverh/.conda/envs/cylc1:
cylc                      8.0a1                    py37_2    kinow
cylc-flow                 8.0a1                    py37_3    kinow
cylc-uiserver             0.1                      py37_1    kinow

From the cylc conda recipe:

source:
  # Cylc UI
  - url: https://github.com/cylc/cylc-ui/releases/download/{{ ui_version }}/cylc-ui-{{ ui_version }}-dist.zip
    sha256: {{ ui_hash }}
folder: cylc-ui

Is there some way of "discovering" this, once installed?

[kinow]

Is there some way of "discovering" this, once installed?

I don't think so. The Cylc UI information that we have in the recipe is just the URL, the UI version, and SHA 256 sum. However, once the package is built, this information is lost.

Package information is also kept in the repository data, but the extra variables are not included: https://conda.anaconda.org/kinow/linux-64/repodata.json

So I think there is no way to retrieve that information (the ui_version Jinja variable or its rendered value... maybe if https://github.com/conda/conda/issues/6439 is accepted, though I don't see it happening anytime soon).

I think there is also no way of knowing that in non-conda installations, so maybe we will need to tell users to look at the browser footer to find the Cylc UI version?