Cylc-flow: conf: configuration file changes

[oliver-sanders]

As documented in the config change proposal document:

• [x] The .cylc extension and the new configuration file names.
  
  
  
  • [x] With support for the deprecated suite.rc filename with a warning of some kind for the user.
  
  
• [x] The new global config hierarchy.
• [ ] Any changes to CYLC_CONF_PATH required.
• [x] The new CYLC_SITE_CONF_PATH variable.

Pull requests welcome!

[MetRonnie]

Are we looking to totally rename suite --> flow?

[oliver-sanders]

Yep, though I think we may still be a little unsettled on terminology, generally speaking:

• Suite has been renamed flow (as no one knows what a suite is) but we also call them workflows.
• Parallel executions of the same workflow under SoD are called flows.

[hjoliver]

To avoid ambiguity, I wonder if we should:

• refer to a "suite" only as a "workflow"
• reserve "flow" for "parallel execution of (part of) the same workflow under SoD"

The only spanner-in-the-works here would be that we agreed on flow.cylc as the new config file name, which rolls off the tongue nicely. But we haven't implemented that that change yet, so we could potentially go for complete clarity: workflow.cylc

[TomekTrzeciak]

What's the upshot of renaming suite.rc for Cylc 8, given that the file format will hardly change?

[hjoliver]

There’s not much “upshot” (consequences?) but the reason is (quoting Oliver on our chat platform today):

we have been using the word suite for a while now, however, very few people seem to know what this means and it makes us sound like niche software. So we have come to call suites what they really are, workflows, and Cylc, the thing which runs the workflows, a workflow engine.

[hjoliver]

(We will of course maintain backward compatibility with suite.rc)
(Similarly .rc is a rather archaic file suffix that few use anymore, and not very descriptive in our context)

[oliver-sanders]

Update: We've recently moved to setting the Cylc version to nextversion-dev, when we set up the config hierarchy it would be good if the dev version fits nicely into that schema i.e:

• 8
  
  
  
  • 8.0
  
  
  • a1
  
  
  • a2
    
    
    
    
    
    • a3
    
    
    
    • dev
    
    
    
  
  
  • 8.1
  
  
  • 8.2
  
  

This way the dev version can be configured separately, but also, it can just pick up the config for the next release which makes life easier.

[TomekTrzeciak]

There’s not much “upshot” (consequences?) but the reason is (quoting Oliver on our chat platform today):

we have been using the word suite for a while now, however, very few people seem to know what this means and it makes us sound like niche software. So we have come to call suites what they really are, workflows, and Cylc, the thing which runs the workflows, a workflow engine.

(We will of course maintain backward compatibility with suite.rc)
(Similarly .rc is a rather archaic file suffix that few use anymore, and not very descriptive in our context)

All good points, but is it worth it without any real "meat" behind this change? I don't think the "suite" nomenclature is going away anytime soon - we have it all over our systems (environment variables, file paths, documentation, ...) and I expect similar for other users.

[MetRonnie]

What do people think of this way of upgrading existing suite.rc to flow.cylc? https://github.com/cylc/cylc-flow/pull/3755#pullrequestreview-461602562

[oliver-sanders]

is it worth it

Personally I didn't want to go arbitrary renaming things, however, as Cylc8 has evolved the case has deepened. We are aiming for something of a clean slate with Cylc8 and nomenclature is a large part of that theory. The old suite.rc filename will still be supported so the only real impact on users is the confusion of explaining that the file has been renamed.

There are a few pros to the change albeit small ones:

• Cylc7 suites will run under Cylc8, however, Cylc8 workflows won't be back-compatible to Cylc7 so changing the filename is kinda helpful there.
• The .cylc extension enables us to more easily associate files with the Cylc configuration language for text editors, GitHub, etc (the glob suite*.rc only works in a few places).
• Whilst in some communities the word "suite" will likely live on for some time, when users stop running rose suite-run and start using "Cylc Flow" to run the workflow in their flow.cylc file this will change, slowly. This terminology change may assist the transition from Rose -> Cylc, suites -> workflows.
• It will be less of a barrier to new adopters, especially in non-meteorological areas who don't realise Cylc's generality or even its existence due to our going against the grain of accepted industry terminology.
• It's just generally awkward introducing new users to our workflow engine then explaining that they write their workflows in a file called "suite".

environment variables, file paths, documentation

All of these will change at our end with Cylc8 (obviously with back-compat), the word "suite" will be eradicated from the system pre release.

[MetRonnie]

What about flow-tests.rc? Should it become global-tests.cylc?

Also I've noticed several references to global.rc - what's that?

[hjoliver]

Also I've noticed several references to global.rc - what's that?

The original global config file name (and still on branch 7.8.x). Changing to global.cylc https://cylc.github.io/cylc-admin/proposal-config-changes.html#config-file-naming

[hjoliver]

What about flow-tests.rc? Should it become global-tests.cylc?

Yes :+1:

[MetRonnie]

After the PR's merged, would it make sense to add a GH Actions workflow/step that checks PRs' additions don't include suite.rc etc going forward, in case of old habits dying hard?

[hjoliver]

After the PR's merged, would it make sense to add a GH Actions workflow/step that checks PRs' additions don't include suite.rc etc going forward, in case of old habits dying hard?

That could potentially be annoying. We might still have to use the word "suite" sometimes, e.g. when talking about backward compatibility and transitioning to Cylc 8 ...

[oliver-sanders]

I think we can manually police this one if worried we can always add our own pre-commit hooks to grep git diff --cached or something like that.

[MetRonnie]

Re: new global config hierarchy

Both the site and user config may consist of a number of files. For example, for release 8.0.1 the following config files will be read in (if they exist) in order from least specific to most specific:

• <config-dir>/flow/global.cylc
  
  
  
  • <config-dir>/flow/8/global.cylc
  
  
  • <config-dir>/flow/8.0/global.cylc
  
  
  • <config-dir>/flow/8.0.1/global.cylc
  
  

Does that mean the configs should "cascade" (like in CSS), or is it that the most specific one _only_ should be used?

[oliver-sanders]

the following config files will be read in (if they exist) in order from least specific to most specific:

So for this example all four files would be read.

[MetRonnie]

What should we do with cylc get-site-config --print-site-dir etc? Should it print <config-dir>/flow/8.0.1/? Or should it print the hierarchy?

[oliver-sanders]

I think --print-site-dir and --print-user-dir should probably be removed.

The main use of these options is in tests/f/lib/bash/test_header where it is used to source the global-tests.cylc file. I'm not quite sure what the future of the global-tests.cylc is, the work I'm currently doing on the test battery makes it somewhat moot.

For now the simplest way forward would be to remove the concept of a "site" global-tests.cylc file (as it's not currently used) and make it so that the global-tests.cylc file has to be located in the directory with the full version i.e:

# untested
- SITE_TESTS_CONF_FILE="$(cylc get-global --print-site-dir)/global-tests.cylc"         
- USER_TESTS_CONF_FILE="$(cylc get-global --print-user-dir)/global-tests.cylc
+ USER_TESTS_CONF_FILE="$HOME/.cylc/flow/$(cylc version)/global-tests.cylc"

[MetRonnie]

• Any changes to CYLC_CONF_PATH required.

@oliver-sanders I take it there aren't any changes needed and this can be ticked off?