Cylc-flow: remove run and work dir configurations

[oliver-sanders]

Remove:

• global.cylc[platforms][<platform name>]run directory
• global.cylc[platforms][<platform name>]work directory

These have been made obsolete by the global.cylc[symlink dirs] configurations.

(We now install all flows into ~/cylc-run/<flow-name>, creating symlinks to locations on other filesystems where $HOME is not suitable).

Pull requests welcome!

[dpmatthews]

After looking again at #2779 and metomi/rose#2252 I fear there is still a requirement for run directory.

[oliver-sanders]

$HOME on the SSH remote is not the same $HOME as on the execution host.

If I've read them correctly these issues relate to $HOME on login nodes not being the same as $HOME on compute nodes which makes sense to me and is probably quite common.

If we can assume the compute nodes can see the login node $HOME filesystem and that it is mounted the same way on both nodes then we could potentially just expand $HOME on the login node before job submission.

This would also address the cases where $HOME is not set on compute nodes.

[oliver-sanders]

Dumping the result of a conversion with @dpmatthews on the matter here so we don't forget.

The remaining issues relate to $HOME differing between the submission and compute nodes on certain job hosts.

The new symlink dirs functionality could potentially provide a way forward here for cases where the login node can see the $HOME filesystem of the compute node, providing that this does not differ between compute nodes accessible from the same login node:

login-node:
  /home/
    login/
      me/ ($HOME)
       cylc-run/
         my-suite -> /home/compute/me/cylc-run/my-suite
compute-node:
  /home/
    compute/
      me/ ($HOME)
       cylc-run/
         my-suite

[dpmatthews]

I think we should go ahead and remove the existing settings (they don't make sense as platform settings in any case). If we find there is missing functionality that requires a new setting this can be addressed later.

[MetRonnie]

Just below is global.cylc[platforms][<platform name>]suite definition directory, which doesn't have a description. What is it, and can it be got rid of?

[dpmatthews]

I think that's probably a mistake and needs removing. Is it used anywhere?

[MetRonnie]

https://github.com/cylc/cylc-flow/blob/c45aa272fb45b21d9822a42ba2da4db1fb2c84e0/cylc/flow/task_job_mgr.py#L1055-L1055

@wxtim Do you know about it? I see in your PR #4126

Remove, and add obseletion warning for config item flow.cylc:[runtime][__TASK__][remote]suite definition directory (But _not_ the similar platform item from global.cylc).

[wxtim]

I have no idea why there was an exception for the platforms item. Does it allow the suite definition dir to change by platform? Seems odd, and like it is a mistake.

[MetRonnie]

The only real reference to the global suite definition directory is https://github.com/cylc/cylc-flow/blob/c45aa272fb45b21d9822a42ba2da4db1fb2c84e0/cylc/flow/task_job_mgr.py#L1055-L1055

And I can't find any references to remote_suite_d other than unit tests

[MetRonnie]

Also, the config change proposal says

(Note that we also need to deprecate the environment variables $CYLC_SUITE_DEF_PATH & $CYLC_SUITE_DEF_PATH_ON_SUITE_HOST.)

which suggests the global item should go too

[hjoliver]

I believe "remote suite definition directory" is an ancient thing that refers to where the user manually installed suite source files on the remote job host. Not needed any more.

[MetRonnie]

Should the --run-dir option of cylc suite-state be removed? I can't see any reason to keep it

https://github.com/cylc/cylc-flow/blob/4f4db8078dd5e7ca7c285c55d77fbb400f89846b/cylc/flow/scripts/suite_state.py#L148-L153

While I'm at it, I'm guessing we can remove this obsolete option altogether?

https://github.com/cylc/cylc-flow/blob/4f4db8078dd5e7ca7c285c55d77fbb400f89846b/cylc/flow/scripts/suite_state.py#L143-L146

[hjoliver]

Should the --run-dir option of cylc suite-state be removed? I can't see any reason to keep it

Still needed to interrogate other-user DBs, no?

While I'm at it, I'm guessing we can remove this obsolete option altogether?

:+1:

[MetRonnie]

Still needed to interrogate other-user DBs, no?

Clearly I am blind 🤪