Cylc-flow: job environment init

[hjoliver]

All suite-running functional tests are now failing in my Python venv cylc-flow environment. Pretty sure they used to work, and I'm not sure what I've change d recently, but the problem is that suite jobs in the test battery (as executed by ./etc/bin/run-functional-tests.sh) do not have access to the venv cylc library (if I just run a suite from within the venv, jobs do work). So:

1. I tried using a flow-tests.rc to configure to activate then environment

   • hence: #3375

2. I (and our docs) are confused about global init-script and job-init-env.sh for Cylc 8

   • topic of this issue

User Guide

The only references to job-init-env.sh are:

1. Under "Task Job Access to Cylc":

   (login shell etc...)
   To customize the environment more generally for Cylc on jobs hosts, use job-init-env.sh.

2. Under "Installation"

   Your site administrator may customise the environment for all task jobs by adding a site job-init-env.sh file and populate it with appropriate contents. If customisation is still required, you can add your own ${HOME}/.cylc/job-init-env.sh file and populate it with the
   appropriate contents.

The global init-script item is also mentioned, just once (in the global config reference):

If specified, the value of this setting will be inserted to just before the init-script section of all job scripts that are to be submitted to the specified remote host.

Under suite config reference, for init-script it says:

The original intention for this item was to allow remote tasks to source login scripts to configure their access to cylc, but this should no longer be necessary (see :ref:HowTasksGetAccessToCylc).

(where the ref suggests use job-init-env.sh as above)

cylc-flow code

The only reference to job-init-env.sh is in etc/job.sh:

    typeset JOB_INIT_ENV=
    # conf/job-init-env.sh for back-compat pre 7.7.0.
    JOB_INIT_ENV="${HOME}/.cylc/job-init-env.sh"
    if [[ -f "${JOB_INIT_ENV}" ]]; then
        if "${CYLC_DEBUG:-false}"; then
            . "${JOB_INIT_ENV}"
        else
            . "${JOB_INIT_ENV}" 1>'/dev/null' 2>&1
        fi
    fi
    # Init-Script
    cylc__job__run_inst_func 'global_init_script'
    cylc__job__run_inst_func 'init_script'

Clearly only a user job-init-env.sh is sourced here (no site one), just before global init-script. And what's that back-compat comment: does it imply job-init-env itself is only retained here for back compat with cylc-7, or does the comment just need to be deleted because it refers to some now-deleted back-compat code.

[hjoliver]

@matthewrmshin - I need to scrape your brain on this in your final week! My thoughts:

In terms of functionality both global init-script and job-init-env.sh can achieve the same thing (they are used in the same place in task job scripts) but one is a host config item, and one has to be installed on hosts (but is then invisible as cylc config, which is probably a downside). But as shown above, we don't load a site job-init-env.sh anymore, and the documentation is a bit confused.

Do we need both? Can we get rid of job-init-env.sh? Once @wxtim's config unification is done, I imagine we can just have init script which is set in site or user config, to do any local environment config that's needed?

[hjoliver]

(We haven't had to use either of these at NIWA, and I'm unsure of what you might be using for cylc-7 site environment config at Met Office ... if more than just login scripts).

[matthewrmshin]

After #2438 (which gives our job scripts #!/bin/bash --login or equivalent), I believe that we have mostly eliminated the need for the init-script ecosystem. But...

We are still trying to determine how this will work in the world of Conda.

[hjoliver]

Right, so job scripts always use a login shell now (I had forgotten that :grimacing: ).

Looking back at 7.8.x we still support a global job-init-env.sh in CYLC_DIR, and well as a user one. ... but it looks like we removed the global one for cylc-8 since CYLC_DIR no longer exists.

I think to close this issue I should either:

• restore global job-init-env.sh support, via /etc - currently the designated location for site global config files (or sub-dirs thereof at least). Unlike the config file, this probably doesn't need to be cylc version specific
• OR remove support for job-init-env.sh altogether

@matthewrmshin - do you have an opinion on this based on Met Office use of cylc-7? Your comment above suggests you don't use init-script config anymore, but what about job-init-env.sh?

[matthewrmshin]

No, we don't use job-init-env.sh either. The login shell has been fully sufficient.

I'll vote for a full removal of the old ecosystem, and instead focus on how to do this in the world of Conda environment, etc.