Cylc-flow: Cylc functional tests do not appear to respect cylc work/run directories

[kinow]

Hi,

I have an empty ~/cylc-run. And moved my suite run directory and work directory to /tmp/cylc-run, also empty.

# file $CYLC_HOME/etc/global-tests.rc
[hosts]
[[localhost]]
run directory=/tmp/cylc-run/

I ran cylc test-battery ... and noticed that the suite directory was created in both directories. Then I did

$ export CYLC_TEST_RUN_GENERIC=${CYLC_TEST_RUN_GENERIC:-true}
$ export CYLC_TEST_RUN_PLATFORM=${CYLC_TEST_RUN_PLATFORM:-true}
$ export CYLC_TEST_SKIP=${CYLC_TEST_SKIP:-}
$ export CYLC_TEST_IS_GENERIC=true
$ CYLC_TEST_TIME_INIT="$(date -u +'%Y%m%dT%H%M%SZ')"
$ export CYLC_TEST_TIME_INIT
$ prove -s -r tests/shutdown/00-cycle.t

Meaning that it would not go through cylc-test-battery I think. Only using the shell scripts imported from our tests. Here's the result:

kinow@kinow-VirtualBox:~$ ls ~/cylc-run
'cylctb-Wed Apr 10 10:51:54 NZST 2019'
kinow@kinow-VirtualBox:~$ ls /tmp/cylc-run
'cylctb-Wed Apr 10 10:51:54 NZST 2019'

And looks like only the structure was created in ~/cylc-run, while the files were correctly created in my new location for cylc run/work directory.

kinow@kinow-VirtualBox:~$ tree ~/cylc-run/cylctb-Wed\ Apr\ 10\ 10\:51\:54\ NZST\ 2019/
/home/kinow/cylc-run/cylctb-Wed Apr 10 10:51:54 NZST 2019/
└── shutdown
    └── 02-no-dir-sym
        ├── share
        └── work
            └── 1
                └── bar

6 directories, 0 files
/tmp/cylc-run/cylctb-Wed Apr 10 10:51:54 NZST 2019/
└── shutdown
    └── 02-no-dir
        ├── log
        │   ├── db
        │   ├── job
        │   │   └── 1
        │   │       └── bar
        │   │           ├── 01
        │   │           │   ├── job
        │   │           │   ├── job-activity.log
        │   │           │   ├── job.err
        │   │           │   ├── job.out
        │   │           │   ├── job.status
        │   │           │   └── job.xtrace
        │   │           └── NN -> 01
        │   ├── suite
        │   │   ├── log -> log.20190410T113019+12
        │   │   └── log.20190410T113019+12
        │   └── suiterc
        │       └── 20190409T233019Z-run.rc
        └── suite.rc.processed

10 directories, 11 files

[kinow]

Have not tested Cylc 7, due to the amount of file changed right now, and have a few things stashed already in Git.

[hjoliver]

Believe it or not, that's the expected result! As the global config reference says, run directory and work directory can be configured separately, and both default separately to $HOME/cylc-run/.

[kinow]

Ah! So that's because I forgot to change my work directory too! Let me try that again (I thought I had changed that too, sorry)

[hjoliver]

What is a bit confusing about this, unfortunately (and for historical reasons, I think), is that run directory sounds like it is the top level "suite run directory" that contains the suite service directory, log directories, and work directories. But in this context, run directory is the top level directory for everything except the work directories (which may have a much larger data requirement...).

[kinow]

run directory sounds like it is the top level "suite run directory"

Hmmm, just read your sentence here... does it mean that when we go with workflow.rc or whatever we adopt... that we will have to go through our other suite- options/configuration/files, etc?

If so, we will have an opportunity to change this perhaps?

[hjoliver]

Hmmm... there is a bug though. (This run directory configuration is probably not well tested because it is - or was - incompatible with suite invocation by rose suite-run ...which the developer sites all use)....

[hjoliver]

... if I only configure run directory for tests, some tests might be assuming the standard location for work directories (in fact some tests might assume the standard location for everything, if they don't extract the configuration correctly). Example:

cylc test-b -v tests/cyclers/00-daily.t                                                                                                                                        7.8.x  

ok 1 - 00-daily-validate
ok 2 - 00-daily.graph.plain-cmp-ok
ok 3 - 00-daily-run
find: ‘work’: No such file or directory
--- /home/oliverh/cylc-git/worktrees/7.8.x/tests/cyclers/00-daily-find.out      2019-04-09 23:36:31.039224108 +1200
+++ 00-daily-find       2019-04-10 14:10:01.649000000 +1200
@@ -34,9 +34,3 @@
 log/job/20140102T2300Z/foo/01/job.status
 log/job/20140102T2300Z/foo/01/job.xtrace
 log/job/20140102T2300Z/foo/01/job-activity.log
-work/20131231T2300Z/bar/typing
-work/20131231T2300Z/foo/typing
-work/20140101T2300Z/bar/typing
-work/20140101T2300Z/foo/typing
-work/20140102T2300Z/bar/typing
-work/20140102T2300Z/foo/typing
not ok 4 - 00-daily-find-cmp-ok

    stdout and stderr stored in: /tmp/oliverh/cylctb-20190410T020941Z/cyclers/00-daily
Failed 1/4 subtests 

Test Summary Report
-------------------
tests/cyclers/00-daily.t (Wstat: 0 Tests: 4 Failed: 1)
  Failed test:  4
Files=1, Tests=4, 19 wallclock secs ( 0.04 usr  0.02 sys +  3.51 cusr  0.68 csys =  4.25 CPU)
Result: FAIL

[hjoliver]

If so, we will have an opportunity to change this perhaps? ...

Well, we do quite frequently change config items names and locations, and just provide an automatic "upgrade" path via the parsec functionality for that purpose.

[matthewrmshin]

Hmmm, just read your sentence here... does it mean that when we go with workflow.rc or whatever we adopt... that we will have to go through our other suite- options/configuration/files, etc?

Yes, we intend to do this as part of the work to migrate rose suite-run functionality to Cylc - as we'll need to consolidate this sort of configuration settings between (current) Rose and Cylc.

[hjoliver]

On the bug I found above: tests/cyclers/00-daily.t correctly determines the run directory, but then assumes that the work directory is under the run directory.

[hjoliver]

@kinow - I'll mark this as invalid and close it. Reopen or post another if you think something still needs addressing here.

@matthewrmshin - can you add a reminder to the appropriate issue (whichever it is!) to fix the aforementioned ambiguity of this "run directory" (vs "work directory") config item, when we come to consolidate these settings?