Cylc-flow: Remove reference test functionality from cylc.

[hjoliver]

Reference tests were a way to write a self-testing suite in the absence of a proper test framework. Now that we have such a framework, we might as well remove all the reference test config and coding from cylc. Instead of using shutdown handlers to execute diagnosis scripts (usually to compare triggering results with that in a reference log) we should use use proper tests (e.g.cmp_triggering_ok within the framework...)

[hjoliver]

[meeting] we agreed:

• worth doing (for one thing, users will no longer see reference test functionality in the UI)
• all irrelevant lines should be deleted from the many test reference.log files (which will still be used when this issue is dealt with).

[matthewrmshin]

Recent notes:

• With #3286, the only reference test item left in the suite.rc file is expected task failures.
• We should aim to move our tests to work under pytest, and aiming to do a lot more unit tests. [UPDATE (hjoliver) moved this point to #3505]

[hjoliver]

Status of this Issue.

• most reference test-specific code has already been removed - all runs now write triggering info to the log.
• the remaining config item needs to stay as test-specific (as per quick discussion in CylcCon 2020)

So I think this can be closed.

[hjoliver]

Re-opening due to possibly-unresolved minor questions:

• can we get rid of the reference test CLI options for cylc run?
• (is the last remaining ref test config item, expected task failures is the blocker for that?... )

From the suite.rc reference:

Reference tests are finite-duration suite runs that abort with non-zero exit status if cylc fails, if any task fails, if the suite times out, or if a shutdown event handler that (by default) compares the test run with a reference run reports failure.

Aborting if any task fails by default is probably unnecessary in a ref test run. abort if any task fails can be a generic (not test-specific) item. And expected task failures excludes a few tasks from the general abort order... which also seems reasonable to me as a general config item. We need to revisit this...

[matthewrmshin]

Chipping in here.

You can get rid of the CLI reference test option any time. The simplest way is to detect reference.log file. If it exists with the suite definition directory, then generate a reference log for the current run and perform a reference test on suite completion. There is no need to have a CLI option.

The expected task failure option is a separate thing. You should replace it with other more modern built-in task failure handling mechanisms.