Cylc-flow: Add setuptools and package Cylc as a Python module

[kinow]

This way we will be able to use Cylc as a dependency in other projects such as part of larger systems, Jupyter Notebooks, or even in experiments to extend Cylc.

This is also useful to create packages for PYPI, Conda, and RPM (see #1337). Will serve as basis for #2802 too, in order to have script commands as code, with tests, and even re-using in components such as Cylc UI Server to run a workflow for instance.

Things to consider when confirming this issue can be closed:

• [x] Install and uninstall via pip
• [x] Basic tests, unit
• [x] Functional tests
• [x] Documentation generation
• [x] Run simple suites from terminal
• [x] Use it as a module, i.e. from cylc import ... in a terminal, but also in Python code, and Jupyter Notebooks
• [x] CYLC_DIR usage (moved to #3080)
• [x] Test that the CYLC_VERSION works. (moved to #3081)
  
  
  
  • Comment from @hjoliver : ATM users can run different suites at different versions if necessary (e.g. long-running suite that's not upgraded yet, and a new suite written for new cylc) and every suite job invokes the right cylc version thanks to $CYLC_VERSION in the job environment, with cylc versions installed in (e.g.) in /opt/cylc/[VERSION].
  
  

[kinow]

Note to self: need a section for the documentation dependencies. We need sphinx at least.

[kinow]

Group added, docs, with only sphinx for now. Also updated Travis install.sh to do a pip install .[doc] instead of installing sphinx. So other dependencies can also be grouped in setup.py :+1:

[kinow]

Looking at some tests failing in Travis, and noticed a warning that appears but without failing the test from what I could tell.

===(     197;600  1/2  1/2  2/6  1/8  0/2 )=============================python3: can't open file '/home/travis/virtualenv/python3.7.1/tests/lib/python/diffr.py': [Errno 2] No such file or directory
===(     202;633  1/2  1/2  5/6  2/8  1/2 )=============================python3: can't open file '/home/travis/virtualenv/python3.7.1/tests/lib/python/diffr.py': [Errno 2] No such file or directory
===(     203;634  1/2  1/2  6/6  2/8  1/2 )=============================

~I could not find any reference to diffr in lib/cylc, so it looks like this one is a requirement for running the tests in Cylc, and the library is this one from PYPI. I'll update #2990 to include it.~

So actually diffr.py is in tests/lib/python/diffr.py, and is used by tests/lib/python/test_diffr.py, but probably due to the changes in this branch that test is not able to use the diffr.py file. Reverting commit and will see what needs to be done to get that test to work again in Travis, with virtualenv, and setuptools.

[kinow]

A quick way to analyze project import dependencies and check what's used is using pydeps.

$ PYTHONPATH=./lib pydeps --externals lib/cylc
[
    "collections",
    "colorama",
    "isodatetime",
    "jose",
    "logging",
    "mock",
    "parsec",
    "pytest",
    "sqlite3",
    "testfixtures",
    "zmq"
]
['collections', 'colorama', 'isodatetime', 'jose', 'logging', 'mock', 'parsec', 'pytest', 'sqlite3', 'testfixtures', 'zmq']

This can be useful to check against setup.py/setup.cfg and verify the list of dependencies. Note that it finds parsec as that is considered an external dependency to cylc for now :+1:

[kinow]

Finally! It was CYLC_DIR. cylc-test-battery was cd'ing into CYLC_DIR. I changed it to cd into TRAVIS_BUILD_DIR if running in Travis. But then several tests failed.

Then, changed to actually `export CYLC_DIR="${TRAVIS_BUILD_DIR}", and now it worked. I've added tasks (bullet items) to the issue description.

The next one that needs further work is documentation (hopefully easier). It appears to be broken in virtual environments too. I'm getting the same error message as Travis right now (even though it says success, the last state of the build for documentation actually failed!):

Building the HTML Cylc Documentation with Sphinx:
/home/travis/virtualenv/python3.7.1/bin/cylc-make-docs: line 39: cd: /home/travis/virtualenv/python3.7.1/doc/: No such file or directory
... Generating the command reference ...
/home/travis/virtualenv/python3.7.1/bin/cylc-make-docs: line 41: ./src/custom/make-commands.sh: No such file or directory
... Generating multi-page User Guide...
Application error:
config directory doesn't contain a conf.py file (./src)
... Generating single-page User Guide...
Application error:
config directory doesn't contain a conf.py file (./src)
Done.
The command "cylc make-docs" exited with 0.

[kinow]

And once docs are working, I'm planning on creating a separate ticket for CYLC_DIR, and another one for the CYLC_VERSION, and preparing the pull request for setup.py for review, as it's usable, and the CYLC_DIR can be treated as a separate issue :+1:

[hjoliver]

Phew, well done solving that.

[kinow]

Alright, had to fix make-docs, but now I think the basics are working. Will sleep now and review once again tomorrow. Hoooopefully ready to be reviewed on Monday. Travis CI is happy too \o/