Tuf: Consider switching to GitHub Actions for automated testing

[lukpueh]

Description of issue or feature request:
We switched from automated testing on Travis CI (Linux) and Appveyor (Windows) to GitHub Actions in our sister-project in-toto-golang (see https://github.com/in-toto/in-toto-golang/pull/72), mostly due to availability concerns about Appveyor and because it seems appealing to configure only one service for all platforms.

Let's consider switching here too and testing on more platforms than we currently do.

Current behavior:
Use Travis CI and Appveyor for automated testing on Linux for Windows (no tests on macOS)

Expected behavior:
Use GitHub Actions instead and also test on macOS.

[lukpueh]

Related: https://github.com/in-toto/in-toto/issues/407 and https://github.com/secure-systems-lab/securesystemslib/issues/297

[joshuagl]

Being able to test on Windows/Linux/macOS with the same CI make this a no-brainer to me.

[lukpueh]

Also consider injection vulnerabilities reported in https://crbug.com/project-zero/2070

[lukpueh]

See @jku's https://github.com/theupdateframework/tuf/issues/1204#issuecomment-725291899 as backing data point for the "configure only one service for all platforms" argument.

[jku]

I think the next step is someone testing this to see if it's good enough.

Documenting a potential test matrix:
python: Latest release of all supported cpython 3.x versions, cpython 2.7. Currently that would be [2.7, 3.6, 3.7, 3.8, 3.9]
platform: Latest release of Linux, Windows, Mac: [ubuntu-latest, windows-latest, macos-latest]

Github does offer specific platform versions as well but I'm not sure it makes sense to use those? The matrix size as proposed is 15 test runs. As a bonus it might be nice to add pypy3 to python version list (not sure how much additional benefits it gives -- but it is something pip tests with and currently TUF tests pass with pypy3): that should probably be done separately as the build dependencies might require a bit of tweaking there.

There's also at least two special cases that we want to test (they do not need to be part of the matrix):
special: Securesystemslib master branch on "ubuntu-latest" with python "3.x" (this should not fail a build)
special: Lint on "ubuntu-latest" with python "3.x"

This should be a good starting point: https://docs.github.com/en/free-pro-team@latest/actions/guides/building-and-testing-python
Note this suggestion:

You'll need to invoke tox using the -e py option to choose the version of Python in your PATH, rather than specifying a specific version.

Something like this might work as the matrix setup in workflow yaml:

strategy:
  matrix:
    python-version: [2.7, 3.6, 3.7, 3.8, 3.9]
    os: [ubuntu-latest, windows-latest, macos-latest]
    toxenv: py
    include:
      - python-version: 3.x
        os: ubuntu-latest
        toxenv: with-sslib-master
        experimental: true
      - python-version: 3.x
        os: ubuntu-latest
        toxenv: lint

Things to keep in mind:

• caching pip seems to be fairly easy but requires setting up the path per OS (eww): https://github.com/actions/cache/blob/main/examples.md#python---pip
• I just realized we could cache even the tox directory: this would speed things up even more

[lukpueh]

Also, we just ran out of build minutes on Travis. :(

[joshuagl]

Also, we just ran out of build minutes on Travis. :(

Wow. I didn't even realise we had a build minutes limit. Does that mean we can't do any more Travis builds in December?

[lukpueh]

I didn't know either until I noticed that builds didn't finish and came across this message on https://travis-ci.com/organizations/theupdateframework:

Builds have been temporarily disabled for private and public repositories due to a negative credit balance. Please go to the Plan page to replenish your credit balance.

Not exactly sure when credits will reset.

[lukpueh]

IIUC, GitHub actions has no limits for open source projects (see first sentence in Billing for GitHub Actions). I am happy to set this up first thing next week.

[joshuagl]

IIUC, GitHub actions has no limits for open source projects (see first sentence in Billing for GitHub Actions). I am happy to set this up first thing next week.

No limits for open source is my understanding too, thanks for prioritising this @lukpueh!

[jku]

am happy to set this up first thing next week.

I did some experiments but am likely to not concentrate on this at least early this coming week so feel free to copy-paste or just take notes from my branch:

• example results: https://github.com/jku/tuf/actions
• commit with workflow: https://github.com/jku/tuf/commit/3ba1d0de847ff5eb7193c9c848583f12f52fa38e

Notes:

• should disable workflow commands in steps that might log untrusted things https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/
• strategy.fail-fast default is true -- should consider what it should be and whether maybe lint should be run first as fail-fast and then others as matrix but not fail-fast (not sure how that would happen, just thinking out loud)
• I disabled anything that didn't work on first try just to get something passing
  
  
  
  • python 2.7 failed to pip install something
  
  
  • windows failed test_6_download_target
  
  
  • pylint does not seem to support python 3.9 yet?
  
  
  • weird bug in pip leads to nasty results with tox (maybe this was what travis spent the minutes on): https://github.com/pypa/pip/issues/9215
  
  

[lukpueh]

Thanks for the excellent spade-work, @jku! If you don't mind I'll try to PR something tomorrow.

Regarding your notes:

• I haven't fully understood how, as an actions user, I can protect myself against set-env and add-path vulnerabilities. Do I need to audit the actions I use? Will need to further investigate.
• Not sure what the best solution for fail-fast is. Your thoughts seem quite reasonable. I can try to create a configuration for that, but maybe in a follow-up PR.
• Tests on 2.7 worked for me, not sure why they didn't for you. Do you still have the build results somewhere?
• I also fixed windows builds, they just required a small patch in test_6_download_target.
• pylint indeed seems to not yet fully support python 3.9 (see e.g. https://github.com/PyCQA/pylint/issues/3882). But 3.8 should be just as fine.

Apart from that I still need to finalize coveralls and fossa configuration.

[jku]

• I haven't fully understood how, as an actions user, I can protect myself against set-env and add-path vulnerabilities. Do I need to audit the actions I use? Will need to further investigate.

We shouldn't really be especially vulnerable I think -- none of our steps look like they could be easily made to print commands without control of the source code -- I'd say only the "Run tox" step is suspect (in the sense that there's so much code running we can't really rule out the possibility of untrusted output).

I think the only thing we can do is something like

echo "::stop-commands::`echo -n ${{ github.token }} | sha256sum | head -c 64`"

we could do that already in "Find pip cache dir" immediately after the ::set-output command: that should prevent any commands from being accepted from the output in the whole build after that.

• Not sure what the best solution for fail-fast is. Your thoughts seem quite reasonable. I can try to create a configuration for that, but maybe in a follow-up PR.

:+1:

• Tests on 2.7 worked for me, not sure why they didn't for you. Do you still have the build results somewhere?

Uh no, I seem to have deleted some builds I didn't intend to... Maybe it was a mistake in an earlier configuration

[lukpueh]

1242 replaced travis/appveyor with GitHub Actions. Remaining issues discussed here are ticketized in https://github.com/theupdateframework/tuf/issues/1243, https://github.com/theupdateframework/tuf/issues/1244, https://github.com/theupdateframework/tuf/issues/1246, https://github.com/theupdateframework/tuf/issues/1247 and https://github.com/theupdateframework/tuf/issues/1248. Closing...