Chapel: Github Actions CI Testing Wishlist

[ben-albrecht]

This meta-issue tracks improvements we'd like to make to pre-merge testing (Travis CI), which executes the test script: util/buildRelease/smokeTest

• [x] make mason support
• [ ] .futures tracking (#4964)
• [ ] Awareness of tests modified in PR (#4951)
• [ ] chplspell support (#6731)
• [ ] pylint support
• [ ] Shell/Makefile linters, e.g. shellcheck
• [ ] Check for trailing whitespace and tabs (PR: #7848)
  
  
  
  • This functionality exists, but is disabled until we remove all trailing whitespace and tabs from the source code.
  
  
• [ ] Check .graph/GRAPHFILES (#6422)
• [ ] Make sure PR isn't committing a binary (occasionally done for tests)
• [ ] Make sure absolute paths are not committed to files in test directory, like these.
• [ ] Make sure CHIPs build properly and all CHIPs are listed in README
• [ ] Make sure there are no dangling Chapel programs without a .notest
• [ ] Make sure parser has been rebuilt if any parser source files have been modified.
  
  
  
  • Bonus points if we can ensure the parser was rebuilt with the correct version of bison\
  
  
• [ ] Host documentation that is built in pre-merge testing to be easily viewed from a browser
  
  
  
  • docs can now be viewed as an artifact in the github-actions CI and viewed locally, but this not an ideal solution
  
  
• [x] Separate Travis CI tests for pleasingly parallel tests
• [x] make beginning of individual Travis build names distinct for quick comprehension (so you don't have to scroll to the end of TEST_COMMAND=\"make test-venv && CHPL_HOME=$PWD ./util/run-in-venv.bash ./util/test/check_annotations.py\" to know that line is for annotations, for instance)
• [x] make docs support (PR: #6921)
• [x] lookForBadAlloc support (PR: #6917)
• [x] annotate.py support (#4998) (PR: #6917)

[ben-albrecht]

A small correction to my earlier estimate -- because we already build chpldoc in the Travis CI testing, building the docs would add an additional ~30 seconds to the testing time.

[ronawho]

I'd also be in support of adding a python linter as well as running util/devel/lookForBadAlloc (#6163)

[ben-albrecht]

When discussing adding make docs to the testing, some suggested adding it as a separate Travis CI job that runs in parallel. Because chpldoc requires a built compiler, a separate job for chpldoc and make docs would not save a significant amount of time.

I think this optimization may be useful for us in the future, especially as we beef up Travis CI testing. However, I don't think it's necessary for adding make docs, or any of the other changes listed in this issue (assuming they run relatively quickly).

For reference, make docs would add ~30 seconds of build time, and current build times fluctuate between 2-4 minutes.

[ronawho]

Running make docs as a concurrent travis build with https://github.com/chapel-lang/chapel/pull/6921

We weren't building chpldoc as part of the travis testing before, so doing make docs would have added 90-120 seconds to the travis build.

[ronawho]

Added a link to an issue for running the .graph/GRAPHFILES checks that we do as part of start_test -- https://github.com/chapel-lang/chapel/issues/6422

[lydia-duncan]

Low priority: added a line about make the configurations more visually distinct, because looking at this isn't helpful when I just want to know what the build is that is still running -

[ronawho]

Low priority: added a line about make the configurations more visually distinct, because looking at this isn't helpful when I just want to know what the build is that is still running -

Done in https://github.com/chapel-lang/chapel/pull/10420

[ronawho]

Github actions (github's CI) is out of beta as of today. Might be worth looking at to see if there's any concrete benefits for us over Travis

https://github.com/features/actions

[ronawho]

It looks like github actions has more than just CI, so I've forked that to https://github.com/chapel-lang/chapel/issues/14442

[ben-albrecht]

Added make mason to the wish list after @e-kayrakli discovered a deprecation warning in mason that has gone unnoticed for a while (doesn't cause an error in nightly testing).

[ben-albrecht]

https://github.com/chapel-lang/chapel/pull/14985 is another case where mason broke on master due to lack of make mason in Travis CI

[e-kayrakli]

This should probably be updated/repurposed for Github Actions world.